00-Pandas_Tutorial_FULL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PANDAS TUTORIAL - COREY SCHAFFER \n",
    "\n",
    "this is all files merged into one with `nbmerge` may have issues because of possible variable name conflicts.\n",
    "\n",
    "i will try to fix this but i suggest not to run this notebook and use it as reading material only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 01 - Getting Started with Data Analysis - Installation and Loading Data\n",
    "\n",
    "https://youtu.be/ZyhVh-qRZPA?si=kIUi2jlhJuGFYyWX\n",
    "\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<p style=\"background:black\">\n",
    "<code style=\"background:black;color:white\">jupyter notebook\n",
    "</code>\n",
    "</p> in shell will start the server and start new notebook. \n",
    "\n",
    "am using vscode for that purpose so i could also use *git* in in bult terminal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load csv files is easy unlike opening and closing file in regular python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataframes are just like excel sheets with rows and columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>I am not primarily a developer, but I write co...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Thailand</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Web development or web design</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech meetups or events in your area;Courses on...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>United States</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>22.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Ukraine</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech meetups or events in your area;Courses on...</td>\n",
       "      <td>30.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent;Multiracial</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88878</th>\n",
       "      <td>88377</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once a month but more than once per ...</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>Canada</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88879</th>\n",
       "      <td>88601</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88880</th>\n",
       "      <td>88802</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88881</th>\n",
       "      <td>88816</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Independent contractor, freelancer, or self-em...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88882</th>\n",
       "      <td>88863</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>Spain</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Professional degree (JD, MD, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>Hispanic or Latino/Latina;White or of European...</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>88883 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Respondent                                         MainBranch Hobbyist  \\\n",
       "0               1             I am a student who is learning to code      Yes   \n",
       "1               2             I am a student who is learning to code       No   \n",
       "2               3  I am not primarily a developer, but I write co...      Yes   \n",
       "3               4                     I am a developer by profession       No   \n",
       "4               5                     I am a developer by profession      Yes   \n",
       "...           ...                                                ...      ...   \n",
       "88878       88377                                                NaN      Yes   \n",
       "88879       88601                                                NaN       No   \n",
       "88880       88802                                                NaN       No   \n",
       "88881       88816                                                NaN       No   \n",
       "88882       88863                                                NaN      Yes   \n",
       "\n",
       "                                             OpenSourcer  \\\n",
       "0                                                  Never   \n",
       "1                                Less than once per year   \n",
       "2                                                  Never   \n",
       "3                                                  Never   \n",
       "4                             Once a month or more often   \n",
       "...                                                  ...   \n",
       "88878  Less than once a month but more than once per ...   \n",
       "88879                                              Never   \n",
       "88880                                              Never   \n",
       "88881                                              Never   \n",
       "88882                            Less than once per year   \n",
       "\n",
       "                                              OpenSource  \\\n",
       "0      The quality of OSS and closed source software ...   \n",
       "1      The quality of OSS and closed source software ...   \n",
       "2      The quality of OSS and closed source software ...   \n",
       "3      The quality of OSS and closed source software ...   \n",
       "4      OSS is, on average, of HIGHER quality than pro...   \n",
       "...                                                  ...   \n",
       "88878  The quality of OSS and closed source software ...   \n",
       "88879  The quality of OSS and closed source software ...   \n",
       "88880                                                NaN   \n",
       "88881  OSS is, on average, of HIGHER quality than pro...   \n",
       "88882  OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                                              Employment  \\\n",
       "0                 Not employed, and not looking for work   \n",
       "1                     Not employed, but looking for work   \n",
       "2                                     Employed full-time   \n",
       "3                                     Employed full-time   \n",
       "4                                     Employed full-time   \n",
       "...                                                  ...   \n",
       "88878             Not employed, and not looking for work   \n",
       "88879                                                NaN   \n",
       "88880                                 Employed full-time   \n",
       "88881  Independent contractor, freelancer, or self-em...   \n",
       "88882             Not employed, and not looking for work   \n",
       "\n",
       "                      Country         Student  \\\n",
       "0              United Kingdom              No   \n",
       "1      Bosnia and Herzegovina  Yes, full-time   \n",
       "2                    Thailand              No   \n",
       "3               United States              No   \n",
       "4                     Ukraine              No   \n",
       "...                       ...             ...   \n",
       "88878                  Canada              No   \n",
       "88879                     NaN             NaN   \n",
       "88880                     NaN             NaN   \n",
       "88881                     NaN             NaN   \n",
       "88882                   Spain  Yes, full-time   \n",
       "\n",
       "                                                 EdLevel  \\\n",
       "0                              Primary/elementary school   \n",
       "1      Secondary school (e.g. American high school, G...   \n",
       "2               Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "3               Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "4               Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "...                                                  ...   \n",
       "88878                          Primary/elementary school   \n",
       "88879                                                NaN   \n",
       "88880                                                NaN   \n",
       "88881                                                NaN   \n",
       "88882                 Professional degree (JD, MD, etc.)   \n",
       "\n",
       "                                          UndergradMajor  ...  \\\n",
       "0                                                    NaN  ...   \n",
       "1                                                    NaN  ...   \n",
       "2                          Web development or web design  ...   \n",
       "3      Computer science, computer engineering, or sof...  ...   \n",
       "4      Computer science, computer engineering, or sof...  ...   \n",
       "...                                                  ...  ...   \n",
       "88878                                                NaN  ...   \n",
       "88879                                                NaN  ...   \n",
       "88880                                                NaN  ...   \n",
       "88881                                                NaN  ...   \n",
       "88882  Computer science, computer engineering, or sof...  ...   \n",
       "\n",
       "                                  WelcomeChange  \\\n",
       "0       Just as welcome now as I felt last year   \n",
       "1       Just as welcome now as I felt last year   \n",
       "2       Just as welcome now as I felt last year   \n",
       "3       Just as welcome now as I felt last year   \n",
       "4       Just as welcome now as I felt last year   \n",
       "...                                         ...   \n",
       "88878                                       NaN   \n",
       "88879                                       NaN   \n",
       "88880                                       NaN   \n",
       "88881                                       NaN   \n",
       "88882  Somewhat less welcome now than last year   \n",
       "\n",
       "                                            SONewContent   Age Gender Trans  \\\n",
       "0      Tech articles written by other developers;Indu...  14.0    Man    No   \n",
       "1      Tech articles written by other developers;Indu...  19.0    Man    No   \n",
       "2      Tech meetups or events in your area;Courses on...  28.0    Man    No   \n",
       "3      Tech articles written by other developers;Indu...  22.0    Man    No   \n",
       "4      Tech meetups or events in your area;Courses on...  30.0    Man    No   \n",
       "...                                                  ...   ...    ...   ...   \n",
       "88878  Tech articles written by other developers;Tech...   NaN    Man    No   \n",
       "88879                                                NaN   NaN    NaN   NaN   \n",
       "88880                                                NaN   NaN    NaN   NaN   \n",
       "88881                                                NaN   NaN    NaN   NaN   \n",
       "88882  Tech articles written by other developers;Indu...  18.0    Man    No   \n",
       "\n",
       "                     Sexuality  \\\n",
       "0      Straight / Heterosexual   \n",
       "1      Straight / Heterosexual   \n",
       "2      Straight / Heterosexual   \n",
       "3      Straight / Heterosexual   \n",
       "4      Straight / Heterosexual   \n",
       "...                        ...   \n",
       "88878                      NaN   \n",
       "88879                      NaN   \n",
       "88880                      NaN   \n",
       "88881                      NaN   \n",
       "88882  Straight / Heterosexual   \n",
       "\n",
       "                                               Ethnicity Dependents  \\\n",
       "0                                                    NaN         No   \n",
       "1                                                    NaN         No   \n",
       "2                                                    NaN        Yes   \n",
       "3                           White or of European descent         No   \n",
       "4               White or of European descent;Multiracial         No   \n",
       "...                                                  ...        ...   \n",
       "88878                                                NaN         No   \n",
       "88879                                                NaN        NaN   \n",
       "88880                                                NaN        NaN   \n",
       "88881                                                NaN        NaN   \n",
       "88882  Hispanic or Latino/Latina;White or of European...         No   \n",
       "\n",
       "                SurveyLength                  SurveyEase  \n",
       "0      Appropriate in length  Neither easy nor difficult  \n",
       "1      Appropriate in length  Neither easy nor difficult  \n",
       "2      Appropriate in length  Neither easy nor difficult  \n",
       "3      Appropriate in length                        Easy  \n",
       "4      Appropriate in length                        Easy  \n",
       "...                      ...                         ...  \n",
       "88878  Appropriate in length                        Easy  \n",
       "88879                    NaN                         NaN  \n",
       "88880                    NaN                         NaN  \n",
       "88881                    NaN                         NaN  \n",
       "88882  Appropriate in length                        Easy  \n",
       "\n",
       "[88883 rows x 85 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "shape gives size of the dataframe as tuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(88883, 85)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "info gives more detailed info like the type of columns. its *info()* and not just *info*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*object types are just strings*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 88883 entries, 0 to 88882\n",
      "Data columns (total 85 columns):\n",
      " #   Column                  Non-Null Count  Dtype  \n",
      "---  ------                  --------------  -----  \n",
      " 0   Respondent              88883 non-null  int64  \n",
      " 1   MainBranch              88331 non-null  object \n",
      " 2   Hobbyist                88883 non-null  object \n",
      " 3   OpenSourcer             88883 non-null  object \n",
      " 4   OpenSource              86842 non-null  object \n",
      " 5   Employment              87181 non-null  object \n",
      " 6   Country                 88751 non-null  object \n",
      " 7   Student                 87014 non-null  object \n",
      " 8   EdLevel                 86390 non-null  object \n",
      " 9   UndergradMajor          75614 non-null  object \n",
      " 10  EduOther                84260 non-null  object \n",
      " 11  OrgSize                 71791 non-null  object \n",
      " 12  DevType                 81335 non-null  object \n",
      " 13  YearsCode               87938 non-null  object \n",
      " 14  Age1stCode              87634 non-null  object \n",
      " 15  YearsCodePro            74331 non-null  object \n",
      " 16  CareerSat               72847 non-null  object \n",
      " 17  JobSat                  70988 non-null  object \n",
      " 18  MgrIdiot                61159 non-null  object \n",
      " 19  MgrMoney                61157 non-null  object \n",
      " 20  MgrWant                 61232 non-null  object \n",
      " 21  JobSeek                 80555 non-null  object \n",
      " 22  LastHireDate            79854 non-null  object \n",
      " 23  LastInt                 67155 non-null  object \n",
      " 24  FizzBuzz                71344 non-null  object \n",
      " 25  JobFactors              79371 non-null  object \n",
      " 26  ResumeUpdate            77877 non-null  object \n",
      " 27  CurrencySymbol          71392 non-null  object \n",
      " 28  CurrencyDesc            71392 non-null  object \n",
      " 29  CompTotal               55945 non-null  float64\n",
      " 30  CompFreq                63268 non-null  object \n",
      " 31  ConvertedComp           55823 non-null  float64\n",
      " 32  WorkWeekHrs             64503 non-null  float64\n",
      " 33  WorkPlan                68914 non-null  object \n",
      " 34  WorkChallenge           68141 non-null  object \n",
      " 35  WorkRemote              70284 non-null  object \n",
      " 36  WorkLoc                 70055 non-null  object \n",
      " 37  ImpSyn                  71779 non-null  object \n",
      " 38  CodeRev                 70390 non-null  object \n",
      " 39  CodeRevHrs              49790 non-null  float64\n",
      " 40  UnitTests               62668 non-null  object \n",
      " 41  PurchaseHow             61108 non-null  object \n",
      " 42  PurchaseWhat            62029 non-null  object \n",
      " 43  LanguageWorkedWith      87569 non-null  object \n",
      " 44  LanguageDesireNextYear  84088 non-null  object \n",
      " 45  DatabaseWorkedWith      76026 non-null  object \n",
      " 46  DatabaseDesireNextYear  69147 non-null  object \n",
      " 47  PlatformWorkedWith      80714 non-null  object \n",
      " 48  PlatformDesireNextYear  77443 non-null  object \n",
      " 49  WebFrameWorkedWith      65022 non-null  object \n",
      " 50  WebFrameDesireNextYear  62944 non-null  object \n",
      " 51  MiscTechWorkedWith      59586 non-null  object \n",
      " 52  MiscTechDesireNextYear  64511 non-null  object \n",
      " 53  DevEnviron              87317 non-null  object \n",
      " 54  OpSys                   87851 non-null  object \n",
      " 55  Containers              85366 non-null  object \n",
      " 56  BlockchainOrg           48175 non-null  object \n",
      " 57  BlockchainIs            60165 non-null  object \n",
      " 58  BetterLife              86269 non-null  object \n",
      " 59  ITperson                87141 non-null  object \n",
      " 60  OffOn                   86663 non-null  object \n",
      " 61  SocialMedia             84437 non-null  object \n",
      " 62  Extraversion            87305 non-null  object \n",
      " 63  ScreenName              80486 non-null  object \n",
      " 64  SOVisit1st              83877 non-null  object \n",
      " 65  SOVisitFreq             88263 non-null  object \n",
      " 66  SOVisitTo               88086 non-null  object \n",
      " 67  SOFindAnswer            87816 non-null  object \n",
      " 68  SOTimeSaved             86344 non-null  object \n",
      " 69  SOHowMuchTime           68378 non-null  object \n",
      " 70  SOAccount               87828 non-null  object \n",
      " 71  SOPartFreq              74692 non-null  object \n",
      " 72  SOJobs                  88066 non-null  object \n",
      " 73  EntTeams                87841 non-null  object \n",
      " 74  SOComm                  88131 non-null  object \n",
      " 75  WelcomeChange           85855 non-null  object \n",
      " 76  SONewContent            69560 non-null  object \n",
      " 77  Age                     79210 non-null  float64\n",
      " 78  Gender                  85406 non-null  object \n",
      " 79  Trans                   83607 non-null  object \n",
      " 80  Sexuality               76147 non-null  object \n",
      " 81  Ethnicity               76668 non-null  object \n",
      " 82  Dependents              83059 non-null  object \n",
      " 83  SurveyLength            86984 non-null  object \n",
      " 84  SurveyEase              87081 non-null  object \n",
      "dtypes: float64(5), int64(1), object(79)\n",
      "memory usage: 57.6+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to display all columns & rows use *set_option*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_columns', 85)\n",
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get first 5 data of the dataframe we  use *df.head()*\n",
    "\n",
    "*df.head(10)* gives first 10 data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>I am not primarily a developer, but I write co...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Thailand</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Web development or web design</td>\n",
       "      <td>...</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech meetups or events in your area;Courses on...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Respondent                                         MainBranch Hobbyist  \\\n",
       "0           1             I am a student who is learning to code      Yes   \n",
       "1           2             I am a student who is learning to code       No   \n",
       "2           3  I am not primarily a developer, but I write co...      Yes   \n",
       "\n",
       "               OpenSourcer                                         OpenSource  \\\n",
       "0                    Never  The quality of OSS and closed source software ...   \n",
       "1  Less than once per year  The quality of OSS and closed source software ...   \n",
       "2                    Never  The quality of OSS and closed source software ...   \n",
       "\n",
       "                               Employment                 Country  \\\n",
       "0  Not employed, and not looking for work          United Kingdom   \n",
       "1      Not employed, but looking for work  Bosnia and Herzegovina   \n",
       "2                      Employed full-time                Thailand   \n",
       "\n",
       "          Student                                            EdLevel  \\\n",
       "0              No                          Primary/elementary school   \n",
       "1  Yes, full-time  Secondary school (e.g. American high school, G...   \n",
       "2              No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "\n",
       "                  UndergradMajor  ...  \\\n",
       "0                            NaN  ...   \n",
       "1                            NaN  ...   \n",
       "2  Web development or web design  ...   \n",
       "\n",
       "                             WelcomeChange  \\\n",
       "0  Just as welcome now as I felt last year   \n",
       "1  Just as welcome now as I felt last year   \n",
       "2  Just as welcome now as I felt last year   \n",
       "\n",
       "                                        SONewContent   Age Gender Trans  \\\n",
       "0  Tech articles written by other developers;Indu...  14.0    Man    No   \n",
       "1  Tech articles written by other developers;Indu...  19.0    Man    No   \n",
       "2  Tech meetups or events in your area;Courses on...  28.0    Man    No   \n",
       "\n",
       "                 Sexuality Ethnicity Dependents           SurveyLength  \\\n",
       "0  Straight / Heterosexual       NaN         No  Appropriate in length   \n",
       "1  Straight / Heterosexual       NaN         No  Appropriate in length   \n",
       "2  Straight / Heterosexual       NaN        Yes  Appropriate in length   \n",
       "\n",
       "                   SurveyEase  \n",
       "0  Neither easy nor difficult  \n",
       "1  Neither easy nor difficult  \n",
       "2  Neither easy nor difficult  \n",
       "\n",
       "[3 rows x 85 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get datas at end of dataframe use *df.tail()*\n",
    "\n",
    "*df.tail(10)* gives last 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Respondent</th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>...</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>88880</th>\n",
       "      <td>88802</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88881</th>\n",
       "      <td>88816</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Independent contractor, freelancer, or self-em...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88882</th>\n",
       "      <td>88863</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>Spain</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Professional degree (JD, MD, etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>...</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>18.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>Hispanic or Latino/Latina;White or of European...</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Respondent MainBranch Hobbyist              OpenSourcer  \\\n",
       "88880       88802        NaN       No                    Never   \n",
       "88881       88816        NaN       No                    Never   \n",
       "88882       88863        NaN      Yes  Less than once per year   \n",
       "\n",
       "                                              OpenSource  \\\n",
       "88880                                                NaN   \n",
       "88881  OSS is, on average, of HIGHER quality than pro...   \n",
       "88882  OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                                              Employment Country  \\\n",
       "88880                                 Employed full-time     NaN   \n",
       "88881  Independent contractor, freelancer, or self-em...     NaN   \n",
       "88882             Not employed, and not looking for work   Spain   \n",
       "\n",
       "              Student                             EdLevel  \\\n",
       "88880             NaN                                 NaN   \n",
       "88881             NaN                                 NaN   \n",
       "88882  Yes, full-time  Professional degree (JD, MD, etc.)   \n",
       "\n",
       "                                          UndergradMajor  ...  \\\n",
       "88880                                                NaN  ...   \n",
       "88881                                                NaN  ...   \n",
       "88882  Computer science, computer engineering, or sof...  ...   \n",
       "\n",
       "                                  WelcomeChange  \\\n",
       "88880                                       NaN   \n",
       "88881                                       NaN   \n",
       "88882  Somewhat less welcome now than last year   \n",
       "\n",
       "                                            SONewContent   Age Gender Trans  \\\n",
       "88880                                                NaN   NaN    NaN   NaN   \n",
       "88881                                                NaN   NaN    NaN   NaN   \n",
       "88882  Tech articles written by other developers;Indu...  18.0    Man    No   \n",
       "\n",
       "                     Sexuality  \\\n",
       "88880                      NaN   \n",
       "88881                      NaN   \n",
       "88882  Straight / Heterosexual   \n",
       "\n",
       "                                               Ethnicity Dependents  \\\n",
       "88880                                                NaN        NaN   \n",
       "88881                                                NaN        NaN   \n",
       "88882  Hispanic or Latino/Latina;White or of European...         No   \n",
       "\n",
       "                SurveyLength SurveyEase  \n",
       "88880                    NaN        NaN  \n",
       "88881                    NaN        NaN  \n",
       "88882  Appropriate in length       Easy  \n",
       "\n",
       "[3 rows x 85 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "loading results schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema_df = pd.read_csv('data/survey_results_schema.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(85, 2)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 02 Dataframe & Series basics\n",
    "\n",
    "https://youtu.be/zmdjNSmRXF4?si=m69-X1HqZwkKlidV\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv')\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_columns', 85)\n",
    "pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Column</th>\n",
       "      <th>QuestionText</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Respondent</td>\n",
       "      <td>Randomized respondent ID number (not in order ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MainBranch</td>\n",
       "      <td>Which of the following options best describes ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hobbyist</td>\n",
       "      <td>Do you code as a hobby?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OpenSourcer</td>\n",
       "      <td>How often do you contribute to open source?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OpenSource</td>\n",
       "      <td>How do you feel about the quality of open sour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Employment</td>\n",
       "      <td>Which of the following best describes your cur...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Country</td>\n",
       "      <td>In which country do you currently reside?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Student</td>\n",
       "      <td>Are you currently enrolled in a formal, degree...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>EdLevel</td>\n",
       "      <td>Which of the following best describes the high...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>UndergradMajor</td>\n",
       "      <td>What was your main or most important field of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>EduOther</td>\n",
       "      <td>Which of the following types of non-degree edu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>OrgSize</td>\n",
       "      <td>Approximately how many people are employed by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>DevType</td>\n",
       "      <td>Which of the following describe you? Please se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>YearsCode</td>\n",
       "      <td>Including any education, how many years have y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Age1stCode</td>\n",
       "      <td>At what age did you write your first line of c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>YearsCodePro</td>\n",
       "      <td>How many years have you coded professionally (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>CareerSat</td>\n",
       "      <td>Overall, how satisfied are you with your caree...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>JobSat</td>\n",
       "      <td>How satisfied are you with your current job? (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>MgrIdiot</td>\n",
       "      <td>How confident are you that your manager knows ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>MgrMoney</td>\n",
       "      <td>Do you believe that you need to be a manager t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>MgrWant</td>\n",
       "      <td>Do you want to become a manager yourself in th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>JobSeek</td>\n",
       "      <td>Which of the following best describes your cur...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>LastHireDate</td>\n",
       "      <td>When was the last time that you took a job wit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>LastInt</td>\n",
       "      <td>In your most recent successful job interview (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>FizzBuzz</td>\n",
       "      <td>Have you ever been asked to solve FizzBuzz in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>JobFactors</td>\n",
       "      <td>Imagine that you are deciding between two job ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>ResumeUpdate</td>\n",
       "      <td>Think back to the last time you updated your r...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>CurrencySymbol</td>\n",
       "      <td>Which currency do you use day-to-day? If your ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>CurrencyDesc</td>\n",
       "      <td>Which currency do you use day-to-day? If your ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>CompTotal</td>\n",
       "      <td>What is your current total compensation (salar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>CompFreq</td>\n",
       "      <td>Is that compensation weekly, monthly, or yearly?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>ConvertedComp</td>\n",
       "      <td>Salary converted to annual USD salaries using ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>WorkWeekHrs</td>\n",
       "      <td>On average, how many hours per week do you work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>WorkPlan</td>\n",
       "      <td>How structured or planned is your work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>WorkChallenge</td>\n",
       "      <td>Of these options, what are your greatest chall...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>WorkRemote</td>\n",
       "      <td>How often do you work remotely?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>WorkLoc</td>\n",
       "      <td>Where would you prefer to work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>ImpSyn</td>\n",
       "      <td>For the specific work you do, and the years of...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>CodeRev</td>\n",
       "      <td>Do you review code as part of your work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>CodeRevHrs</td>\n",
       "      <td>On average, how many hours per week do you spe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>UnitTests</td>\n",
       "      <td>Does your company regularly employ unit tests ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>PurchaseHow</td>\n",
       "      <td>How does your company make decisions about pur...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>PurchaseWhat</td>\n",
       "      <td>What level of influence do you, personally, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>LanguageWorkedWith</td>\n",
       "      <td>Which of the following programming, scripting,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>LanguageDesireNextYear</td>\n",
       "      <td>Which of the following programming, scripting,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>DatabaseWorkedWith</td>\n",
       "      <td>Which of the following database environments h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>DatabaseDesireNextYear</td>\n",
       "      <td>Which of the following database environments h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>PlatformWorkedWith</td>\n",
       "      <td>Which of the following platforms have you done...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>PlatformDesireNextYear</td>\n",
       "      <td>Which of the following platforms have you done...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>WebFrameWorkedWith</td>\n",
       "      <td>Which of the following web frameworks have you...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>WebFrameDesireNextYear</td>\n",
       "      <td>Which of the following web frameworks have you...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>MiscTechWorkedWith</td>\n",
       "      <td>Which of the following other frameworks, libra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>MiscTechDesireNextYear</td>\n",
       "      <td>Which of the following other frameworks, libra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>DevEnviron</td>\n",
       "      <td>Which development environment(s) do you use re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>OpSys</td>\n",
       "      <td>What is the primary operating system in which ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>Containers</td>\n",
       "      <td>How do you use containers (Docker, Open Contai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>BlockchainOrg</td>\n",
       "      <td>How is your organization thinking about or imp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>BlockchainIs</td>\n",
       "      <td>Blockchain / cryptocurrency technology is prim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>BetterLife</td>\n",
       "      <td>Do you think people born today will have a bet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>ITperson</td>\n",
       "      <td>Are you the \"IT support person\" for your family?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>OffOn</td>\n",
       "      <td>Have you tried turning it off and on again?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>SocialMedia</td>\n",
       "      <td>What social media site do you use the most?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>Extraversion</td>\n",
       "      <td>Do you prefer online chat or IRL conversations?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>ScreenName</td>\n",
       "      <td>What do you call it?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>SOVisit1st</td>\n",
       "      <td>To the best of your memory, when did you first...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>SOVisitFreq</td>\n",
       "      <td>How frequently would you say you visit Stack O...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>SOVisitTo</td>\n",
       "      <td>I visit Stack Overflow to... (check all that a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>SOFindAnswer</td>\n",
       "      <td>On average, how many times a week do you find ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>SOTimeSaved</td>\n",
       "      <td>Think back to the last time you solved a codin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>SOHowMuchTime</td>\n",
       "      <td>About how much time did you save? If you're no...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>SOAccount</td>\n",
       "      <td>Do you have a Stack Overflow account?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>SOPartFreq</td>\n",
       "      <td>How frequently would you say you participate i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>SOJobs</td>\n",
       "      <td>Have you ever used or visited Stack Overflow J...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>EntTeams</td>\n",
       "      <td>Have you ever used Stack Overflow for Enterpri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>SOComm</td>\n",
       "      <td>Do you consider yourself a member of the Stack...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>WelcomeChange</td>\n",
       "      <td>Compared to last year, how welcome do you feel...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>SONewContent</td>\n",
       "      <td>Would you like to see any of the following on ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>Age</td>\n",
       "      <td>What is your age (in years)? If you prefer not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>Gender</td>\n",
       "      <td>Which of the following do you currently identi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>Trans</td>\n",
       "      <td>Do you identify as transgender?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>Sexuality</td>\n",
       "      <td>Which of the following do you currently identi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>Ethnicity</td>\n",
       "      <td>Which of the following do you identify as? Ple...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>Dependents</td>\n",
       "      <td>Do you have any dependents (e.g., children, el...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>SurveyLength</td>\n",
       "      <td>How do you feel about the length of the survey...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>SurveyEase</td>\n",
       "      <td>How easy or difficult was this survey to compl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Column                                       QuestionText\n",
       "0               Respondent  Randomized respondent ID number (not in order ...\n",
       "1               MainBranch  Which of the following options best describes ...\n",
       "2                 Hobbyist                            Do you code as a hobby?\n",
       "3              OpenSourcer        How often do you contribute to open source?\n",
       "4               OpenSource  How do you feel about the quality of open sour...\n",
       "5               Employment  Which of the following best describes your cur...\n",
       "6                  Country          In which country do you currently reside?\n",
       "7                  Student  Are you currently enrolled in a formal, degree...\n",
       "8                  EdLevel  Which of the following best describes the high...\n",
       "9           UndergradMajor  What was your main or most important field of ...\n",
       "10                EduOther  Which of the following types of non-degree edu...\n",
       "11                 OrgSize  Approximately how many people are employed by ...\n",
       "12                 DevType  Which of the following describe you? Please se...\n",
       "13               YearsCode  Including any education, how many years have y...\n",
       "14              Age1stCode  At what age did you write your first line of c...\n",
       "15            YearsCodePro  How many years have you coded professionally (...\n",
       "16               CareerSat  Overall, how satisfied are you with your caree...\n",
       "17                  JobSat  How satisfied are you with your current job? (...\n",
       "18                MgrIdiot  How confident are you that your manager knows ...\n",
       "19                MgrMoney  Do you believe that you need to be a manager t...\n",
       "20                 MgrWant  Do you want to become a manager yourself in th...\n",
       "21                 JobSeek  Which of the following best describes your cur...\n",
       "22            LastHireDate  When was the last time that you took a job wit...\n",
       "23                 LastInt  In your most recent successful job interview (...\n",
       "24                FizzBuzz  Have you ever been asked to solve FizzBuzz in ...\n",
       "25              JobFactors  Imagine that you are deciding between two job ...\n",
       "26            ResumeUpdate  Think back to the last time you updated your r...\n",
       "27          CurrencySymbol  Which currency do you use day-to-day? If your ...\n",
       "28            CurrencyDesc  Which currency do you use day-to-day? If your ...\n",
       "29               CompTotal  What is your current total compensation (salar...\n",
       "30                CompFreq   Is that compensation weekly, monthly, or yearly?\n",
       "31           ConvertedComp  Salary converted to annual USD salaries using ...\n",
       "32             WorkWeekHrs   On average, how many hours per week do you work?\n",
       "33                WorkPlan            How structured or planned is your work?\n",
       "34           WorkChallenge  Of these options, what are your greatest chall...\n",
       "35              WorkRemote                    How often do you work remotely?\n",
       "36                 WorkLoc                    Where would you prefer to work?\n",
       "37                  ImpSyn  For the specific work you do, and the years of...\n",
       "38                 CodeRev           Do you review code as part of your work?\n",
       "39              CodeRevHrs  On average, how many hours per week do you spe...\n",
       "40               UnitTests  Does your company regularly employ unit tests ...\n",
       "41             PurchaseHow  How does your company make decisions about pur...\n",
       "42            PurchaseWhat  What level of influence do you, personally, ha...\n",
       "43      LanguageWorkedWith  Which of the following programming, scripting,...\n",
       "44  LanguageDesireNextYear  Which of the following programming, scripting,...\n",
       "45      DatabaseWorkedWith  Which of the following database environments h...\n",
       "46  DatabaseDesireNextYear  Which of the following database environments h...\n",
       "47      PlatformWorkedWith  Which of the following platforms have you done...\n",
       "48  PlatformDesireNextYear  Which of the following platforms have you done...\n",
       "49      WebFrameWorkedWith  Which of the following web frameworks have you...\n",
       "50  WebFrameDesireNextYear  Which of the following web frameworks have you...\n",
       "51      MiscTechWorkedWith  Which of the following other frameworks, libra...\n",
       "52  MiscTechDesireNextYear  Which of the following other frameworks, libra...\n",
       "53              DevEnviron  Which development environment(s) do you use re...\n",
       "54                   OpSys  What is the primary operating system in which ...\n",
       "55              Containers  How do you use containers (Docker, Open Contai...\n",
       "56           BlockchainOrg  How is your organization thinking about or imp...\n",
       "57            BlockchainIs  Blockchain / cryptocurrency technology is prim...\n",
       "58              BetterLife  Do you think people born today will have a bet...\n",
       "59                ITperson   Are you the \"IT support person\" for your family?\n",
       "60                   OffOn        Have you tried turning it off and on again?\n",
       "61             SocialMedia        What social media site do you use the most?\n",
       "62            Extraversion    Do you prefer online chat or IRL conversations?\n",
       "63              ScreenName                               What do you call it?\n",
       "64              SOVisit1st  To the best of your memory, when did you first...\n",
       "65             SOVisitFreq  How frequently would you say you visit Stack O...\n",
       "66               SOVisitTo  I visit Stack Overflow to... (check all that a...\n",
       "67            SOFindAnswer  On average, how many times a week do you find ...\n",
       "68             SOTimeSaved  Think back to the last time you solved a codin...\n",
       "69           SOHowMuchTime  About how much time did you save? If you're no...\n",
       "70               SOAccount              Do you have a Stack Overflow account?\n",
       "71              SOPartFreq  How frequently would you say you participate i...\n",
       "72                  SOJobs  Have you ever used or visited Stack Overflow J...\n",
       "73                EntTeams  Have you ever used Stack Overflow for Enterpri...\n",
       "74                  SOComm  Do you consider yourself a member of the Stack...\n",
       "75           WelcomeChange  Compared to last year, how welcome do you feel...\n",
       "76            SONewContent  Would you like to see any of the following on ...\n",
       "77                     Age  What is your age (in years)? If you prefer not...\n",
       "78                  Gender  Which of the following do you currently identi...\n",
       "79                   Trans                    Do you identify as transgender?\n",
       "80               Sexuality  Which of the following do you currently identi...\n",
       "81               Ethnicity  Which of the following do you identify as? Ple...\n",
       "82              Dependents  Do you have any dependents (e.g., children, el...\n",
       "83            SurveyLength  How do you feel about the length of the survey...\n",
       "84              SurveyEase  How easy or difficult was this survey to compl..."
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JaneDoe@email.com\n",
       "2   John      Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataframes are like dictionaries or tables of data but more\n",
    "\n",
    "Series is like all the rows in column of dataframe\n",
    "\n",
    "so a dataframe is like seriesa container of mutiple SERIES object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(dft.email)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can access data in dataframe like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    CoreyMSchafer@gmail.com\n",
       "1          JaneDoe@email.com\n",
       "2          JohnDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or with dot notation like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    CoreyMSchafer@gmail.com\n",
       "1          JaneDoe@email.com\n",
       "2          JohnDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.email"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Brakets is better as dot notation may have conflicting names with functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access multiple columns in dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      last                    email\n",
       "0  Schafer  CoreyMSchafer@gmail.com\n",
       "1      Doe        JaneDoe@email.com\n",
       "2      Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft[['last', 'email']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`dft[['last', 'email']]` here we gave a list of columns so [brackets [ in brakets ] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "also the result is no longer a series since we use two columns. its just another dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(dft[['last', 'email']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "can get list of all columns like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['first', 'last', 'email'], dtype='object')"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get all rows we Use\n",
    "\n",
    "<mark>- **loc**  uses labels for index</mark> _sometimes uses numbers if theres no custom labels_  \n",
    "<mark>- **iloc**  uses numbers for indes </mark>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### iloc - integer loc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first                      Corey\n",
       "last                     Schafer\n",
       "email    CoreyMSchafer@gmail.com\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "can also pass multiple index as list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JaneDoe@email.com\n",
       "2   John      Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.iloc[[0,1,2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access specific column we use a second argument like this\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Schafer\n",
       "1        Doe\n",
       "Name: last, dtype: object"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.iloc[[0,1], 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      last\n",
       "0  Schafer\n",
       "1      Doe"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.iloc[[0,1], [1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "using `[]` is giving us a _dataframe_ as result while just a number is giving us _series_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(dft.iloc[[0,1], 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(dft.iloc[[0,1], [1]])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "can give multiple list for second argument too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JaneDoe@email.com"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.iloc[[0,1], [0,1,2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## loc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "loc uses labels as index to get data, _can use numbers in case of absence of named labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    CoreyMSchafer@gmail.com\n",
       "1          JaneDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[[0,1], 'email'] #NO named labels for row so we use numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "like iloc `[]` returns *dataframe* and no [] returns _series-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email     last\n",
       "0  CoreyMSchafer@gmail.com  Schafer\n",
       "1        JaneDoe@email.com      Doe"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[[0,1], ['email', 'last']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns will be displayed in the order we use them in the list we give.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(88883, 85)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Respondent', 'MainBranch', 'Hobbyist', 'OpenSourcer', 'OpenSource',\n",
       "       'Employment', 'Country', 'Student', 'EdLevel', 'UndergradMajor',\n",
       "       'EduOther', 'OrgSize', 'DevType', 'YearsCode', 'Age1stCode',\n",
       "       'YearsCodePro', 'CareerSat', 'JobSat', 'MgrIdiot', 'MgrMoney',\n",
       "       'MgrWant', 'JobSeek', 'LastHireDate', 'LastInt', 'FizzBuzz',\n",
       "       'JobFactors', 'ResumeUpdate', 'CurrencySymbol', 'CurrencyDesc',\n",
       "       'CompTotal', 'CompFreq', 'ConvertedComp', 'WorkWeekHrs', 'WorkPlan',\n",
       "       'WorkChallenge', 'WorkRemote', 'WorkLoc', 'ImpSyn', 'CodeRev',\n",
       "       'CodeRevHrs', 'UnitTests', 'PurchaseHow', 'PurchaseWhat',\n",
       "       'LanguageWorkedWith', 'LanguageDesireNextYear', 'DatabaseWorkedWith',\n",
       "       'DatabaseDesireNextYear', 'PlatformWorkedWith',\n",
       "       'PlatformDesireNextYear', 'WebFrameWorkedWith',\n",
       "       'WebFrameDesireNextYear', 'MiscTechWorkedWith',\n",
       "       'MiscTechDesireNextYear', 'DevEnviron', 'OpSys', 'Containers',\n",
       "       'BlockchainOrg', 'BlockchainIs', 'BetterLife', 'ITperson', 'OffOn',\n",
       "       'SocialMedia', 'Extraversion', 'ScreenName', 'SOVisit1st',\n",
       "       'SOVisitFreq', 'SOVisitTo', 'SOFindAnswer', 'SOTimeSaved',\n",
       "       'SOHowMuchTime', 'SOAccount', 'SOPartFreq', 'SOJobs', 'EntTeams',\n",
       "       'SOComm', 'WelcomeChange', 'SONewContent', 'Age', 'Gender', 'Trans',\n",
       "       'Sexuality', 'Ethnicity', 'Dependents', 'SurveyLength', 'SurveyEase'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        Yes\n",
       "1         No\n",
       "2        Yes\n",
       "3         No\n",
       "4        Yes\n",
       "        ... \n",
       "88878    Yes\n",
       "88879     No\n",
       "88880     No\n",
       "88881     No\n",
       "88882    Yes\n",
       "Name: Hobbyist, Length: 88883, dtype: object"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Hobbyist']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### to get count of unique rows in an column `.value_counts()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Hobbyist\n",
       "Yes    71257\n",
       "No     17626\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Hobbyist'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hobbyist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Hobbyist\n",
       "0      Yes\n",
       "1       No\n",
       "2      Yes"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[[0, 1, 2], ['Hobbyist']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Can use `slicing`\n",
    "\n",
    "slicing is `inclusive` in ***loc*** because we use labels in index  \n",
    "but exclusive (normal python behaviour) in ***iloc***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Hobbyist              OpenSourcer  \\\n",
       "0      Yes                    Never   \n",
       "1       No  Less than once per year   \n",
       "2      Yes                    Never   \n",
       "3       No                    Never   \n",
       "\n",
       "                                          OpenSource  \\\n",
       "0  The quality of OSS and closed source software ...   \n",
       "1  The quality of OSS and closed source software ...   \n",
       "2  The quality of OSS and closed source software ...   \n",
       "3  The quality of OSS and closed source software ...   \n",
       "\n",
       "                               Employment  \n",
       "0  Not employed, and not looking for work  \n",
       "1      Not employed, but looking for work  \n",
       "2                      Employed full-time  \n",
       "3                      Employed full-time  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0:3, 'Hobbyist':'Employment'] #inclusive of EMPLOYMENT\n",
    "      # ^ no need to add [] for slicing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Pandas Tutorial (Part 3): Indexes - How to Set, Reset, and Use Indexes\n",
    "\n",
    "https://youtu.be/W9XjRYFkkyw?si=apz7oeCtj54tIAUQ\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Snippets\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Pndas doesnt enforce all index to be unique (?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "most of the changes we make in pandas are not modified in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>email</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CoreyMSchafer@gmail.com</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JaneDoe@email.com</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JohnDoe@email.com</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         first     last\n",
       "email                                  \n",
       "CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "JaneDoe@email.com         Jane      Doe\n",
       "JohnDoe@email.com         John      Doe"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.set_index('email')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JaneDoe@email.com\n",
       "2   John      Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=3, step=1)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dft.set_index('email', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "email is a good udid kinda data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['CoreyMSchafer@gmail.com', 'JaneDoe@email.com', 'JohnDoe@email.com'], dtype='object', name='email')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>email</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CoreyMSchafer@gmail.com</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JaneDoe@email.com</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JohnDoe@email.com</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         first     last\n",
       "email                                  \n",
       "CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "JaneDoe@email.com         Jane      Doe\n",
       "JohnDoe@email.com         John      Doe"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first    John\n",
       "last      Doe\n",
       "Name: JohnDoe@email.com, dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc['JohnDoe@email.com'] # dft.loc[0] will give type error now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can still access the integer index with *iloc*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first    John\n",
       "last      Doe\n",
       "Name: JohnDoe@email.com, dtype: object"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.iloc[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### To reset Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_0</th>\n",
       "      <th>index</th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   level_0  index                    email  first     last\n",
       "0        0      0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        1      1        JaneDoe@email.com   Jane      Doe\n",
       "2        2      2        JohnDoe@email.com   John      Doe"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.reset_index(inplace=True)\n",
    "dft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_0</th>\n",
       "      <th>index</th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   level_0  index                    email  first     last\n",
       "0        0      0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        1      1        JaneDoe@email.com   Jane      Doe\n",
       "2        2      2        JohnDoe@email.com   John      Doe"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Real world Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using stackoverflow data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to know what column means without searching through the dataset. can set column name as index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> can choose index while loading data like this below "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_columns', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, desktop or enterprise applications;...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I am actively looking for a job</td>\n",
       "      <td>I've never had a job</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Financial performance or funding status of the...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C++;HTML/CSS;Python</td>\n",
       "      <td>C++;HTML/CSS;JavaScript;SQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Django</td>\n",
       "      <td>Django</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Atom;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Instagram</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>No, I knew that Stack Overflow had a job board...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist  \\\n",
       "Respondent                                                    \n",
       "1           I am a student who is learning to code      Yes   \n",
       "2           I am a student who is learning to code       No   \n",
       "\n",
       "                        OpenSourcer  \\\n",
       "Respondent                            \n",
       "1                             Never   \n",
       "2           Less than once per year   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "2           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment                 Country  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work          United Kingdom   \n",
       "2               Not employed, but looking for work  Bosnia and Herzegovina   \n",
       "\n",
       "                   Student                                            EdLevel  \\\n",
       "Respondent                                                                      \n",
       "1                       No                          Primary/elementary school   \n",
       "2           Yes, full-time  Secondary school (e.g. American high school, G...   \n",
       "\n",
       "           UndergradMajor                                           EduOther  \\\n",
       "Respondent                                                                     \n",
       "1                     NaN  Taught yourself a new language, framework, or ...   \n",
       "2                     NaN  Taken an online course in programming or softw...   \n",
       "\n",
       "           OrgSize                                            DevType  \\\n",
       "Respondent                                                              \n",
       "1              NaN                                                NaN   \n",
       "2              NaN  Developer, desktop or enterprise applications;...   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "2                NaN         17          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant                          JobSeek  \\\n",
       "Respondent                                                     \n",
       "1               NaN     NaN                              NaN   \n",
       "2               NaN     NaN  I am actively looking for a job   \n",
       "\n",
       "                    LastHireDate LastInt FizzBuzz  \\\n",
       "Respondent                                          \n",
       "1                            NaN     NaN      NaN   \n",
       "2           I've never had a job     NaN      NaN   \n",
       "\n",
       "                                                   JobFactors  \\\n",
       "Respondent                                                      \n",
       "1                                                         NaN   \n",
       "2           Financial performance or funding status of the...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "1                                                         NaN            NaN   \n",
       "2           Something else changed (education, award, medi...            NaN   \n",
       "\n",
       "           CurrencyDesc  CompTotal CompFreq  ConvertedComp  WorkWeekHrs  \\\n",
       "Respondent                                                                \n",
       "1                   NaN        NaN      NaN            NaN          NaN   \n",
       "2                   NaN        NaN      NaN            NaN          NaN   \n",
       "\n",
       "           WorkPlan WorkChallenge WorkRemote WorkLoc ImpSyn CodeRev  \\\n",
       "Respondent                                                            \n",
       "1               NaN           NaN        NaN     NaN    NaN     NaN   \n",
       "2               NaN           NaN        NaN     NaN    NaN     NaN   \n",
       "\n",
       "            CodeRevHrs UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                  \n",
       "1                  NaN       NaN         NaN          NaN   \n",
       "2                  NaN       NaN         NaN          NaN   \n",
       "\n",
       "                         LanguageWorkedWith  \\\n",
       "Respondent                                    \n",
       "1           HTML/CSS;Java;JavaScript;Python   \n",
       "2                       C++;HTML/CSS;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "2                               C++;HTML/CSS;JavaScript;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "2                         NaN                  MySQL            Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "2                           Windows             Django                 Django   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "2                         NaN                    NaN   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "2                         Atom;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg                                       BlockchainIs  \\\n",
       "Respondent                                                                    \n",
       "1                    NaN                                                NaN   \n",
       "2                    NaN  Useful across many domains and could change ma...   \n",
       "\n",
       "           BetterLife                                  ITperson OffOn  \\\n",
       "Respondent                                                              \n",
       "1                 Yes  Fortunately, someone else has that title   Yes   \n",
       "2                 Yes                                       Yes   Yes   \n",
       "\n",
       "           SocialMedia Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                                  \n",
       "1              Twitter       Online   Username       2017   \n",
       "2            Instagram       Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "2                     Daily or almost daily   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "2           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "2           3-5 times per week  Stack Overflow was much faster  11-30 minutes   \n",
       "\n",
       "           SOAccount                       SOPartFreq  \\\n",
       "Respondent                                              \n",
       "1                 No                              NaN   \n",
       "2                Yes  A few times per month or weekly   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "2           No, I knew that Stack Overflow had a job board...   \n",
       "\n",
       "                                       EntTeams         SOComm  \\\n",
       "Respondent                                                       \n",
       "1           No, and I don't know what those are        Neutral   \n",
       "2           No, and I don't know what those are  Yes, somewhat   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "2           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "2           Tech articles written by other developers;Indu...  19.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "2             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  \n",
       "2           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QuestionText</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <td>Randomized respondent ID number (not in order ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MainBranch</th>\n",
       "      <td>Which of the following options best describes ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 QuestionText\n",
       "Column                                                       \n",
       "Respondent  Randomized respondent ID number (not in order ...\n",
       "MainBranch  Which of the following options best describes ..."
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now we can easily find what a column means since we have added column as index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QuestionText    Do you code as a hobby?\n",
       "Name: Hobbyist, dtype: object"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.loc[\"Hobbyist\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QuestionText    How confident are you that your manager knows ...\n",
       "Name: MgrIdiot, dtype: object"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.loc['MgrIdiot']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'How confident are you that your manager knows what they’re doing?'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.loc['MgrIdiot', 'QuestionText'] #to get full text by choosing specific columnb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But to easily search we can sort the index with `.sort_index()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QuestionText</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>What is your age (in years)? If you prefer not...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age1stCode</th>\n",
       "      <td>At what age did you write your first line of c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BetterLife</th>\n",
       "      <td>Do you think people born today will have a bet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BlockchainIs</th>\n",
       "      <td>Blockchain / cryptocurrency technology is prim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <td>How is your organization thinking about or imp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WorkPlan</th>\n",
       "      <td>How structured or planned is your work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WorkRemote</th>\n",
       "      <td>How often do you work remotely?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <td>On average, how many hours per week do you work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsCode</th>\n",
       "      <td>Including any education, how many years have y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsCodePro</th>\n",
       "      <td>How many years have you coded professionally (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>85 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    QuestionText\n",
       "Column                                                          \n",
       "Age            What is your age (in years)? If you prefer not...\n",
       "Age1stCode     At what age did you write your first line of c...\n",
       "BetterLife     Do you think people born today will have a bet...\n",
       "BlockchainIs   Blockchain / cryptocurrency technology is prim...\n",
       "BlockchainOrg  How is your organization thinking about or imp...\n",
       "...                                                          ...\n",
       "WorkPlan                 How structured or planned is your work?\n",
       "WorkRemote                       How often do you work remotely?\n",
       "WorkWeekHrs     On average, how many hours per week do you work?\n",
       "YearsCode      Including any education, how many years have y...\n",
       "YearsCodePro   How many years have you coded professionally (...\n",
       "\n",
       "[85 rows x 1 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to sort in descending order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QuestionText</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>YearsCodePro</th>\n",
       "      <td>How many years have you coded professionally (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsCode</th>\n",
       "      <td>Including any education, how many years have y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <td>On average, how many hours per week do you work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WorkRemote</th>\n",
       "      <td>How often do you work remotely?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WorkPlan</th>\n",
       "      <td>How structured or planned is your work?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <td>How is your organization thinking about or imp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BlockchainIs</th>\n",
       "      <td>Blockchain / cryptocurrency technology is prim...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BetterLife</th>\n",
       "      <td>Do you think people born today will have a bet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age1stCode</th>\n",
       "      <td>At what age did you write your first line of c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>What is your age (in years)? If you prefer not...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>85 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    QuestionText\n",
       "Column                                                          \n",
       "YearsCodePro   How many years have you coded professionally (...\n",
       "YearsCode      Including any education, how many years have y...\n",
       "WorkWeekHrs     On average, how many hours per week do you work?\n",
       "WorkRemote                       How often do you work remotely?\n",
       "WorkPlan                 How structured or planned is your work?\n",
       "...                                                          ...\n",
       "BlockchainOrg  How is your organization thinking about or imp...\n",
       "BlockchainIs   Blockchain / cryptocurrency technology is prim...\n",
       "BetterLife     Do you think people born today will have a bet...\n",
       "Age1stCode     At what age did you write your first line of c...\n",
       "Age            What is your age (in years)? If you prefer not...\n",
       "\n",
       "[85 rows x 1 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# schema_df.sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 04 - Filtering - Using Conditionals to Filter Rows and Columns \n",
    "\n",
    "https://youtu.be/Lw2rlcxScZY?si=GV9Pm1IzeNSndh-p\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com']\n",
    "}\n",
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get everyone with last name Doe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we get only a `series` with yes or no"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "Name: last, dtype: bool"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['last'] == 'Doe'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so we are adding it to a variable for clarity and then later we can use it to filter and get the data we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "filt = (dft['last'] == 'Doe') # () added for clarity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "Name: last, dtype: bool"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can put it directly in the brackets for dataframe  but for clarity we are adding it variable and use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first last              email\n",
       "1  Jane  Doe  JaneDoe@email.com\n",
       "2  John  Doe  JohnDoe@email.com"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft[filt]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but its better to use `.loc` since we can also filter using column if needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    JaneDoe@email.com\n",
       "2    JohnDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[filt, 'email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### AND OR Operator in pandas\n",
    "> AND = `&`  \n",
    "> OR  = `|`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "filt2 = (dft['last'] == 'Doe') & (dft['first'] == 'John')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    JohnDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[filt2, 'email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to negate the filter we use `~` (kinda like NOT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    CoreyMSchafer@gmail.com\n",
       "1          JaneDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[~filt2, 'email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Survey Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist OpenSourcer  \\\n",
       "Respondent                                                                \n",
       "1           I am a student who is learning to code      Yes       Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment         Country Student  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work  United Kingdom      No   \n",
       "\n",
       "                              EdLevel UndergradMajor  \\\n",
       "Respondent                                             \n",
       "1           Primary/elementary school            NaN   \n",
       "\n",
       "                                                     EduOther OrgSize DevType  \\\n",
       "Respondent                                                                      \n",
       "1           Taught yourself a new language, framework, or ...     NaN     NaN   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant JobSeek LastHireDate LastInt FizzBuzz JobFactors  \\\n",
       "Respondent                                                                     \n",
       "1               NaN     NaN     NaN          NaN     NaN      NaN        NaN   \n",
       "\n",
       "           ResumeUpdate CurrencySymbol CurrencyDesc  CompTotal CompFreq  \\\n",
       "Respondent                                                                \n",
       "1                   NaN            NaN          NaN        NaN      NaN   \n",
       "\n",
       "            ConvertedComp  WorkWeekHrs WorkPlan WorkChallenge WorkRemote  \\\n",
       "Respondent                                                                 \n",
       "1                     NaN          NaN      NaN           NaN        NaN   \n",
       "\n",
       "           WorkLoc ImpSyn CodeRev  CodeRevHrs UnitTests PurchaseHow  \\\n",
       "Respondent                                                            \n",
       "1              NaN    NaN     NaN         NaN       NaN         NaN   \n",
       "\n",
       "           PurchaseWhat               LanguageWorkedWith  \\\n",
       "Respondent                                                 \n",
       "1                   NaN  HTML/CSS;Java;JavaScript;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg BlockchainIs BetterLife  \\\n",
       "Respondent                                         \n",
       "1                    NaN          NaN        Yes   \n",
       "\n",
       "                                            ITperson OffOn SocialMedia  \\\n",
       "Respondent                                                               \n",
       "1           Fortunately, someone else has that title   Yes     Twitter   \n",
       "\n",
       "           Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                      \n",
       "1                Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "\n",
       "           SOAccount SOPartFreq  \\\n",
       "Respondent                        \n",
       "1                 No        NaN   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "\n",
       "                                       EntTeams   SOComm  \\\n",
       "Respondent                                                 \n",
       "1           No, and I don't know what those are  Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QuestionText</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <td>Randomized respondent ID number (not in order ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MainBranch</th>\n",
       "      <td>Which of the following options best describes ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hobbyist</th>\n",
       "      <td>Do you code as a hobby?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OpenSourcer</th>\n",
       "      <td>How often do you contribute to open source?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OpenSource</th>\n",
       "      <td>How do you feel about the quality of open sour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sexuality</th>\n",
       "      <td>Which of the following do you currently identi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ethnicity</th>\n",
       "      <td>Which of the following do you identify as? Ple...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dependents</th>\n",
       "      <td>Do you have any dependents (e.g., children, el...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SurveyLength</th>\n",
       "      <td>How do you feel about the length of the survey...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SurveyEase</th>\n",
       "      <td>How easy or difficult was this survey to compl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>85 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   QuestionText\n",
       "Column                                                         \n",
       "Respondent    Randomized respondent ID number (not in order ...\n",
       "MainBranch    Which of the following options best describes ...\n",
       "Hobbyist                                Do you code as a hobby?\n",
       "OpenSourcer         How often do you contribute to open source?\n",
       "OpenSource    How do you feel about the quality of open sour...\n",
       "...                                                         ...\n",
       "Sexuality     Which of the following do you currently identi...\n",
       "Ethnicity     Which of the following do you identify as? Ple...\n",
       "Dependents    Do you have any dependents (e.g., children, el...\n",
       "SurveyLength  How do you feel about the length of the survey...\n",
       "SurveyEase    How easy or difficult was this survey to compl...\n",
       "\n",
       "[85 rows x 1 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To filter high salary and know what languages they know"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "filt_highsal = (df['ConvertedComp'] > 70000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Canada</td>\n",
       "      <td>Java;R;SQL</td>\n",
       "      <td>366420.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>New Zealand</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;P...</td>\n",
       "      <td>95179.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;HTML/CSS;JavaScript;PHP;...</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;T...</td>\n",
       "      <td>455352.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript;...</td>\n",
       "      <td>103000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88876</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;HTML/CSS;Java;Python;...</td>\n",
       "      <td>180000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88877</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;C;Clojure;HTML/CSS;Java;...</td>\n",
       "      <td>2000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88878</th>\n",
       "      <td>United States</td>\n",
       "      <td>HTML/CSS;JavaScript;Scala;TypeScript</td>\n",
       "      <td>130000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88879</th>\n",
       "      <td>Finland</td>\n",
       "      <td>Bash/Shell/PowerShell;C++;Python</td>\n",
       "      <td>82488.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88882</th>\n",
       "      <td>Netherlands</td>\n",
       "      <td>C#;HTML/CSS;Java;JavaScript;PHP;Python</td>\n",
       "      <td>588012.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>22289 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Country                                 LanguageWorkedWith  \\\n",
       "Respondent                                                                      \n",
       "6                   Canada                                         Java;R;SQL   \n",
       "9              New Zealand  Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;P...   \n",
       "13           United States  Bash/Shell/PowerShell;HTML/CSS;JavaScript;PHP;...   \n",
       "16          United Kingdom  Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;T...   \n",
       "22           United States  Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript;...   \n",
       "...                    ...                                                ...   \n",
       "88876        United States  Bash/Shell/PowerShell;C#;HTML/CSS;Java;Python;...   \n",
       "88877        United States  Bash/Shell/PowerShell;C;Clojure;HTML/CSS;Java;...   \n",
       "88878        United States               HTML/CSS;JavaScript;Scala;TypeScript   \n",
       "88879              Finland                   Bash/Shell/PowerShell;C++;Python   \n",
       "88882          Netherlands             C#;HTML/CSS;Java;JavaScript;PHP;Python   \n",
       "\n",
       "            ConvertedComp  \n",
       "Respondent                 \n",
       "6                366420.0  \n",
       "9                 95179.0  \n",
       "13                90000.0  \n",
       "16               455352.0  \n",
       "22               103000.0  \n",
       "...                   ...  \n",
       "88876            180000.0  \n",
       "88877           2000000.0  \n",
       "88878            130000.0  \n",
       "88879             82488.0  \n",
       "88882            588012.0  \n",
       "\n",
       "[22289 rows x 3 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[filt_highsal, ['Country', 'LanguageWorkedWith', 'ConvertedComp']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries = ['India', 'Germany']\n",
    "country_filt = df['Country'].isin(countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>13293.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>C++</td>\n",
       "      <td>57060.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Java;Kotlin;Python</td>\n",
       "      <td>20628.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80319</th>\n",
       "      <td>Bash/Shell/PowerShell;HTML/CSS;Java;JavaScript...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83786</th>\n",
       "      <td>HTML/CSS;JavaScript;Kotlin;TypeScript</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83862</th>\n",
       "      <td>Assembly;C;C++;HTML/CSS;Java;JavaScript;Object...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84299</th>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python;Swift;TypeScript</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86012</th>\n",
       "      <td>Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14927 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           LanguageWorkedWith  ConvertedComp\n",
       "Respondent                                                                  \n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...            NaN\n",
       "10                              C#;Go;JavaScript;Python;R;SQL        13293.0\n",
       "14                                                        C++        57060.0\n",
       "15          Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...            NaN\n",
       "36                                         Java;Kotlin;Python        20628.0\n",
       "...                                                       ...            ...\n",
       "80319       Bash/Shell/PowerShell;HTML/CSS;Java;JavaScript...            NaN\n",
       "83786                   HTML/CSS;JavaScript;Kotlin;TypeScript            NaN\n",
       "83862       Assembly;C;C++;HTML/CSS;Java;JavaScript;Object...            NaN\n",
       "84299        HTML/CSS;Java;JavaScript;Python;Swift;TypeScript            NaN\n",
       "86012           Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript            NaN\n",
       "\n",
       "[14927 rows x 2 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df[country_filt, ['LanguageWorkedWith', 'ConvertedComp'] ] this gives error\n",
    "df.loc[country_filt, ['LanguageWorkedWith', 'ConvertedComp'] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### String Methods Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "language filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang_filt = df['LanguageWorkedWith'].str.contains('Python', na=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use `.str.contains()` and give `na=False` for those columns that contains no values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C++;HTML/CSS;Python</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C;C++;C#;Python;SQL</td>\n",
       "      <td>61000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>C++;HTML/CSS;Java;JavaScript;Python;SQL;VBA</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84539</th>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85738</th>\n",
       "      <td>Bash/Shell/PowerShell;C++;Python;Ruby;Other(s):</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86566</th>\n",
       "      <td>Bash/Shell/PowerShell;HTML/CSS;Python;Other(s):</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87739</th>\n",
       "      <td>C;C++;HTML/CSS;JavaScript;PHP;Python;SQL</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88212</th>\n",
       "      <td>HTML/CSS;JavaScript;Python</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>36443 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           LanguageWorkedWith  ConvertedComp\n",
       "Respondent                                                                  \n",
       "1                             HTML/CSS;Java;JavaScript;Python            NaN\n",
       "2                                         C++;HTML/CSS;Python            NaN\n",
       "4                                         C;C++;C#;Python;SQL        61000.0\n",
       "5                 C++;HTML/CSS;Java;JavaScript;Python;SQL;VBA            NaN\n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...            NaN\n",
       "...                                                       ...            ...\n",
       "84539       Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...            NaN\n",
       "85738         Bash/Shell/PowerShell;C++;Python;Ruby;Other(s):            NaN\n",
       "86566         Bash/Shell/PowerShell;HTML/CSS;Python;Other(s):            NaN\n",
       "87739                C;C++;HTML/CSS;JavaScript;PHP;Python;SQL            NaN\n",
       "88212                              HTML/CSS;JavaScript;Python            NaN\n",
       "\n",
       "[36443 rows x 2 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[lang_filt, ['LanguageWorkedWith', 'ConvertedComp']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 05 - Updating Rows and Columns - Modifying Data Within DataFrames\n",
    "\n",
    "https://youtu.be/DCDe29sIKcE?si=d4DUqMqXt7kSc3o-\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#Setup for learning data\n",
    "\n",
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com']\n",
    "}\n",
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Setup for Real world data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Update columns/column names\n",
    "\n",
    "\n",
    "How to update data within rows and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['first', 'last', 'email'], dtype='object')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f</th>\n",
       "      <th>l</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       f        l                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JaneDoe@email.com\n",
       "2   John      Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.columns = [ 'f', 'l', 'email']\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Renames all the column in the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dft.columns = [ 'f', 'l']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But that method is only used when renaming all the columns in the dataframe and not when some of the columns are left\n",
    "\n",
    "to rename specific column have to give a Dictionary (key:value pairs) like 'Old name' : 'new name'  to `.rename(columns = {})` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first name</th>\n",
       "      <th>last name</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first name last name                    email\n",
       "0      Corey   Schafer  CoreyMSchafer@gmail.com\n",
       "1       Jane       Doe        JaneDoe@email.com\n",
       "2       John       Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.rename( columns = {'f': 'first name', 'l': 'last name'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but the above wont change data in place so we have to use `inplace = True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first name</th>\n",
       "      <th>last name</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first name last name                    email\n",
       "0      Corey   Schafer  CoreyMSchafer@gmail.com\n",
       "1       Jane       Doe        JaneDoe@email.com\n",
       "2       John       Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.rename( columns = {'f': 'first name', 'l': 'last name'}, inplace=True)\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "can use string methods to uppercase the column using list comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FIRST NAME</th>\n",
       "      <th>LAST NAME</th>\n",
       "      <th>EMAIL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  FIRST NAME LAST NAME                    EMAIL\n",
       "0      Corey   Schafer  CoreyMSchafer@gmail.com\n",
       "1       Jane       Doe        JaneDoe@email.com\n",
       "2       John       Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.columns = [ x.upper() for x in dft.columns ]\n",
    "dft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first name</th>\n",
       "      <th>last name</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first name last name                    email\n",
       "0      Corey   Schafer  CoreyMSchafer@gmail.com\n",
       "1       Jane       Doe        JaneDoe@email.com\n",
       "2       John       Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.columns = dft.columns.str.lower() #can also use this method\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or remove the spaces in between column names for ease of use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first_name</th>\n",
       "      <th>last_name</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first_name last_name                    email\n",
       "0      Corey   Schafer  CoreyMSchafer@gmail.com\n",
       "1       Jane       Doe        JaneDoe@email.com\n",
       "2       John       Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.columns = dft.columns.str.replace( ' ', '_')\n",
    "dft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "dft.columns = ['first', 'last', 'email'] #Back to original template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Update data in rows\n",
    "###### [Timestamp](https://youtu.be/DCDe29sIKcE?si=9Xbty9dqhF9it79K)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "can change all values in dataframe like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Smith</td>\n",
       "      <td>JohnSMith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JaneDoe@email.com\n",
       "2   John    Smith      JohnSMith@email.com"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[2] = ['John', 'Smith', 'JohnSMith@email.com']\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we only want to change couple of columns/values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Smith</td>\n",
       "      <td>JohnSMith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JohnDoe@email.com\n",
       "2   John    Smith      JohnSMith@email.com"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[1, ['last', 'email']] = ['Doe', 'JohnDoe@email.com']\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas has indexer called `.at` which is meant specifically for changing single value  \n",
    "###### not any difference or performance gains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnSMith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JohnDoe@email.com\n",
       "2   John      Doe      JohnSMith@email.com"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.at[2, 'last'] = 'Doe'\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>Errors when</mark> Indexers like `.loc` or `.at` is <mark>NOT</mark> used\n",
    "###### [Timestamp](https://youtu.be/DCDe29sIKcE?si=KKqz5VgyaYQDaOui&t=722)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first last              email\n",
       "1  Jane  Doe  JohnDoe@email.com"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#lets create a filter\n",
    "filt = dft['email'] == 'JohnDoe@email.com'\n",
    "dft[filt]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/0n/pm6lp6w14r92cpm712ssqf1c0000gn/T/ipykernel_35969/3772588233.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  dft[filt]['last'] = 'Smith' #if we try to assign like this\n"
     ]
    }
   ],
   "source": [
    "dft[filt]['last'] = 'Smith' #if we try to assign like this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get a warning like above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnSMith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane      Doe        JohnDoe@email.com\n",
       "2   John      Doe      JohnSMith@email.com"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the name is not actually changed. \n",
    "\n",
    "##### SO Always use `.loc` or `.at` when setting values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Smith</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnSMith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com\n",
       "1   Jane    Smith        JohnDoe@email.com\n",
       "2   John      Doe      JohnSMith@email.com"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[filt, 'last'] = 'Smith'\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Update Multiple rows of data \n",
    "###### [Timestamp](https://youtu.be/DCDe29sIKcE?si=CwAJCgjQhG_i8hLo&t=935)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>coreymschafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Smith</td>\n",
       "      <td>johndoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>johnsmith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  coreymschafer@gmail.com\n",
       "1   Jane    Smith        johndoe@email.com\n",
       "2   John      Doe      johnsmith@email.com"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['email'] = dft['email'].str.lower() #one way to change mutiple rows (to lowercase)\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### There are 4 ways to change multiple rows at once\n",
    "#####  1) Apply\n",
    "            Applies on Series if applied on DataFrame, Applies on values if applied on Series\n",
    "#####  2) ApplyMap\n",
    "            Applies to every value in a dataframe.  \n",
    "            * Only works on DataFrame * \n",
    "<mark>FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.</mark>\n",
    "#####  3) Map\n",
    "\n",
    "#####  4) Replace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) APPLY\n",
    "\n",
    "used for calling a function on value. can work on Datafram or Series object.  \n",
    "but behaviour is different based on the type\n",
    "\n",
    "##### APPLY on SERIES\n",
    "\n",
    "can apply a function to every value in the series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    23\n",
       "1    17\n",
       "2    19\n",
       "Name: email, dtype: int64"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['email'].apply(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_upper(email):\n",
    "    return email.upper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    COREYMSCHAFER@GMAIL.COM\n",
       "1          JOHNDOE@EMAIL.COM\n",
       "2        JOHNSMITH@EMAIL.COM\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['email'].apply(to_upper) \n",
    "#wont change in place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`to_upper` and not `to_upper()` since we are just <mark>passing</mark> it a function (as argument) and <mark> not calling</mark>  the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>COREYMSCHAFER@GMAIL.COM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Smith</td>\n",
       "      <td>JOHNDOE@EMAIL.COM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JOHNSMITH@EMAIL.COM</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  COREYMSCHAFER@GMAIL.COM\n",
       "1   Jane    Smith        JOHNDOE@EMAIL.COM\n",
       "2   John      Doe      JOHNSMITH@EMAIL.COM"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['email'] = dft['email'].apply(to_upper) # Changes in place\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use `lambda` functions in that place if we want like below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>coreymschafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Smith</td>\n",
       "      <td>johndoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>johnsmith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Corey  Schafer  coreymschafer@gmail.com\n",
       "1   Jane    Smith        johndoe@email.com\n",
       "2   John      Doe      johnsmith@email.com"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['email'] = dft['email'].apply(lambda x: x.lower())\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### APPLY on DATAFRAMES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first    3\n",
       "last     3\n",
       "email    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dft['email'].apply(len) # --- This returns a series so it works\n",
    "\n",
    "dft.apply(len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "instead of applying values to data in dataframes, it applies to the series instead so `len` returns the lenght of the series here "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dft['last']) # Like this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we want to apply on columns we can change axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    3\n",
       "1    3\n",
       "2    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.apply(len, axis = 'columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use functions that can be applied on series object to applying them on dataframe  \n",
    "like finding minimum or maximum values. Ex: `min` method of series object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first                      Corey\n",
       "last                         Doe\n",
       "email    coreymschafer@gmail.com\n",
       "dtype: object"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.apply(pd.Series.min) #WORKS BETTER WITH NUMERICAL DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can also use `lambda` functions but it applies on series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first                      Corey\n",
       "last                         Doe\n",
       "email    coreymschafer@gmail.com\n",
       "dtype: object"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.apply(lambda x: x.min()) #same result as above, applies the function on whole series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running apply to a Series applies function to every value in a Series\n",
    "#### Running apply to a DataFrame applies function to every Series in a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Apply Map\n",
    "---\n",
    "\n",
    "Applies function to every value in DataFrame\n",
    "###### Instead of every Series in DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/0n/pm6lp6w14r92cpm712ssqf1c0000gn/T/ipykernel_35969/4052204359.py:1: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
      "  dft.applymap(len)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first  last  email\n",
       "0      5     7     23\n",
       "1      4     5     17\n",
       "2      4     3     19"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.applymap(len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first  last  email\n",
       "0      5     7     23\n",
       "1      4     5     17\n",
       "2      4     3     19"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.map(len) #not part of the YT video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/0n/pm6lp6w14r92cpm712ssqf1c0000gn/T/ipykernel_35969/4206962566.py:1: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n",
      "  dft.applymap(str.lower)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>corey</td>\n",
       "      <td>schafer</td>\n",
       "      <td>coreymschafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jane</td>\n",
       "      <td>smith</td>\n",
       "      <td>johndoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>john</td>\n",
       "      <td>doe</td>\n",
       "      <td>johnsmith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  corey  schafer  coreymschafer@gmail.com\n",
       "1   jane    smith        johndoe@email.com\n",
       "2   john      doe      johnsmith@email.com"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.applymap(str.lower)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3) MAP\n",
    "---\n",
    "\n",
    "It only works on Series <mark>(?)</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Chris\n",
       "1     Mary\n",
       "2      NaN\n",
       "Name: first, dtype: object"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first'].map({'Corey': 'Chris', 'Jane': 'Mary'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values that we dont substitute will change to `NaN` values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4) Replace\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but we can use `replace()` to keep unassigned values intect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Chris\n",
       "1     Mary\n",
       "2     John\n",
       "Name: first, dtype: object"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first'].replace({'Corey': 'Chris', 'Jane': 'Mary'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>coreymschafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mary</td>\n",
       "      <td>Smith</td>\n",
       "      <td>johndoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>johnsmith@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email\n",
       "0  Chris  Schafer  coreymschafer@gmail.com\n",
       "1   Mary    Smith        johndoe@email.com\n",
       "2   John      Doe      johnsmith@email.com"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first'] = dft['first'].replace({'Corey': 'Chris', 'Jane': 'Mary'}) #to apply changes\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "#### SO Example \n",
    "###### [Timestamp](https://youtu.be/DCDe29sIKcE?si=CNkM04w4H5l2Jwfb&t=1936)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rename column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist OpenSourcer  \\\n",
       "Respondent                                                                \n",
       "1           I am a student who is learning to code      Yes       Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment         Country Student  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work  United Kingdom      No   \n",
       "\n",
       "                              EdLevel UndergradMajor  \\\n",
       "Respondent                                             \n",
       "1           Primary/elementary school            NaN   \n",
       "\n",
       "                                                     EduOther OrgSize DevType  \\\n",
       "Respondent                                                                      \n",
       "1           Taught yourself a new language, framework, or ...     NaN     NaN   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant JobSeek LastHireDate LastInt FizzBuzz JobFactors  \\\n",
       "Respondent                                                                     \n",
       "1               NaN     NaN     NaN          NaN     NaN      NaN        NaN   \n",
       "\n",
       "           ResumeUpdate CurrencySymbol CurrencyDesc  CompTotal CompFreq  \\\n",
       "Respondent                                                                \n",
       "1                   NaN            NaN          NaN        NaN      NaN   \n",
       "\n",
       "            ConvertedComp  WorkWeekHrs WorkPlan WorkChallenge WorkRemote  \\\n",
       "Respondent                                                                 \n",
       "1                     NaN          NaN      NaN           NaN        NaN   \n",
       "\n",
       "           WorkLoc ImpSyn CodeRev  CodeRevHrs UnitTests PurchaseHow  \\\n",
       "Respondent                                                            \n",
       "1              NaN    NaN     NaN         NaN       NaN         NaN   \n",
       "\n",
       "           PurchaseWhat               LanguageWorkedWith  \\\n",
       "Respondent                                                 \n",
       "1                   NaN  HTML/CSS;Java;JavaScript;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg BlockchainIs BetterLife  \\\n",
       "Respondent                                         \n",
       "1                    NaN          NaN        Yes   \n",
       "\n",
       "                                            ITperson OffOn SocialMedia  \\\n",
       "Respondent                                                               \n",
       "1           Fortunately, someone else has that title   Yes     Twitter   \n",
       "\n",
       "           Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                      \n",
       "1                Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "\n",
       "           SOAccount SOPartFreq  \\\n",
       "Respondent                        \n",
       "1                 No        NaN   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "\n",
       "                                       EntTeams   SOComm  \\\n",
       "Respondent                                                 \n",
       "1           No, and I don't know what those are  Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.rename(columns= {'ConvertedComp': 'SalaryUSD'})\n",
    "df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the advantage of `inplace=True` is we can apply only after we see the result and not mess with our original data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>SalaryUSD</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist OpenSourcer  \\\n",
       "Respondent                                                                \n",
       "1           I am a student who is learning to code      Yes       Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment         Country Student  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work  United Kingdom      No   \n",
       "\n",
       "                              EdLevel UndergradMajor  \\\n",
       "Respondent                                             \n",
       "1           Primary/elementary school            NaN   \n",
       "\n",
       "                                                     EduOther OrgSize DevType  \\\n",
       "Respondent                                                                      \n",
       "1           Taught yourself a new language, framework, or ...     NaN     NaN   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant JobSeek LastHireDate LastInt FizzBuzz JobFactors  \\\n",
       "Respondent                                                                     \n",
       "1               NaN     NaN     NaN          NaN     NaN      NaN        NaN   \n",
       "\n",
       "           ResumeUpdate CurrencySymbol CurrencyDesc  CompTotal CompFreq  \\\n",
       "Respondent                                                                \n",
       "1                   NaN            NaN          NaN        NaN      NaN   \n",
       "\n",
       "            SalaryUSD  WorkWeekHrs WorkPlan WorkChallenge WorkRemote WorkLoc  \\\n",
       "Respondent                                                                     \n",
       "1                 NaN          NaN      NaN           NaN        NaN     NaN   \n",
       "\n",
       "           ImpSyn CodeRev  CodeRevHrs UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                                 \n",
       "1             NaN     NaN         NaN       NaN         NaN          NaN   \n",
       "\n",
       "                         LanguageWorkedWith  \\\n",
       "Respondent                                    \n",
       "1           HTML/CSS;Java;JavaScript;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg BlockchainIs BetterLife  \\\n",
       "Respondent                                         \n",
       "1                    NaN          NaN        Yes   \n",
       "\n",
       "                                            ITperson OffOn SocialMedia  \\\n",
       "Respondent                                                               \n",
       "1           Fortunately, someone else has that title   Yes     Twitter   \n",
       "\n",
       "           Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                      \n",
       "1                Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "\n",
       "           SOAccount SOPartFreq  \\\n",
       "Respondent                        \n",
       "1                 No        NaN   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "\n",
       "                                       EntTeams   SOComm  \\\n",
       "Respondent                                                 \n",
       "1           No, and I don't know what those are  Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.rename(columns= {'ConvertedComp': 'SalaryUSD'}, inplace=True)\n",
    "df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use `map` for columns with Yes/No values to change it into `True` or `False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Hobbyist']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "1         True\n",
       "2        False\n",
       "3         True\n",
       "4        False\n",
       "5         True\n",
       "         ...  \n",
       "88377     True\n",
       "88601    False\n",
       "88802    False\n",
       "88816    False\n",
       "88863     True\n",
       "Name: Hobbyist, Length: 88883, dtype: object"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Hobbyist'].map({'Yes': 'True', 'No': 'False'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Theres no `implace` for Map so we have to set that Series = Mapped version of that series\n",
    "\n",
    "! Map replaces the empty values with `NaN`\n",
    "\n",
    "so we can use `replace` in case we dont want to do that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 06 - Add/Remove Rows and Columns From DataFrames\n",
    "\n",
    "https://youtu.be/HQ6XO9eT-fc?si=6STRvIlEn1b-zwp8\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#Setup for learning data\n",
    "\n",
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com']\n",
    "}\n",
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup for Real world data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 284,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "# schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "# pd.set_option('display.max_columns', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 285,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding and removing columns\n",
    "\n",
    "Combining two columns into one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Corey Schafer\n",
       "1         Jane Doe\n",
       "2         John Doe\n",
       "dtype: object"
      ]
     },
     "execution_count": 286,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first'] + ' ' + dft['last']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " can assign and create a new column like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email     first_name\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com  Corey Schafer\n",
       "1   Jane      Doe        JaneDoe@email.com       Jane Doe\n",
       "2   John      Doe        JohnDoe@email.com       John Doe"
      ]
     },
     "execution_count": 287,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first_name'] = dft['first'] + ' ' + dft['last']\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to <mark>remove</mark> a column  \n",
    "we have to use the `drop()` method on DataFrame  \n",
    "###### [TimeStamp](https://youtu.be/HQ6XO9eT-fc?si=n4XTkr6gsG4xjlN9&t=184)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 288,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email     first_name\n",
       "0  CoreyMSchafer@gmail.com  Corey Schafer\n",
       "1        JaneDoe@email.com       Jane Doe\n",
       "2        JohnDoe@email.com       John Doe"
      ]
     },
     "execution_count": 288,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.drop(columns= ['first', 'last'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 289,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email     first_name\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com  Corey Schafer\n",
       "1   Jane      Doe        JaneDoe@email.com       Jane Doe\n",
       "2   John      Doe        JohnDoe@email.com       John Doe"
      ]
     },
     "execution_count": 289,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to make changes in place have to use  `inplace=True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 290,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email     first_name\n",
       "0  CoreyMSchafer@gmail.com  Corey Schafer\n",
       "1        JaneDoe@email.com       Jane Doe\n",
       "2        JohnDoe@email.com       John Doe"
      ]
     },
     "execution_count": 290,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.drop(columns= ['first', 'last'], inplace=True)\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "to <mark> create a new column from existing columns</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [Corey, Schafer]\n",
       "1         [Jane, Doe]\n",
       "2         [John, Doe]\n",
       "Name: first_name, dtype: object"
      ]
     },
     "execution_count": 291,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first_name'].str.split(' ')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we get first and last name in a list above. to assign this to two different columns we have to  expand `expand=True` this into two different columns like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0        1\n",
       "0  Corey  Schafer\n",
       "1   Jane      Doe\n",
       "2   John      Doe"
      ]
     },
     "execution_count": 292,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['first_name'].str.split(' ', expand=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email     first_name\n",
       "0  CoreyMSchafer@gmail.com  Corey Schafer\n",
       "1        JaneDoe@email.com       Jane Doe\n",
       "2        JohnDoe@email.com       John Doe"
      ]
     },
     "execution_count": 293,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email     first_name\n",
       "0  CoreyMSchafer@gmail.com  Corey Schafer\n",
       "1        JaneDoe@email.com       Jane Doe\n",
       "2        JohnDoe@email.com       John Doe"
      ]
     },
     "execution_count": 294,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft[['email', 'first_name']] # we have to use double brackets to access dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so we have to us `[]` to create a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first_name</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey Schafer</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane Doe</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John Doe</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email     first_name  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey Schafer  Corey  Schafer\n",
       "1        JaneDoe@email.com       Jane Doe   Jane      Doe\n",
       "2        JohnDoe@email.com       John Doe   John      Doe"
      ]
     },
     "execution_count": 295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft[['first', 'last']] = dft['first_name'].str.split(' ', expand=True)\n",
    "dft#^               ^"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dft.columns\n",
    "# column_name = ('first', 'last')\n",
    "# dft.drop(column_name, axis=1, inplace=True)\n",
    "# dft\n",
    "# had an error with accidentally setting up column title as tuple so had to use the code above to remove\n",
    "\n",
    "###### Result #####\n",
    "#Index(['email', 'first_name', ('first', 'last'), 'first', 'last'], dtype='object')\n",
    "#                                column name is a tuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [],
   "source": [
    "dft.drop(columns= ['first_name'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Adding or removing data\n",
    "1) Add Single row to data\n",
    "2) combine two DataFrame\n",
    "\n",
    "TO add single data to DataFrame we do that with `._append()` method   \n",
    "<mark> `.append()` is deprecated </mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Can only append a dict if ignore_index=True",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/0n/pm6lp6w14r92cpm712ssqf1c0000gn/T/ipykernel_65619/443968578.py\u001b[0m in \u001b[0;36m?\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdft\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_append\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m'first'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Tata'\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/Developer/Python/Practice/pandas/.venv/lib/python3.11/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m?\u001b[0;34m(self, other, ignore_index, verify_integrity, sort)\u001b[0m\n\u001b[1;32m  10188\u001b[0m     ) -> DataFrame:\n\u001b[1;32m  10189\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mSeries\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m  10190\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m  10191\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m> 10192\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Can only append a dict if ignore_index=True\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m  10193\u001b[0m                 \u001b[0mother\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mSeries\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mother\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m  10194\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mother\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mignore_index\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m  10195\u001b[0m                 raise TypeError(\n",
      "\u001b[0;31mTypeError\u001b[0m: Can only append a dict if ignore_index=True"
     ]
    }
   ],
   "source": [
    "dft._append({'first': 'Tata'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It gives error because we dont have index in the DataFrame  \n",
    "we can fix this with `ignoreindex=True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Tata</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        JaneDoe@email.com   Jane      Doe\n",
       "2        JohnDoe@email.com   John      Doe\n",
       "3                      NaN   Tata      NaN"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft._append({'first': 'Tata'}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "great_people = {\n",
    "    \"first\": [\"Ratan\", 'Abdul'], \n",
    "    \"last\": [\"Tata\", 'Kalam'], \n",
    "    \"email\": [\"ratan@tata.com\", 'AbdulKalam@isro.com']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>ratan@tata.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Abdul</td>\n",
       "      <td>Kalam</td>\n",
       "      <td>AbdulKalam@isro.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first   last                email\n",
       "0  Ratan   Tata       ratan@tata.com\n",
       "1  Abdul  Kalam  AbdulKalam@isro.com"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfp = pd.DataFrame(great_people)\n",
    "dfp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now this is a DataFrame we can combine this with existing DF  \n",
    "they have  conflicting indexes  \n",
    "and columns not in same order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ratan@tata.com</td>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AbdulKalam@isro.com</td>\n",
       "      <td>Abdul</td>\n",
       "      <td>Kalam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        JaneDoe@email.com   Jane      Doe\n",
       "2        JohnDoe@email.com   John      Doe\n",
       "0           ratan@tata.com  Ratan     Tata\n",
       "1      AbdulKalam@isro.com  Abdul    Kalam"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft._append(dfp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this may give error sometimes. i give error in video because its old pandas where it can be fixed by `ignoreindex=True` and `sort = False` (sort is False by default now)\n",
    "![Alt text](06-image-01.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ratan@tata.com</td>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AbdulKalam@isro.com</td>\n",
       "      <td>Abdul</td>\n",
       "      <td>Kalam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        JaneDoe@email.com   Jane      Doe\n",
       "2        JohnDoe@email.com   John      Doe\n",
       "0           ratan@tata.com  Ratan     Tata\n",
       "1      AbdulKalam@isro.com  Abdul    Kalam"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the data is not added to the DF  \n",
    "but we have to no `inplace=True` here like drop  \n",
    "so we have to assign it\n",
    "\n",
    "we do have to use `ignoreindex=True` else pandas keep both the index and we have conflicting index like above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ratan@tata.com</td>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AbdulKalam@isro.com</td>\n",
       "      <td>Abdul</td>\n",
       "      <td>Kalam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        JaneDoe@email.com   Jane      Doe\n",
       "2        JohnDoe@email.com   John      Doe\n",
       "3           ratan@tata.com  Ratan     Tata\n",
       "4      AbdulKalam@isro.com  Abdul    Kalam"
      ]
     },
     "execution_count": 264,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft = dft._append(dfp, ignore_index=True)\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now the index is from 0 - 4 instead of 0, 1, 2, 0, 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we want to <mark>drop a single row</mark>, we can use index of the row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AbdulKalam@isro.com</td>\n",
       "      <td>Abdul</td>\n",
       "      <td>Kalam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "1        JaneDoe@email.com   Jane      Doe\n",
       "2        JohnDoe@email.com   John      Doe\n",
       "4      AbdulKalam@isro.com  Abdul    Kalam"
      ]
     },
     "execution_count": 265,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.drop(index= 3, inplace=True) # have to change in place here\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can also use filter to delete rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "4    False\n",
       "Name: last, dtype: bool"
      ]
     },
     "execution_count": 281,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['last'] == 'Doe'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               email first last\n",
       "1  JaneDoe@email.com  Jane  Doe\n",
       "2  JohnDoe@email.com  John  Doe"
      ]
     },
     "execution_count": 282,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = (dft[dft['last'] == 'Doe'])\n",
    "filt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>email</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AbdulKalam@isro.com</td>\n",
       "      <td>Abdul</td>\n",
       "      <td>Kalam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     email  first     last\n",
       "0  CoreyMSchafer@gmail.com  Corey  Schafer\n",
       "4      AbdulKalam@isro.com  Abdul    Kalam"
      ]
     },
     "execution_count": 280,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.drop(index= filt.index)\n",
    "# dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Info:</b> \n",
    "like this in video  \n",
    "\n",
    "![Alt text](06-image-02.png)\n",
    "\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 07 - Sorting Data\n",
    "\n",
    "https://youtu.be/T11QYVfZoD0?si=_nMLS7XWhASycD4M\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#Setup for learning data\n",
    "\n",
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John', 'Ratan', 'Arjun', 'Pushpa', 'Rajinikanth'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe', 'Tata', 'Reddy', 'No Inti Peru', 'Rajinikanth'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com', 'Ratan@tata.com', 'Aready@email.com', 'thaggedhele@puspha.com', 'gmail@rajinikanth.com' ]\n",
    "}\n",
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup for Real world data.\n",
    "###### uncomment when in need to avoid conflicting code autocomplete suggestions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>Ratan@tata.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arjun</td>\n",
       "      <td>Reddy</td>\n",
       "      <td>Aready@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pushpa</td>\n",
       "      <td>No Inti Peru</td>\n",
       "      <td>thaggedhele@puspha.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>gmail@rajinikanth.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         first          last                    email\n",
       "0        Corey       Schafer  CoreyMSchafer@gmail.com\n",
       "1         Jane           Doe        JaneDoe@email.com\n",
       "2         John           Doe        JohnDoe@email.com\n",
       "3        Ratan          Tata           Ratan@tata.com\n",
       "4        Arjun         Reddy         Aready@email.com\n",
       "5       Pushpa  No Inti Peru   thaggedhele@puspha.com\n",
       "6  Rajinikanth   Rajinikanth    gmail@rajinikanth.com"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in order to sor this by last name\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pushpa</td>\n",
       "      <td>No Inti Peru</td>\n",
       "      <td>thaggedhele@puspha.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>gmail@rajinikanth.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arjun</td>\n",
       "      <td>Reddy</td>\n",
       "      <td>Aready@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>Ratan@tata.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         first          last                    email\n",
       "1         Jane           Doe        JaneDoe@email.com\n",
       "2         John           Doe        JohnDoe@email.com\n",
       "5       Pushpa  No Inti Peru   thaggedhele@puspha.com\n",
       "6  Rajinikanth   Rajinikanth    gmail@rajinikanth.com\n",
       "4        Arjun         Reddy         Aready@email.com\n",
       "0        Corey       Schafer  CoreyMSchafer@gmail.com\n",
       "3        Ratan          Tata           Ratan@tata.com"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.sort_values(by='last')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>Ratan@tata.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arjun</td>\n",
       "      <td>Reddy</td>\n",
       "      <td>Aready@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>gmail@rajinikanth.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pushpa</td>\n",
       "      <td>No Inti Peru</td>\n",
       "      <td>thaggedhele@puspha.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         first          last                    email\n",
       "3        Ratan          Tata           Ratan@tata.com\n",
       "0        Corey       Schafer  CoreyMSchafer@gmail.com\n",
       "4        Arjun         Reddy         Aready@email.com\n",
       "6  Rajinikanth   Rajinikanth    gmail@rajinikanth.com\n",
       "5       Pushpa  No Inti Peru   thaggedhele@puspha.com\n",
       "1         Jane           Doe        JaneDoe@email.com\n",
       "2         John           Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.sort_values(by='last', ascending=False) #for descending"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to sort a list of columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>Ratan@tata.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arjun</td>\n",
       "      <td>Reddy</td>\n",
       "      <td>Aready@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>gmail@rajinikanth.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pushpa</td>\n",
       "      <td>No Inti Peru</td>\n",
       "      <td>thaggedhele@puspha.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         first          last                    email\n",
       "3        Ratan          Tata           Ratan@tata.com\n",
       "0        Corey       Schafer  CoreyMSchafer@gmail.com\n",
       "4        Arjun         Reddy         Aready@email.com\n",
       "6  Rajinikanth   Rajinikanth    gmail@rajinikanth.com\n",
       "5       Pushpa  No Inti Peru   thaggedhele@puspha.com\n",
       "2         John           Doe        JohnDoe@email.com\n",
       "1         Jane           Doe        JaneDoe@email.com"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.sort_values(by=['last', 'first'], ascending=False) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "each column in different order. we can give an list of `True` or `False` values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>Ratan@tata.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arjun</td>\n",
       "      <td>Reddy</td>\n",
       "      <td>Aready@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>gmail@rajinikanth.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pushpa</td>\n",
       "      <td>No Inti Peru</td>\n",
       "      <td>thaggedhele@puspha.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         first          last                    email\n",
       "3        Ratan          Tata           Ratan@tata.com\n",
       "0        Corey       Schafer  CoreyMSchafer@gmail.com\n",
       "4        Arjun         Reddy         Aready@email.com\n",
       "6  Rajinikanth   Rajinikanth    gmail@rajinikanth.com\n",
       "5       Pushpa  No Inti Peru   thaggedhele@puspha.com\n",
       "1         Jane           Doe        JaneDoe@email.com\n",
       "2         John           Doe        JohnDoe@email.com"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.sort_values(by=['last', 'first'], ascending=[False, True], inplace=True) \n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to bring it back to how it was sorted by index like before  \n",
    "we use `sort_index()` methos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ratan</td>\n",
       "      <td>Tata</td>\n",
       "      <td>Ratan@tata.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Arjun</td>\n",
       "      <td>Reddy</td>\n",
       "      <td>Aready@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pushpa</td>\n",
       "      <td>No Inti Peru</td>\n",
       "      <td>thaggedhele@puspha.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>Rajinikanth</td>\n",
       "      <td>gmail@rajinikanth.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         first          last                    email\n",
       "0        Corey       Schafer  CoreyMSchafer@gmail.com\n",
       "1         Jane           Doe        JaneDoe@email.com\n",
       "2         John           Doe        JohnDoe@email.com\n",
       "3        Ratan          Tata           Ratan@tata.com\n",
       "4        Arjun         Reddy         Aready@email.com\n",
       "5       Pushpa  No Inti Peru   thaggedhele@puspha.com\n",
       "6  Rajinikanth   Rajinikanth    gmail@rajinikanth.com"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to sort a single column. we can sort a single series as well  \n",
    "series also has sort method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1             Doe\n",
       "2             Doe\n",
       "5    No Inti Peru\n",
       "6     Rajinikanth\n",
       "4           Reddy\n",
       "0         Schafer\n",
       "3            Tata\n",
       "Name: last, dtype: object"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['last'].sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "# SO Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist OpenSourcer  \\\n",
       "Respondent                                                                \n",
       "1           I am a student who is learning to code      Yes       Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment         Country Student  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work  United Kingdom      No   \n",
       "\n",
       "                              EdLevel UndergradMajor  \\\n",
       "Respondent                                             \n",
       "1           Primary/elementary school            NaN   \n",
       "\n",
       "                                                     EduOther OrgSize DevType  \\\n",
       "Respondent                                                                      \n",
       "1           Taught yourself a new language, framework, or ...     NaN     NaN   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant JobSeek LastHireDate LastInt FizzBuzz JobFactors  \\\n",
       "Respondent                                                                     \n",
       "1               NaN     NaN     NaN          NaN     NaN      NaN        NaN   \n",
       "\n",
       "           ResumeUpdate CurrencySymbol CurrencyDesc  CompTotal CompFreq  \\\n",
       "Respondent                                                                \n",
       "1                   NaN            NaN          NaN        NaN      NaN   \n",
       "\n",
       "            ConvertedComp  WorkWeekHrs WorkPlan WorkChallenge WorkRemote  \\\n",
       "Respondent                                                                 \n",
       "1                     NaN          NaN      NaN           NaN        NaN   \n",
       "\n",
       "           WorkLoc ImpSyn CodeRev  CodeRevHrs UnitTests PurchaseHow  \\\n",
       "Respondent                                                            \n",
       "1              NaN    NaN     NaN         NaN       NaN         NaN   \n",
       "\n",
       "           PurchaseWhat               LanguageWorkedWith  \\\n",
       "Respondent                                                 \n",
       "1                   NaN  HTML/CSS;Java;JavaScript;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg BlockchainIs BetterLife  \\\n",
       "Respondent                                         \n",
       "1                    NaN          NaN        Yes   \n",
       "\n",
       "                                            ITperson OffOn SocialMedia  \\\n",
       "Respondent                                                               \n",
       "1           Fortunately, someone else has that title   Yes     Twitter   \n",
       "\n",
       "           Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                      \n",
       "1                Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "\n",
       "           SOAccount SOPartFreq  \\\n",
       "Respondent                        \n",
       "1                 No        NaN   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "\n",
       "                                       EntTeams   SOComm  \\\n",
       "Respondent                                                 \n",
       "1           No, and I don't know what those are  Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>39258</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of LOWER quality than prop...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>100 to 499 employees</td>\n",
       "      <td>Database administrator;Designer;Developer, bac...</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Very confident</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>1-2 years ago</td>\n",
       "      <td>Write code by hand (e.g., on a whiteboard);Int...</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>AFN</td>\n",
       "      <td>Afghan afghani</td>\n",
       "      <td>120000.0</td>\n",
       "      <td>Monthly</td>\n",
       "      <td>19152.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>There is a schedule and/or spec (made by me or...</td>\n",
       "      <td>Not enough people for the workload;Time spent ...</td>\n",
       "      <td>Less than once per month / Never</td>\n",
       "      <td>Office</td>\n",
       "      <td>Far above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Yes, it's not part of our process but the deve...</td>\n",
       "      <td>Developers typically have the most influence o...</td>\n",
       "      <td>I have some influence</td>\n",
       "      <td>HTML/CSS;JavaScript;PHP;SQL;TypeScript</td>\n",
       "      <td>C#;Dart;JavaScript;SQL;TypeScript</td>\n",
       "      <td>MariaDB;MongoDB;MySQL;PostgreSQL</td>\n",
       "      <td>Couchbase;Firebase;MariaDB;MongoDB;Redis</td>\n",
       "      <td>Android;AWS;Linux;Windows;WordPress</td>\n",
       "      <td>Android;AWS;Docker;Heroku;iOS;Linux;Raspberry ...</td>\n",
       "      <td>Angular/Angular.js;Drupal;Express;Laravel</td>\n",
       "      <td>Angular/Angular.js;Drupal;Express</td>\n",
       "      <td>Cordova;Node.js</td>\n",
       "      <td>Hadoop;Node.js</td>\n",
       "      <td>Notepad++;PHPStorm;Vim;Visual Studio Code</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Production</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Facebook</td>\n",
       "      <td>Online</td>\n",
       "      <td>Login</td>\n",
       "      <td>2016</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>1-2 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>0-10 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>East Asian;South Asian</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63129</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>I never completed any formal education</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>Just me - I am a freelancer, sole proprietor, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Less than 1 year</td>\n",
       "      <td>Younger than 5 years</td>\n",
       "      <td>1</td>\n",
       "      <td>Very dissatisfied</td>\n",
       "      <td>Very dissatisfied</td>\n",
       "      <td>Very confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I am actively looking for a job</td>\n",
       "      <td>I've never had a job</td>\n",
       "      <td>Write any code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>How widely used or impactful my work output wo...</td>\n",
       "      <td>I had a negative experience or interaction at ...</td>\n",
       "      <td>AED</td>\n",
       "      <td>United Arab Emirates dirham</td>\n",
       "      <td>648838511.0</td>\n",
       "      <td>Monthly</td>\n",
       "      <td>1000000.0</td>\n",
       "      <td>168.0</td>\n",
       "      <td>There is a schedule and/or spec (made by me or...</td>\n",
       "      <td>Being tasked with non-development work</td>\n",
       "      <td>It's complicated</td>\n",
       "      <td>Other place, such as a coworking space or cafe</td>\n",
       "      <td>Far below average</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;Other(s):</td>\n",
       "      <td>Bash/Shell/PowerShell;TypeScript</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>Android;iOS;Windows;WordPress;Other(s):</td>\n",
       "      <td>Android;iOS;Windows;WordPress;Other(s):</td>\n",
       "      <td>Angular/Angular.js;Express;jQuery;Laravel;Othe...</td>\n",
       "      <td>Angular/Angular.js;Express;jQuery;Laravel;Othe...</td>\n",
       "      <td>Ansible;Apache Spark;Torch/PyTorch</td>\n",
       "      <td>Apache Spark;Torch/PyTorch</td>\n",
       "      <td>Android Studio;Atom;Coda;Eclipse;Emacs;IPython...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Outside of work, for personal projects</td>\n",
       "      <td>Accepting Bitcoin or other coins and tokens as...</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I don't use social media</td>\n",
       "      <td>Online</td>\n",
       "      <td>UserID</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I have never visited Stack Overflow (before to...</td>\n",
       "      <td>Find answers to specific questions</td>\n",
       "      <td>Less than once per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>0-10 minutes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, not at all</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tech articles written by other developers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too short</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85715</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Information systems, information technology, o...</td>\n",
       "      <td>Received on-the-job training in software devel...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, back-end</td>\n",
       "      <td>8</td>\n",
       "      <td>20</td>\n",
       "      <td>4</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Slightly satisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>1-2 years ago</td>\n",
       "      <td>Write code by hand (e.g., on a whiteboard);Int...</td>\n",
       "      <td>No</td>\n",
       "      <td>Financial performance or funding status of the...</td>\n",
       "      <td>I was preparing for a job search</td>\n",
       "      <td>AFN</td>\n",
       "      <td>Afghan afghani</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Monthly</td>\n",
       "      <td>NaN</td>\n",
       "      <td>48.0</td>\n",
       "      <td>There is a schedule and/or spec (made by me or...</td>\n",
       "      <td>Distracting work environment;Non-work commitme...</td>\n",
       "      <td>Less than once per month / Never</td>\n",
       "      <td>Office</td>\n",
       "      <td>A little above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Yes, it's not part of our process but the deve...</td>\n",
       "      <td>Developers and management have nearly equal in...</td>\n",
       "      <td>I have some influence</td>\n",
       "      <td>HTML/CSS;JavaScript;PHP;TypeScript</td>\n",
       "      <td>HTML/CSS;JavaScript;PHP;SQL;TypeScript</td>\n",
       "      <td>MariaDB;MongoDB;MySQL</td>\n",
       "      <td>MariaDB;MongoDB;Microsoft SQL Server;MySQL;Ora...</td>\n",
       "      <td>Linux;Windows;WordPress</td>\n",
       "      <td>Android;AWS;Linux;Windows</td>\n",
       "      <td>jQuery;Laravel</td>\n",
       "      <td>Express;Laravel</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>.NET Core;Node.js;React Native</td>\n",
       "      <td>Notepad++;PHPStorm;Sublime Text;Visual Studio ...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Facebook</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>I don't remember</td>\n",
       "      <td>Multiple times per day</td>\n",
       "      <td>Find answers to specific questions</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I have never participated in Q&amp;A on Stack Over...</td>\n",
       "      <td>No, I knew that Stack Overflow had a job board...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Not sure</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                MainBranch Hobbyist              OpenSourcer  \\\n",
       "Respondent                                                                     \n",
       "39258       I am a developer by profession      Yes  Less than once per year   \n",
       "63129       I am a developer by profession      Yes                    Never   \n",
       "85715       I am a developer by profession       No  Less than once per year   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "39258       OSS is, on average, of LOWER quality than prop...   \n",
       "63129       OSS is, on average, of HIGHER quality than pro...   \n",
       "85715       The quality of OSS and closed source software ...   \n",
       "\n",
       "                    Employment      Country         Student  \\\n",
       "Respondent                                                    \n",
       "39258       Employed full-time  Afghanistan              No   \n",
       "63129       Employed full-time  Afghanistan  Yes, full-time   \n",
       "85715       Employed full-time  Afghanistan              No   \n",
       "\n",
       "                                             EdLevel  \\\n",
       "Respondent                                             \n",
       "39258       Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "63129         I never completed any formal education   \n",
       "85715       Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "\n",
       "                                               UndergradMajor  \\\n",
       "Respondent                                                      \n",
       "39258       Computer science, computer engineering, or sof...   \n",
       "63129                                                     NaN   \n",
       "85715       Information systems, information technology, o...   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "39258       Taken an online course in programming or softw...   \n",
       "63129       Taken an online course in programming or softw...   \n",
       "85715       Received on-the-job training in software devel...   \n",
       "\n",
       "                                                      OrgSize  \\\n",
       "Respondent                                                      \n",
       "39258                                    100 to 499 employees   \n",
       "63129       Just me - I am a freelancer, sole proprietor, ...   \n",
       "85715                                                     NaN   \n",
       "\n",
       "                                                      DevType  \\\n",
       "Respondent                                                      \n",
       "39258       Database administrator;Designer;Developer, bac...   \n",
       "63129                                                     NaN   \n",
       "85715                                     Developer, back-end   \n",
       "\n",
       "                   YearsCode            Age1stCode YearsCodePro  \\\n",
       "Respondent                                                        \n",
       "39258                      2                    22            1   \n",
       "63129       Less than 1 year  Younger than 5 years            1   \n",
       "85715                      8                    20            4   \n",
       "\n",
       "                    CareerSat              JobSat            MgrIdiot  \\\n",
       "Respondent                                                              \n",
       "39258          Very satisfied      Very satisfied      Very confident   \n",
       "63129       Very dissatisfied   Very dissatisfied      Very confident   \n",
       "85715          Very satisfied  Slightly satisfied  Somewhat confident   \n",
       "\n",
       "           MgrMoney MgrWant  \\\n",
       "Respondent                    \n",
       "39258            No     Yes   \n",
       "63129           Yes     Yes   \n",
       "85715           Yes     Yes   \n",
       "\n",
       "                                                      JobSeek  \\\n",
       "Respondent                                                      \n",
       "39258       I’m not actively looking, but I am open to new...   \n",
       "63129                         I am actively looking for a job   \n",
       "85715       I’m not actively looking, but I am open to new...   \n",
       "\n",
       "                    LastHireDate  \\\n",
       "Respondent                         \n",
       "39258              1-2 years ago   \n",
       "63129       I've never had a job   \n",
       "85715              1-2 years ago   \n",
       "\n",
       "                                                      LastInt FizzBuzz  \\\n",
       "Respondent                                                               \n",
       "39258       Write code by hand (e.g., on a whiteboard);Int...       No   \n",
       "63129                                          Write any code      Yes   \n",
       "85715       Write code by hand (e.g., on a whiteboard);Int...       No   \n",
       "\n",
       "                                                   JobFactors  \\\n",
       "Respondent                                                      \n",
       "39258       Languages, frameworks, and other technologies ...   \n",
       "63129       How widely used or impactful my work output wo...   \n",
       "85715       Financial performance or funding status of the...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "39258       Something else changed (education, award, medi...            AFN   \n",
       "63129       I had a negative experience or interaction at ...            AED   \n",
       "85715                        I was preparing for a job search            AFN   \n",
       "\n",
       "                           CurrencyDesc    CompTotal CompFreq  ConvertedComp  \\\n",
       "Respondent                                                                     \n",
       "39258                    Afghan afghani     120000.0  Monthly        19152.0   \n",
       "63129       United Arab Emirates dirham  648838511.0  Monthly      1000000.0   \n",
       "85715                    Afghan afghani          NaN  Monthly            NaN   \n",
       "\n",
       "            WorkWeekHrs                                           WorkPlan  \\\n",
       "Respondent                                                                   \n",
       "39258              49.0  There is a schedule and/or spec (made by me or...   \n",
       "63129             168.0  There is a schedule and/or spec (made by me or...   \n",
       "85715              48.0  There is a schedule and/or spec (made by me or...   \n",
       "\n",
       "                                                WorkChallenge  \\\n",
       "Respondent                                                      \n",
       "39258       Not enough people for the workload;Time spent ...   \n",
       "63129                  Being tasked with non-development work   \n",
       "85715       Distracting work environment;Non-work commitme...   \n",
       "\n",
       "                                  WorkRemote  \\\n",
       "Respondent                                     \n",
       "39258       Less than once per month / Never   \n",
       "63129                       It's complicated   \n",
       "85715       Less than once per month / Never   \n",
       "\n",
       "                                                   WorkLoc  \\\n",
       "Respondent                                                   \n",
       "39258                                               Office   \n",
       "63129       Other place, such as a coworking space or cafe   \n",
       "85715                                               Office   \n",
       "\n",
       "                            ImpSyn                                  CodeRev  \\\n",
       "Respondent                                                                    \n",
       "39258            Far above average  Yes, because I see value in code review   \n",
       "63129            Far below average                                       No   \n",
       "85715       A little above average  Yes, because I see value in code review   \n",
       "\n",
       "            CodeRevHrs                                          UnitTests  \\\n",
       "Respondent                                                                  \n",
       "39258             10.0  Yes, it's not part of our process but the deve...   \n",
       "63129              NaN                                                NaN   \n",
       "85715              5.0  Yes, it's not part of our process but the deve...   \n",
       "\n",
       "                                                  PurchaseHow  \\\n",
       "Respondent                                                      \n",
       "39258       Developers typically have the most influence o...   \n",
       "63129                                                     NaN   \n",
       "85715       Developers and management have nearly equal in...   \n",
       "\n",
       "                     PurchaseWhat                        LanguageWorkedWith  \\\n",
       "Respondent                                                                    \n",
       "39258       I have some influence    HTML/CSS;JavaScript;PHP;SQL;TypeScript   \n",
       "63129                         NaN  Assembly;Bash/Shell/PowerShell;Other(s):   \n",
       "85715       I have some influence        HTML/CSS;JavaScript;PHP;TypeScript   \n",
       "\n",
       "                            LanguageDesireNextYear  \\\n",
       "Respondent                                           \n",
       "39258            C#;Dart;JavaScript;SQL;TypeScript   \n",
       "63129             Bash/Shell/PowerShell;TypeScript   \n",
       "85715       HTML/CSS;JavaScript;PHP;SQL;TypeScript   \n",
       "\n",
       "                          DatabaseWorkedWith  \\\n",
       "Respondent                                     \n",
       "39258       MariaDB;MongoDB;MySQL;PostgreSQL   \n",
       "63129                                  MySQL   \n",
       "85715                  MariaDB;MongoDB;MySQL   \n",
       "\n",
       "                                       DatabaseDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "39258                Couchbase;Firebase;MariaDB;MongoDB;Redis   \n",
       "63129                                                   MySQL   \n",
       "85715       MariaDB;MongoDB;Microsoft SQL Server;MySQL;Ora...   \n",
       "\n",
       "                                 PlatformWorkedWith  \\\n",
       "Respondent                                            \n",
       "39258           Android;AWS;Linux;Windows;WordPress   \n",
       "63129       Android;iOS;Windows;WordPress;Other(s):   \n",
       "85715                       Linux;Windows;WordPress   \n",
       "\n",
       "                                       PlatformDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "39258       Android;AWS;Docker;Heroku;iOS;Linux;Raspberry ...   \n",
       "63129                 Android;iOS;Windows;WordPress;Other(s):   \n",
       "85715                               Android;AWS;Linux;Windows   \n",
       "\n",
       "                                           WebFrameWorkedWith  \\\n",
       "Respondent                                                      \n",
       "39258               Angular/Angular.js;Drupal;Express;Laravel   \n",
       "63129       Angular/Angular.js;Express;jQuery;Laravel;Othe...   \n",
       "85715                                          jQuery;Laravel   \n",
       "\n",
       "                                       WebFrameDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "39258                       Angular/Angular.js;Drupal;Express   \n",
       "63129       Angular/Angular.js;Express;jQuery;Laravel;Othe...   \n",
       "85715                                         Express;Laravel   \n",
       "\n",
       "                            MiscTechWorkedWith  \\\n",
       "Respondent                                       \n",
       "39258                          Cordova;Node.js   \n",
       "63129       Ansible;Apache Spark;Torch/PyTorch   \n",
       "85715                                  Node.js   \n",
       "\n",
       "                    MiscTechDesireNextYear  \\\n",
       "Respondent                                   \n",
       "39258                       Hadoop;Node.js   \n",
       "63129           Apache Spark;Torch/PyTorch   \n",
       "85715       .NET Core;Node.js;React Native   \n",
       "\n",
       "                                                   DevEnviron        OpSys  \\\n",
       "Respondent                                                                   \n",
       "39258               Notepad++;PHPStorm;Vim;Visual Studio Code  Linux-based   \n",
       "63129       Android Studio;Atom;Coda;Eclipse;Emacs;IPython...      Windows   \n",
       "85715       Notepad++;PHPStorm;Sublime Text;Visual Studio ...      Windows   \n",
       "\n",
       "                                        Containers  \\\n",
       "Respondent                                           \n",
       "39258                                   Production   \n",
       "63129       Outside of work, for personal projects   \n",
       "85715                      I do not use containers   \n",
       "\n",
       "                                                BlockchainOrg  \\\n",
       "Respondent                                                      \n",
       "39258                                                     NaN   \n",
       "63129       Accepting Bitcoin or other coins and tokens as...   \n",
       "85715                                                     NaN   \n",
       "\n",
       "                                                 BlockchainIs BetterLife  \\\n",
       "Respondent                                                                 \n",
       "39258                                                     NaN        Yes   \n",
       "63129       Useful across many domains and could change ma...        Yes   \n",
       "85715                                                     NaN         No   \n",
       "\n",
       "           ITperson OffOn               SocialMedia Extraversion ScreenName  \\\n",
       "Respondent                                                                    \n",
       "39258           Yes    No                  Facebook       Online      Login   \n",
       "63129           Yes   Yes  I don't use social media       Online     UserID   \n",
       "85715           Yes   Yes                  Facebook       Online   Username   \n",
       "\n",
       "                  SOVisit1st  \\\n",
       "Respondent                     \n",
       "39258                   2016   \n",
       "63129                    NaN   \n",
       "85715       I don't remember   \n",
       "\n",
       "                                                  SOVisitFreq  \\\n",
       "Respondent                                                      \n",
       "39258                                    A few times per week   \n",
       "63129       I have never visited Stack Overflow (before to...   \n",
       "85715                                  Multiple times per day   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "39258       Find answers to specific questions;Learn how t...   \n",
       "63129                      Find answers to specific questions   \n",
       "85715                      Find answers to specific questions   \n",
       "\n",
       "                       SOFindAnswer                     SOTimeSaved  \\\n",
       "Respondent                                                            \n",
       "39258            1-2 times per week  Stack Overflow was much faster   \n",
       "63129       Less than once per week  Stack Overflow was much faster   \n",
       "85715            3-5 times per week  Stack Overflow was much faster   \n",
       "\n",
       "            SOHowMuchTime SOAccount  \\\n",
       "Respondent                            \n",
       "39258        0-10 minutes       Yes   \n",
       "63129        0-10 minutes       NaN   \n",
       "85715       11-30 minutes       Yes   \n",
       "\n",
       "                                                   SOPartFreq  \\\n",
       "Respondent                                                      \n",
       "39258                     Less than once per month or monthly   \n",
       "63129                                                     NaN   \n",
       "85715       I have never participated in Q&A on Stack Over...   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "39258                                                     Yes   \n",
       "63129       No, I didn't know that Stack Overflow had a jo...   \n",
       "85715       No, I knew that Stack Overflow had a job board...   \n",
       "\n",
       "                                       EntTeams          SOComm  \\\n",
       "Respondent                                                        \n",
       "39258                                       NaN   Yes, somewhat   \n",
       "63129                                       NaN  No, not at all   \n",
       "85715       No, and I don't know what those are        Not sure   \n",
       "\n",
       "                                       WelcomeChange  \\\n",
       "Respondent                                             \n",
       "39258                                            NaN   \n",
       "63129                                            NaN   \n",
       "85715       Somewhat less welcome now than last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "39258       Tech articles written by other developers;Indu...  26.0    Man   \n",
       "63129               Tech articles written by other developers   NaN    NaN   \n",
       "85715       Tech articles written by other developers;Indu...  28.0    Man   \n",
       "\n",
       "           Trans                Sexuality               Ethnicity Dependents  \\\n",
       "Respondent                                                                     \n",
       "39258         No  Straight / Heterosexual  East Asian;South Asian        Yes   \n",
       "63129        NaN                      NaN                     NaN        Yes   \n",
       "85715         No                      NaN                     NaN        Yes   \n",
       "\n",
       "           SurveyLength                  SurveyEase  \n",
       "Respondent                                           \n",
       "39258          Too long                        Easy  \n",
       "63129         Too short                        Easy  \n",
       "85715          Too long  Neither easy nor difficult  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by=['Country']).head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63129</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>1000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50499</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>153216.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39258</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>19152.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58450</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>17556.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7085</th>\n",
       "      <td>Afghanistan</td>\n",
       "      <td>14364.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Country  ConvertedComp\n",
       "Respondent                            \n",
       "63129       Afghanistan      1000000.0\n",
       "50499       Afghanistan       153216.0\n",
       "39258       Afghanistan        19152.0\n",
       "58450       Afghanistan        17556.0\n",
       "7085        Afghanistan        14364.0"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by= ['Country', 'ConvertedComp'], ascending=[True, False], inplace=True)\n",
    "df[['Country', 'ConvertedComp']].head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can  get the 3 largest in an column with `.nlargest(10)` method\n",
    "and smalles with `nsmallest(10)` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "25983    2000000.0\n",
       "87896    2000000.0\n",
       "22013    2000000.0\n",
       "Name: ConvertedComp, dtype: float64"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['ConvertedComp'].nlargest(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "722      0.0\n",
       "28638    0.0\n",
       "13825    0.0\n",
       "Name: ConvertedComp, dtype: float64"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['ConvertedComp'].nsmallest(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can get the realted details like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25983</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Canada</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Received on-the-job training in software devel...</td>\n",
       "      <td>10,000 or more employees</td>\n",
       "      <td>Developer, full-stack</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>Solve a brain-teaser style puzzle;Interview wi...</td>\n",
       "      <td>No</td>\n",
       "      <td>Remote work options;Opportunities for professi...</td>\n",
       "      <td>My job status changed (promotion, new job, etc.)</td>\n",
       "      <td>USD</td>\n",
       "      <td>United States dollar</td>\n",
       "      <td>65500.0</td>\n",
       "      <td>Weekly</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>There is a schedule and/or spec (made by me or...</td>\n",
       "      <td>Being tasked with non-development work;Meeting...</td>\n",
       "      <td>A few days each month</td>\n",
       "      <td>Office</td>\n",
       "      <td>Average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>Developers and management have nearly equal in...</td>\n",
       "      <td>I have little or no influence</td>\n",
       "      <td>Bash/Shell/PowerShell;C++;C#;HTML/CSS;JavaScri...</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;F#;Python;SQL;TypeScript</td>\n",
       "      <td>Microsoft SQL Server;MySQL</td>\n",
       "      <td>Microsoft SQL Server;MySQL</td>\n",
       "      <td>AWS;Docker;Heroku;Linux;MacOS;Raspberry Pi;Win...</td>\n",
       "      <td>AWS;Docker;Linux;MacOS;Windows</td>\n",
       "      <td>Angular/Angular.js</td>\n",
       "      <td>Angular/Angular.js;Flask;React.js</td>\n",
       "      <td>.NET;.NET Core;Node.js</td>\n",
       "      <td>.NET Core;Node.js</td>\n",
       "      <td>Notepad++;Sublime Text;Vim;Visual Studio;Visua...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Development;Production</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful for decentralized currency (i.e., Bitcoin)</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Reddit</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Username</td>\n",
       "      <td>2013</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>1-2 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>No, I knew that Stack Overflow had a job board...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>No, not really</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>NaN</td>\n",
       "      <td>24.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87896</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Germany</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>2-9 employees</td>\n",
       "      <td>Developer, full-stack</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>8</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>No</td>\n",
       "      <td>I am already a manager</td>\n",
       "      <td>I am not interested in new job opportunities</td>\n",
       "      <td>More than 4 years ago</td>\n",
       "      <td>Write any code;Interview with people in senior...</td>\n",
       "      <td>No</td>\n",
       "      <td>Office environment or company culture;Diversit...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>USD</td>\n",
       "      <td>United States dollar</td>\n",
       "      <td>55000.0</td>\n",
       "      <td>Weekly</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>There is a schedule and/or spec (made by me or...</td>\n",
       "      <td>Being tasked with non-development work;Meeting...</td>\n",
       "      <td>All or almost all the time (I'm full-time remote)</td>\n",
       "      <td>Home</td>\n",
       "      <td>A little above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>3.0</td>\n",
       "      <td>No, but I think we should</td>\n",
       "      <td>Developers and management have nearly equal in...</td>\n",
       "      <td>I have a great deal of influence</td>\n",
       "      <td>HTML/CSS;JavaScript;PHP;SQL</td>\n",
       "      <td>HTML/CSS;JavaScript;PHP;Ruby;SQL;TypeScript</td>\n",
       "      <td>MariaDB;MySQL;PostgreSQL</td>\n",
       "      <td>MariaDB;MySQL;PostgreSQL</td>\n",
       "      <td>WordPress</td>\n",
       "      <td>NaN</td>\n",
       "      <td>jQuery;Laravel;Vue.js</td>\n",
       "      <td>Express;Laravel;Ruby on Rails;Vue.js</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Sublime Text;Vim;Visual Studio Code</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>Not at all</td>\n",
       "      <td>Useful for immutable record keeping outside of...</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Handle</td>\n",
       "      <td>2011</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>0-10 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>No, I knew that Stack Overflow had a job board...</td>\n",
       "      <td>No, I've heard of them, but I am not part of a...</td>\n",
       "      <td>No, not really</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>32.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Gay or Lesbian</td>\n",
       "      <td>White or of European descent</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22013</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>India</td>\n",
       "      <td>No</td>\n",
       "      <td>Professional degree (JD, MD, etc.)</td>\n",
       "      <td>A natural science (ex. biology, chemistry, phy...</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>2-9 employees</td>\n",
       "      <td>Academic researcher;Data scientist or machine ...</td>\n",
       "      <td>3</td>\n",
       "      <td>26</td>\n",
       "      <td>3</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>Very satisfied</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I am not interested in new job opportunities</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>Write code by hand (e.g., on a whiteboard);Com...</td>\n",
       "      <td>No</td>\n",
       "      <td>Financial performance or funding status of the...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>USD</td>\n",
       "      <td>United States dollar</td>\n",
       "      <td>1500000.0</td>\n",
       "      <td>Weekly</td>\n",
       "      <td>2000000.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>Inadequate access to necessary tools;Meetings;...</td>\n",
       "      <td>All or almost all the time (I'm full-time remote)</td>\n",
       "      <td>Home</td>\n",
       "      <td>A little below average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>Developers typically have the most influence o...</td>\n",
       "      <td>I have a great deal of influence</td>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;Java;Java...</td>\n",
       "      <td>Python;Swift</td>\n",
       "      <td>Cassandra;Microsoft SQL Server;Oracle</td>\n",
       "      <td>Cassandra</td>\n",
       "      <td>Android;Arduino;Google Cloud Platform;IBM Clou...</td>\n",
       "      <td>IBM Cloud or Watson;Windows</td>\n",
       "      <td>jQuery</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TensorFlow</td>\n",
       "      <td>TensorFlow</td>\n",
       "      <td>Visual Studio;Visual Studio Code</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Development</td>\n",
       "      <td>Implementing our own cryptocurrency</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>Online</td>\n",
       "      <td>Screen Name</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions</td>\n",
       "      <td>1-2 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>60+ minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I have never participated in Q&amp;A on Stack Over...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes, definitely</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Easy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                MainBranch Hobbyist              OpenSourcer  \\\n",
       "Respondent                                                                     \n",
       "25983       I am a developer by profession      Yes  Less than once per year   \n",
       "87896       I am a developer by profession      Yes  Less than once per year   \n",
       "22013       I am a developer by profession      Yes                    Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "25983       OSS is, on average, of HIGHER quality than pro...   \n",
       "87896       The quality of OSS and closed source software ...   \n",
       "22013       The quality of OSS and closed source software ...   \n",
       "\n",
       "                    Employment  Country Student  \\\n",
       "Respondent                                        \n",
       "25983       Employed full-time   Canada      No   \n",
       "87896       Employed full-time  Germany      No   \n",
       "22013       Employed full-time    India      No   \n",
       "\n",
       "                                             EdLevel  \\\n",
       "Respondent                                             \n",
       "25983       Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "87896       Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "22013             Professional degree (JD, MD, etc.)   \n",
       "\n",
       "                                               UndergradMajor  \\\n",
       "Respondent                                                      \n",
       "25983       Computer science, computer engineering, or sof...   \n",
       "87896       Computer science, computer engineering, or sof...   \n",
       "22013       A natural science (ex. biology, chemistry, phy...   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "25983       Received on-the-job training in software devel...   \n",
       "87896       Taken an online course in programming or softw...   \n",
       "22013       Taken an online course in programming or softw...   \n",
       "\n",
       "                             OrgSize  \\\n",
       "Respondent                             \n",
       "25983       10,000 or more employees   \n",
       "87896                  2-9 employees   \n",
       "22013                  2-9 employees   \n",
       "\n",
       "                                                      DevType YearsCode  \\\n",
       "Respondent                                                                \n",
       "25983                                   Developer, full-stack         8   \n",
       "87896                                   Developer, full-stack         9   \n",
       "22013       Academic researcher;Data scientist or machine ...         3   \n",
       "\n",
       "           Age1stCode YearsCodePro       CareerSat          JobSat  \\\n",
       "Respondent                                                           \n",
       "25983              16            2  Very satisfied  Very satisfied   \n",
       "87896              17            8  Very satisfied  Very satisfied   \n",
       "22013              26            3  Very satisfied  Very satisfied   \n",
       "\n",
       "                      MgrIdiot MgrMoney                 MgrWant  \\\n",
       "Respondent                                                        \n",
       "25983       Somewhat confident       No                      No   \n",
       "87896       Somewhat confident       No  I am already a manager   \n",
       "22013                      NaN      NaN                     NaN   \n",
       "\n",
       "                                                      JobSeek  \\\n",
       "Respondent                                                      \n",
       "25983       I’m not actively looking, but I am open to new...   \n",
       "87896            I am not interested in new job opportunities   \n",
       "22013            I am not interested in new job opportunities   \n",
       "\n",
       "                     LastHireDate  \\\n",
       "Respondent                          \n",
       "25983               3-4 years ago   \n",
       "87896       More than 4 years ago   \n",
       "22013               3-4 years ago   \n",
       "\n",
       "                                                      LastInt FizzBuzz  \\\n",
       "Respondent                                                               \n",
       "25983       Solve a brain-teaser style puzzle;Interview wi...       No   \n",
       "87896       Write any code;Interview with people in senior...       No   \n",
       "22013       Write code by hand (e.g., on a whiteboard);Com...       No   \n",
       "\n",
       "                                                   JobFactors  \\\n",
       "Respondent                                                      \n",
       "25983       Remote work options;Opportunities for professi...   \n",
       "87896       Office environment or company culture;Diversit...   \n",
       "22013       Financial performance or funding status of the...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "25983        My job status changed (promotion, new job, etc.)            USD   \n",
       "87896       Something else changed (education, award, medi...            USD   \n",
       "22013       Something else changed (education, award, medi...            USD   \n",
       "\n",
       "                    CurrencyDesc  CompTotal CompFreq  ConvertedComp  \\\n",
       "Respondent                                                            \n",
       "25983       United States dollar    65500.0   Weekly      2000000.0   \n",
       "87896       United States dollar    55000.0   Weekly      2000000.0   \n",
       "22013       United States dollar  1500000.0   Weekly      2000000.0   \n",
       "\n",
       "            WorkWeekHrs                                           WorkPlan  \\\n",
       "Respondent                                                                   \n",
       "25983              45.0  There is a schedule and/or spec (made by me or...   \n",
       "87896              40.0  There is a schedule and/or spec (made by me or...   \n",
       "22013              50.0  There's no schedule or spec; I work on what se...   \n",
       "\n",
       "                                                WorkChallenge  \\\n",
       "Respondent                                                      \n",
       "25983       Being tasked with non-development work;Meeting...   \n",
       "87896       Being tasked with non-development work;Meeting...   \n",
       "22013       Inadequate access to necessary tools;Meetings;...   \n",
       "\n",
       "                                                   WorkRemote WorkLoc  \\\n",
       "Respondent                                                              \n",
       "25983                                   A few days each month  Office   \n",
       "87896       All or almost all the time (I'm full-time remote)    Home   \n",
       "22013       All or almost all the time (I'm full-time remote)    Home   \n",
       "\n",
       "                            ImpSyn                                  CodeRev  \\\n",
       "Respondent                                                                    \n",
       "25983                      Average  Yes, because I see value in code review   \n",
       "87896       A little above average  Yes, because I see value in code review   \n",
       "22013       A little below average  Yes, because I see value in code review   \n",
       "\n",
       "            CodeRevHrs                      UnitTests  \\\n",
       "Respondent                                              \n",
       "25983              2.0  Yes, it's part of our process   \n",
       "87896              3.0      No, but I think we should   \n",
       "22013             10.0  Yes, it's part of our process   \n",
       "\n",
       "                                                  PurchaseHow  \\\n",
       "Respondent                                                      \n",
       "25983       Developers and management have nearly equal in...   \n",
       "87896       Developers and management have nearly equal in...   \n",
       "22013       Developers typically have the most influence o...   \n",
       "\n",
       "                                PurchaseWhat  \\\n",
       "Respondent                                     \n",
       "25983          I have little or no influence   \n",
       "87896       I have a great deal of influence   \n",
       "22013       I have a great deal of influence   \n",
       "\n",
       "                                           LanguageWorkedWith  \\\n",
       "Respondent                                                      \n",
       "25983       Bash/Shell/PowerShell;C++;C#;HTML/CSS;JavaScri...   \n",
       "87896                             HTML/CSS;JavaScript;PHP;SQL   \n",
       "22013       Assembly;Bash/Shell/PowerShell;C;C++;Java;Java...   \n",
       "\n",
       "                                       LanguageDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "25983       Bash/Shell/PowerShell;C#;F#;Python;SQL;TypeScript   \n",
       "87896             HTML/CSS;JavaScript;PHP;Ruby;SQL;TypeScript   \n",
       "22013                                            Python;Swift   \n",
       "\n",
       "                               DatabaseWorkedWith      DatabaseDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "25983                  Microsoft SQL Server;MySQL  Microsoft SQL Server;MySQL   \n",
       "87896                    MariaDB;MySQL;PostgreSQL    MariaDB;MySQL;PostgreSQL   \n",
       "22013       Cassandra;Microsoft SQL Server;Oracle                   Cassandra   \n",
       "\n",
       "                                           PlatformWorkedWith  \\\n",
       "Respondent                                                      \n",
       "25983       AWS;Docker;Heroku;Linux;MacOS;Raspberry Pi;Win...   \n",
       "87896                                               WordPress   \n",
       "22013       Android;Arduino;Google Cloud Platform;IBM Clou...   \n",
       "\n",
       "                    PlatformDesireNextYear     WebFrameWorkedWith  \\\n",
       "Respondent                                                          \n",
       "25983       AWS;Docker;Linux;MacOS;Windows     Angular/Angular.js   \n",
       "87896                                  NaN  jQuery;Laravel;Vue.js   \n",
       "22013          IBM Cloud or Watson;Windows                 jQuery   \n",
       "\n",
       "                          WebFrameDesireNextYear      MiscTechWorkedWith  \\\n",
       "Respondent                                                                 \n",
       "25983          Angular/Angular.js;Flask;React.js  .NET;.NET Core;Node.js   \n",
       "87896       Express;Laravel;Ruby on Rails;Vue.js                     NaN   \n",
       "22013                                        NaN              TensorFlow   \n",
       "\n",
       "           MiscTechDesireNextYear  \\\n",
       "Respondent                          \n",
       "25983           .NET Core;Node.js   \n",
       "87896                     Node.js   \n",
       "22013                  TensorFlow   \n",
       "\n",
       "                                                   DevEnviron        OpSys  \\\n",
       "Respondent                                                                   \n",
       "25983       Notepad++;Sublime Text;Vim;Visual Studio;Visua...      Windows   \n",
       "87896                     Sublime Text;Vim;Visual Studio Code  Linux-based   \n",
       "22013                        Visual Studio;Visual Studio Code      Windows   \n",
       "\n",
       "                         Containers                        BlockchainOrg  \\\n",
       "Respondent                                                                 \n",
       "25983        Development;Production                                  NaN   \n",
       "87896       I do not use containers                           Not at all   \n",
       "22013                   Development  Implementing our own cryptocurrency   \n",
       "\n",
       "                                                 BlockchainIs BetterLife  \\\n",
       "Respondent                                                                 \n",
       "25983       Useful for decentralized currency (i.e., Bitcoin)         No   \n",
       "87896       Useful for immutable record keeping outside of...         No   \n",
       "22013       Useful across many domains and could change ma...        Yes   \n",
       "\n",
       "           ITperson OffOn SocialMedia              Extraversion   ScreenName  \\\n",
       "Respondent                                                                     \n",
       "25983           Yes   Yes      Reddit  In real life (in person)     Username   \n",
       "87896           Yes   Yes     Twitter                    Online       Handle   \n",
       "22013           Yes    No     YouTube                    Online  Screen Name   \n",
       "\n",
       "           SOVisit1st            SOVisitFreq  \\\n",
       "Respondent                                     \n",
       "25983            2013  Daily or almost daily   \n",
       "87896            2011  Daily or almost daily   \n",
       "22013             NaN   A few times per week   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "25983       Find answers to specific questions;Learn how t...   \n",
       "87896                      Find answers to specific questions   \n",
       "22013                      Find answers to specific questions   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "25983       1-2 times per week  Stack Overflow was much faster  11-30 minutes   \n",
       "87896       3-5 times per week  Stack Overflow was much faster   0-10 minutes   \n",
       "22013       1-2 times per week  Stack Overflow was much faster    60+ minutes   \n",
       "\n",
       "           SOAccount                                         SOPartFreq  \\\n",
       "Respondent                                                                \n",
       "25983            Yes                Less than once per month or monthly   \n",
       "87896            Yes                Less than once per month or monthly   \n",
       "22013            Yes  I have never participated in Q&A on Stack Over...   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "25983       No, I knew that Stack Overflow had a job board...   \n",
       "87896       No, I knew that Stack Overflow had a job board...   \n",
       "22013                                                     Yes   \n",
       "\n",
       "                                                     EntTeams  \\\n",
       "Respondent                                                      \n",
       "25983                     No, and I don't know what those are   \n",
       "87896       No, I've heard of them, but I am not part of a...   \n",
       "22013                                                     Yes   \n",
       "\n",
       "                     SOComm                            WelcomeChange  \\\n",
       "Respondent                                                             \n",
       "25983        No, not really  Just as welcome now as I felt last year   \n",
       "87896        No, not really  Just as welcome now as I felt last year   \n",
       "22013       Yes, definitely    A lot more welcome now than last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "25983                                                     NaN  24.0    Man   \n",
       "87896       Tech articles written by other developers;Tech...  32.0    Man   \n",
       "22013       Tech articles written by other developers;Indu...   NaN    Man   \n",
       "\n",
       "           Trans                Sexuality                     Ethnicity  \\\n",
       "Respondent                                                                \n",
       "25983         No  Straight / Heterosexual  White or of European descent   \n",
       "87896         No           Gay or Lesbian  White or of European descent   \n",
       "22013         No  Straight / Heterosexual                           NaN   \n",
       "\n",
       "           Dependents           SurveyLength                  SurveyEase  \n",
       "Respondent                                                                \n",
       "25983              No  Appropriate in length                        Easy  \n",
       "87896              No  Appropriate in length  Neither easy nor difficult  \n",
       "22013             Yes               Too long                        Easy  "
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.nlargest(3, 'ConvertedComp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 08 - Grouping and Aggregating - Analyzing and Exploring Your Data\n",
    "\n",
    "https://youtu.be/txMdrV1Ut64?si=SqRDKz2NN3VBccXA\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#Setup for learning data\n",
    "\n",
    "# people = {\n",
    "#     \"first\": [\"Corey\", 'Jane', 'John', 'Ratan', 'Arjun', 'Pushpa', 'Rajinikanth'], \n",
    "#     \"last\": [\"Schafer\", 'Doe', 'Doe', 'Tata', 'Reddy', 'No Inti Peru', 'Rajinikanth'], \n",
    "#     \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com', 'Ratan@tata.com', 'Aready@email.com', 'thaggedhele@puspha.com', 'gmail@rajinikanth.com' ]\n",
    "# }\n",
    "# dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup for Real world data.\n",
    "###### uncomment when in need to avoid conflicting code autocomplete suggestions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist OpenSourcer  \\\n",
       "Respondent                                                                \n",
       "1           I am a student who is learning to code      Yes       Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment         Country Student  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work  United Kingdom      No   \n",
       "\n",
       "                              EdLevel UndergradMajor  \\\n",
       "Respondent                                             \n",
       "1           Primary/elementary school            NaN   \n",
       "\n",
       "                                                     EduOther OrgSize DevType  \\\n",
       "Respondent                                                                      \n",
       "1           Taught yourself a new language, framework, or ...     NaN     NaN   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant JobSeek LastHireDate LastInt FizzBuzz JobFactors  \\\n",
       "Respondent                                                                     \n",
       "1               NaN     NaN     NaN          NaN     NaN      NaN        NaN   \n",
       "\n",
       "           ResumeUpdate CurrencySymbol CurrencyDesc  CompTotal CompFreq  \\\n",
       "Respondent                                                                \n",
       "1                   NaN            NaN          NaN        NaN      NaN   \n",
       "\n",
       "            ConvertedComp  WorkWeekHrs WorkPlan WorkChallenge WorkRemote  \\\n",
       "Respondent                                                                 \n",
       "1                     NaN          NaN      NaN           NaN        NaN   \n",
       "\n",
       "           WorkLoc ImpSyn CodeRev  CodeRevHrs UnitTests PurchaseHow  \\\n",
       "Respondent                                                            \n",
       "1              NaN    NaN     NaN         NaN       NaN         NaN   \n",
       "\n",
       "           PurchaseWhat               LanguageWorkedWith  \\\n",
       "Respondent                                                 \n",
       "1                   NaN  HTML/CSS;Java;JavaScript;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg BlockchainIs BetterLife  \\\n",
       "Respondent                                         \n",
       "1                    NaN          NaN        Yes   \n",
       "\n",
       "                                            ITperson OffOn SocialMedia  \\\n",
       "Respondent                                                               \n",
       "1           Fortunately, someone else has that title   Yes     Twitter   \n",
       "\n",
       "           Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                      \n",
       "1                Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "\n",
       "           SOAccount SOPartFreq  \\\n",
       "Respondent                        \n",
       "1                 No        NaN   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "\n",
       "                                       EntTeams   SOComm  \\\n",
       "Respondent                                                 \n",
       "1           No, and I don't know what those are  Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get median salary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57287.0"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['ConvertedComp'].median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">can do median for whole dataframe  \n",
    "\n",
    "\n",
    "     error when using on 01 Oct 2023 `Pandas 2.1.1`\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get statistical overview of whole dataframe "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5.594500e+04</td>\n",
       "      <td>5.582300e+04</td>\n",
       "      <td>64503.000000</td>\n",
       "      <td>49790.000000</td>\n",
       "      <td>79210.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.519014e+11</td>\n",
       "      <td>1.271107e+05</td>\n",
       "      <td>42.127197</td>\n",
       "      <td>5.084308</td>\n",
       "      <td>30.336699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>7.331926e+13</td>\n",
       "      <td>2.841523e+05</td>\n",
       "      <td>37.287610</td>\n",
       "      <td>5.513931</td>\n",
       "      <td>9.178390</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.000000e+04</td>\n",
       "      <td>2.577750e+04</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>24.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>6.200000e+04</td>\n",
       "      <td>5.728700e+04</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>29.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.200000e+05</td>\n",
       "      <td>1.000000e+05</td>\n",
       "      <td>44.750000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>35.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000e+16</td>\n",
       "      <td>2.000000e+06</td>\n",
       "      <td>4850.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>99.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          CompTotal  ConvertedComp   WorkWeekHrs    CodeRevHrs           Age\n",
       "count  5.594500e+04   5.582300e+04  64503.000000  49790.000000  79210.000000\n",
       "mean   5.519014e+11   1.271107e+05     42.127197      5.084308     30.336699\n",
       "std    7.331926e+13   2.841523e+05     37.287610      5.513931      9.178390\n",
       "min    0.000000e+00   0.000000e+00      1.000000      0.000000      1.000000\n",
       "25%    2.000000e+04   2.577750e+04     40.000000      2.000000     24.000000\n",
       "50%    6.200000e+04   5.728700e+04     40.000000      4.000000     29.000000\n",
       "75%    1.200000e+05   1.000000e+05     44.750000      6.000000     35.000000\n",
       "max    1.000000e+16   2.000000e+06   4850.000000     99.000000     99.000000"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "count column is number of non NaN rows\n",
    "\n",
    "to count the individual values in the column we use `.value_counts()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Hobbyist\n",
       "Yes    71257\n",
       "No     17626\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Hobbyist'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "1      Twitter\n",
       "2    Instagram\n",
       "3       Reddit\n",
       "Name: SocialMedia, dtype: object"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['SocialMedia'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QuestionText    What social media site do you use the most?\n",
       "Name: SocialMedia, dtype: object"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df.loc['SocialMedia']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SocialMedia\n",
       "Reddit      14374\n",
       "YouTube     13830\n",
       "WhatsApp    13347\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['SocialMedia'].value_counts().head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to show it as <Mark>percentage</mark> we use `normalize=True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SocialMedia\n",
       "Reddit      0.170233\n",
       "YouTube     0.163791\n",
       "WhatsApp    0.158071\n",
       "Name: proportion, dtype: float64"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['SocialMedia'].value_counts(normalize=True).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do we get pupular social media sites based on country so we have to <mark>group by</mark> function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "United States    20949\n",
       "India             9061\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Country'].value_counts().head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x28ceaf990>"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp = df.groupby(['Country'])\n",
    "country_grp # <pandas.core.groupby.generic.DataFrameGroupBy object at 0x28d71c210>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>I code primarily as a hobby</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, back-end;Engineer, site reliability</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...</td>\n",
       "      <td>Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...</td>\n",
       "      <td>Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...</td>\n",
       "      <td>AWS;Docker;Heroku;Linux;MacOS;Slack</td>\n",
       "      <td>Android;Arduino;AWS;Docker;Google Cloud Platfo...</td>\n",
       "      <td>Express;Flask;React.js;Spring</td>\n",
       "      <td>Django;Express;Flask;React.js;Vue.js</td>\n",
       "      <td>Hadoop;Node.js;Pandas</td>\n",
       "      <td>Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...</td>\n",
       "      <td>Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development;Testing;Production;Outside of work...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>SIGH</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Handle</td>\n",
       "      <td>2012</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>Less than once per week</td>\n",
       "      <td>Stack Overflow was slightly faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, definitely</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>India</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10,000 or more employees</td>\n",
       "      <td>Data or business analyst;Data scientist or mac...</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>INR</td>\n",
       "      <td>Indian rupee</td>\n",
       "      <td>950000.0</td>\n",
       "      <td>Yearly</td>\n",
       "      <td>13293.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A few days each month</td>\n",
       "      <td>Home</td>\n",
       "      <td>Far above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>C#;Go;JavaScript;Kotlin;Python;R;SQL</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server;MyS...</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Linux;Raspberry Pi;Windows</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>.NET;Node.js;Pandas;Torch/PyTorch</td>\n",
       "      <td>.NET;Node.js;TensorFlow;Torch/PyTorch</td>\n",
       "      <td>Android Studio;Eclipse;IPython / Jupyter;Notep...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not at all</td>\n",
       "      <td>Useful for immutable record keeping outside of...</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>Neither</td>\n",
       "      <td>Screen Name</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Multiple times per day</td>\n",
       "      <td>Find answers to specific questions;Get a sense...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                MainBranch Hobbyist  \\\n",
       "Respondent                                            \n",
       "8              I code primarily as a hobby      Yes   \n",
       "10          I am a developer by profession      Yes   \n",
       "\n",
       "                           OpenSourcer  \\\n",
       "Respondent                               \n",
       "8              Less than once per year   \n",
       "10          Once a month or more often   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "8           OSS is, on average, of HIGHER quality than pro...   \n",
       "10          OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                                    Employment Country Student  \\\n",
       "Respondent                                                       \n",
       "8           Not employed, but looking for work   India     NaN   \n",
       "10                          Employed full-time   India      No   \n",
       "\n",
       "                                                EdLevel  \\\n",
       "Respondent                                                \n",
       "8              Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "10          Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "\n",
       "                                               UndergradMajor  \\\n",
       "Respondent                                                      \n",
       "8           Computer science, computer engineering, or sof...   \n",
       "10                                                        NaN   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "8           Taught yourself a new language, framework, or ...   \n",
       "10                                                        NaN   \n",
       "\n",
       "                             OrgSize  \\\n",
       "Respondent                             \n",
       "8                                NaN   \n",
       "10          10,000 or more employees   \n",
       "\n",
       "                                                      DevType YearsCode  \\\n",
       "Respondent                                                                \n",
       "8              Developer, back-end;Engineer, site reliability         8   \n",
       "10          Data or business analyst;Data scientist or mac...        12   \n",
       "\n",
       "           Age1stCode YearsCodePro              CareerSat  \\\n",
       "Respondent                                                  \n",
       "8                  16          NaN                    NaN   \n",
       "10                 20           10  Slightly dissatisfied   \n",
       "\n",
       "                           JobSat            MgrIdiot MgrMoney MgrWant  \\\n",
       "Respondent                                                               \n",
       "8                             NaN                 NaN      NaN     NaN   \n",
       "10          Slightly dissatisfied  Somewhat confident      Yes     Yes   \n",
       "\n",
       "                                                      JobSeek   LastHireDate  \\\n",
       "Respondent                                                                     \n",
       "8                                                         NaN            NaN   \n",
       "10          I’m not actively looking, but I am open to new...  3-4 years ago   \n",
       "\n",
       "           LastInt FizzBuzz  \\\n",
       "Respondent                    \n",
       "8              NaN      NaN   \n",
       "10             NaN       No   \n",
       "\n",
       "                                                   JobFactors ResumeUpdate  \\\n",
       "Respondent                                                                   \n",
       "8                                                         NaN          NaN   \n",
       "10          Languages, frameworks, and other technologies ...          NaN   \n",
       "\n",
       "           CurrencySymbol  CurrencyDesc  CompTotal CompFreq  ConvertedComp  \\\n",
       "Respondent                                                                   \n",
       "8                     NaN           NaN        NaN      NaN            NaN   \n",
       "10                    INR  Indian rupee   950000.0   Yearly        13293.0   \n",
       "\n",
       "            WorkWeekHrs                                           WorkPlan  \\\n",
       "Respondent                                                                   \n",
       "8                   NaN                                                NaN   \n",
       "10                 70.0  There's no schedule or spec; I work on what se...   \n",
       "\n",
       "           WorkChallenge             WorkRemote WorkLoc             ImpSyn  \\\n",
       "Respondent                                                                   \n",
       "8                    NaN                    NaN     NaN                NaN   \n",
       "10                   NaN  A few days each month    Home  Far above average   \n",
       "\n",
       "                                            CodeRev  CodeRevHrs  \\\n",
       "Respondent                                                        \n",
       "8                                               NaN         NaN   \n",
       "10          Yes, because I see value in code review         4.0   \n",
       "\n",
       "                                UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                           \n",
       "8                                     NaN         NaN          NaN   \n",
       "10          Yes, it's part of our process         NaN          NaN   \n",
       "\n",
       "                                           LanguageWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...   \n",
       "10                              C#;Go;JavaScript;Python;R;SQL   \n",
       "\n",
       "                                       LanguageDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...   \n",
       "10                       C#;Go;JavaScript;Kotlin;Python;R;SQL   \n",
       "\n",
       "                                           DatabaseWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...   \n",
       "10          Elasticsearch;MongoDB;Microsoft SQL Server;MyS...   \n",
       "\n",
       "                                       DatabaseDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...   \n",
       "10                 Elasticsearch;MongoDB;Microsoft SQL Server   \n",
       "\n",
       "                             PlatformWorkedWith  \\\n",
       "Respondent                                        \n",
       "8           AWS;Docker;Heroku;Linux;MacOS;Slack   \n",
       "10                                Linux;Windows   \n",
       "\n",
       "                                       PlatformDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Android;Arduino;AWS;Docker;Google Cloud Platfo...   \n",
       "10                         Android;Linux;Raspberry Pi;Windows   \n",
       "\n",
       "                                           WebFrameWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8                               Express;Flask;React.js;Spring   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "\n",
       "                                       WebFrameDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8                        Django;Express;Flask;React.js;Vue.js   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "\n",
       "                           MiscTechWorkedWith  \\\n",
       "Respondent                                      \n",
       "8                       Hadoop;Node.js;Pandas   \n",
       "10          .NET;Node.js;Pandas;Torch/PyTorch   \n",
       "\n",
       "                                       MiscTechDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...   \n",
       "10                      .NET;Node.js;TensorFlow;Torch/PyTorch   \n",
       "\n",
       "                                                   DevEnviron        OpSys  \\\n",
       "Respondent                                                                   \n",
       "8           Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...  Linux-based   \n",
       "10          Android Studio;Eclipse;IPython / Jupyter;Notep...      Windows   \n",
       "\n",
       "                                                   Containers BlockchainOrg  \\\n",
       "Respondent                                                                    \n",
       "8           Development;Testing;Production;Outside of work...           NaN   \n",
       "10                                                        NaN    Not at all   \n",
       "\n",
       "                                                 BlockchainIs BetterLife  \\\n",
       "Respondent                                                                 \n",
       "8           Useful across many domains and could change ma...        Yes   \n",
       "10          Useful for immutable record keeping outside of...         No   \n",
       "\n",
       "           ITperson OffOn SocialMedia              Extraversion   ScreenName  \\\n",
       "Respondent                                                                     \n",
       "8              SIGH   Yes     YouTube  In real life (in person)       Handle   \n",
       "10              Yes   Yes     YouTube                   Neither  Screen Name   \n",
       "\n",
       "           SOVisit1st             SOVisitFreq  \\\n",
       "Respondent                                      \n",
       "8                2012    A few times per week   \n",
       "10                NaN  Multiple times per day   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "8           Find answers to specific questions;Learn how t...   \n",
       "10          Find answers to specific questions;Get a sense...   \n",
       "\n",
       "                       SOFindAnswer                         SOTimeSaved  \\\n",
       "Respondent                                                                \n",
       "8           Less than once per week  Stack Overflow was slightly faster   \n",
       "10               3-5 times per week            They were about the same   \n",
       "\n",
       "            SOHowMuchTime SOAccount                           SOPartFreq  \\\n",
       "Respondent                                                                 \n",
       "8           11-30 minutes       Yes  Less than once per month or monthly   \n",
       "10                    NaN       Yes      A few times per month or weekly   \n",
       "\n",
       "           SOJobs                             EntTeams           SOComm  \\\n",
       "Respondent                                                                \n",
       "8             Yes  No, and I don't know what those are  Yes, definitely   \n",
       "10            Yes  No, and I don't know what those are    Yes, somewhat   \n",
       "\n",
       "                                       WelcomeChange  \\\n",
       "Respondent                                             \n",
       "8              A lot more welcome now than last year   \n",
       "10          Somewhat less welcome now than last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "8           Tech articles written by other developers;Indu...  24.0    Man   \n",
       "10          Tech articles written by other developers;Tech...   NaN    NaN   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "8             No  Straight / Heterosexual       NaN        NaN   \n",
       "10           NaN                      NaN       NaN        Yes   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "8           Appropriate in length  Neither easy nor difficult  \n",
       "10                       Too long                   Difficult  "
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp.get_group('India').head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this is similar to running a filter in dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>I code primarily as a hobby</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, back-end;Engineer, site reliability</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...</td>\n",
       "      <td>Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...</td>\n",
       "      <td>Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...</td>\n",
       "      <td>AWS;Docker;Heroku;Linux;MacOS;Slack</td>\n",
       "      <td>Android;Arduino;AWS;Docker;Google Cloud Platfo...</td>\n",
       "      <td>Express;Flask;React.js;Spring</td>\n",
       "      <td>Django;Express;Flask;React.js;Vue.js</td>\n",
       "      <td>Hadoop;Node.js;Pandas</td>\n",
       "      <td>Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...</td>\n",
       "      <td>Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development;Testing;Production;Outside of work...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>SIGH</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Handle</td>\n",
       "      <td>2012</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>Less than once per week</td>\n",
       "      <td>Stack Overflow was slightly faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, definitely</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>India</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10,000 or more employees</td>\n",
       "      <td>Data or business analyst;Data scientist or mac...</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>INR</td>\n",
       "      <td>Indian rupee</td>\n",
       "      <td>950000.0</td>\n",
       "      <td>Yearly</td>\n",
       "      <td>13293.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A few days each month</td>\n",
       "      <td>Home</td>\n",
       "      <td>Far above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>C#;Go;JavaScript;Kotlin;Python;R;SQL</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server;MyS...</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Linux;Raspberry Pi;Windows</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>.NET;Node.js;Pandas;Torch/PyTorch</td>\n",
       "      <td>.NET;Node.js;TensorFlow;Torch/PyTorch</td>\n",
       "      <td>Android Studio;Eclipse;IPython / Jupyter;Notep...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not at all</td>\n",
       "      <td>Useful for immutable record keeping outside of...</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>Neither</td>\n",
       "      <td>Screen Name</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Multiple times per day</td>\n",
       "      <td>Find answers to specific questions;Get a sense...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                MainBranch Hobbyist  \\\n",
       "Respondent                                            \n",
       "8              I code primarily as a hobby      Yes   \n",
       "10          I am a developer by profession      Yes   \n",
       "\n",
       "                           OpenSourcer  \\\n",
       "Respondent                               \n",
       "8              Less than once per year   \n",
       "10          Once a month or more often   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "8           OSS is, on average, of HIGHER quality than pro...   \n",
       "10          OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                                    Employment Country Student  \\\n",
       "Respondent                                                       \n",
       "8           Not employed, but looking for work   India     NaN   \n",
       "10                          Employed full-time   India      No   \n",
       "\n",
       "                                                EdLevel  \\\n",
       "Respondent                                                \n",
       "8              Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "10          Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "\n",
       "                                               UndergradMajor  \\\n",
       "Respondent                                                      \n",
       "8           Computer science, computer engineering, or sof...   \n",
       "10                                                        NaN   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "8           Taught yourself a new language, framework, or ...   \n",
       "10                                                        NaN   \n",
       "\n",
       "                             OrgSize  \\\n",
       "Respondent                             \n",
       "8                                NaN   \n",
       "10          10,000 or more employees   \n",
       "\n",
       "                                                      DevType YearsCode  \\\n",
       "Respondent                                                                \n",
       "8              Developer, back-end;Engineer, site reliability         8   \n",
       "10          Data or business analyst;Data scientist or mac...        12   \n",
       "\n",
       "           Age1stCode YearsCodePro              CareerSat  \\\n",
       "Respondent                                                  \n",
       "8                  16          NaN                    NaN   \n",
       "10                 20           10  Slightly dissatisfied   \n",
       "\n",
       "                           JobSat            MgrIdiot MgrMoney MgrWant  \\\n",
       "Respondent                                                               \n",
       "8                             NaN                 NaN      NaN     NaN   \n",
       "10          Slightly dissatisfied  Somewhat confident      Yes     Yes   \n",
       "\n",
       "                                                      JobSeek   LastHireDate  \\\n",
       "Respondent                                                                     \n",
       "8                                                         NaN            NaN   \n",
       "10          I’m not actively looking, but I am open to new...  3-4 years ago   \n",
       "\n",
       "           LastInt FizzBuzz  \\\n",
       "Respondent                    \n",
       "8              NaN      NaN   \n",
       "10             NaN       No   \n",
       "\n",
       "                                                   JobFactors ResumeUpdate  \\\n",
       "Respondent                                                                   \n",
       "8                                                         NaN          NaN   \n",
       "10          Languages, frameworks, and other technologies ...          NaN   \n",
       "\n",
       "           CurrencySymbol  CurrencyDesc  CompTotal CompFreq  ConvertedComp  \\\n",
       "Respondent                                                                   \n",
       "8                     NaN           NaN        NaN      NaN            NaN   \n",
       "10                    INR  Indian rupee   950000.0   Yearly        13293.0   \n",
       "\n",
       "            WorkWeekHrs                                           WorkPlan  \\\n",
       "Respondent                                                                   \n",
       "8                   NaN                                                NaN   \n",
       "10                 70.0  There's no schedule or spec; I work on what se...   \n",
       "\n",
       "           WorkChallenge             WorkRemote WorkLoc             ImpSyn  \\\n",
       "Respondent                                                                   \n",
       "8                    NaN                    NaN     NaN                NaN   \n",
       "10                   NaN  A few days each month    Home  Far above average   \n",
       "\n",
       "                                            CodeRev  CodeRevHrs  \\\n",
       "Respondent                                                        \n",
       "8                                               NaN         NaN   \n",
       "10          Yes, because I see value in code review         4.0   \n",
       "\n",
       "                                UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                           \n",
       "8                                     NaN         NaN          NaN   \n",
       "10          Yes, it's part of our process         NaN          NaN   \n",
       "\n",
       "                                           LanguageWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...   \n",
       "10                              C#;Go;JavaScript;Python;R;SQL   \n",
       "\n",
       "                                       LanguageDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...   \n",
       "10                       C#;Go;JavaScript;Kotlin;Python;R;SQL   \n",
       "\n",
       "                                           DatabaseWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...   \n",
       "10          Elasticsearch;MongoDB;Microsoft SQL Server;MyS...   \n",
       "\n",
       "                                       DatabaseDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...   \n",
       "10                 Elasticsearch;MongoDB;Microsoft SQL Server   \n",
       "\n",
       "                             PlatformWorkedWith  \\\n",
       "Respondent                                        \n",
       "8           AWS;Docker;Heroku;Linux;MacOS;Slack   \n",
       "10                                Linux;Windows   \n",
       "\n",
       "                                       PlatformDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Android;Arduino;AWS;Docker;Google Cloud Platfo...   \n",
       "10                         Android;Linux;Raspberry Pi;Windows   \n",
       "\n",
       "                                           WebFrameWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8                               Express;Flask;React.js;Spring   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "\n",
       "                                       WebFrameDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8                        Django;Express;Flask;React.js;Vue.js   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "\n",
       "                           MiscTechWorkedWith  \\\n",
       "Respondent                                      \n",
       "8                       Hadoop;Node.js;Pandas   \n",
       "10          .NET;Node.js;Pandas;Torch/PyTorch   \n",
       "\n",
       "                                       MiscTechDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...   \n",
       "10                      .NET;Node.js;TensorFlow;Torch/PyTorch   \n",
       "\n",
       "                                                   DevEnviron        OpSys  \\\n",
       "Respondent                                                                   \n",
       "8           Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...  Linux-based   \n",
       "10          Android Studio;Eclipse;IPython / Jupyter;Notep...      Windows   \n",
       "\n",
       "                                                   Containers BlockchainOrg  \\\n",
       "Respondent                                                                    \n",
       "8           Development;Testing;Production;Outside of work...           NaN   \n",
       "10                                                        NaN    Not at all   \n",
       "\n",
       "                                                 BlockchainIs BetterLife  \\\n",
       "Respondent                                                                 \n",
       "8           Useful across many domains and could change ma...        Yes   \n",
       "10          Useful for immutable record keeping outside of...         No   \n",
       "\n",
       "           ITperson OffOn SocialMedia              Extraversion   ScreenName  \\\n",
       "Respondent                                                                     \n",
       "8              SIGH   Yes     YouTube  In real life (in person)       Handle   \n",
       "10              Yes   Yes     YouTube                   Neither  Screen Name   \n",
       "\n",
       "           SOVisit1st             SOVisitFreq  \\\n",
       "Respondent                                      \n",
       "8                2012    A few times per week   \n",
       "10                NaN  Multiple times per day   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "8           Find answers to specific questions;Learn how t...   \n",
       "10          Find answers to specific questions;Get a sense...   \n",
       "\n",
       "                       SOFindAnswer                         SOTimeSaved  \\\n",
       "Respondent                                                                \n",
       "8           Less than once per week  Stack Overflow was slightly faster   \n",
       "10               3-5 times per week            They were about the same   \n",
       "\n",
       "            SOHowMuchTime SOAccount                           SOPartFreq  \\\n",
       "Respondent                                                                 \n",
       "8           11-30 minutes       Yes  Less than once per month or monthly   \n",
       "10                    NaN       Yes      A few times per month or weekly   \n",
       "\n",
       "           SOJobs                             EntTeams           SOComm  \\\n",
       "Respondent                                                                \n",
       "8             Yes  No, and I don't know what those are  Yes, definitely   \n",
       "10            Yes  No, and I don't know what those are    Yes, somewhat   \n",
       "\n",
       "                                       WelcomeChange  \\\n",
       "Respondent                                             \n",
       "8              A lot more welcome now than last year   \n",
       "10          Somewhat less welcome now than last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "8           Tech articles written by other developers;Indu...  24.0    Man   \n",
       "10          Tech articles written by other developers;Tech...   NaN    NaN   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "8             No  Straight / Heterosexual       NaN        NaN   \n",
       "10           NaN                      NaN       NaN        Yes   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "8           Appropriate in length  Neither easy nor difficult  \n",
       "10                       Too long                   Difficult  "
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = df['Country'] == 'India'\n",
    "df[filt].head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but `group by ` works for multiple countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SocialMedia\n",
       "WhatsApp    2990\n",
       "YouTube     1820\n",
       "LinkedIn     955\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[filt]['SocialMedia'].value_counts().head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if we run this on dataframe `groupby` object, it will give result for all country groups\n",
    "\n",
    "> if we apply function on a single country it will return for specific country, but if we apply it on dataframe group by object it applies to the whole country groups***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country      SocialMedia             \n",
       "Afghanistan  Facebook                    15\n",
       "             YouTube                      9\n",
       "             I don't use social media     6\n",
       "             WhatsApp                     4\n",
       "             Instagram                    1\n",
       "             LinkedIn                     1\n",
       "             Twitter                      1\n",
       "Albania      WhatsApp                    18\n",
       "             Facebook                    16\n",
       "             Instagram                   13\n",
       "             YouTube                     10\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['SocialMedia'].value_counts().head(11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get result for one country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SocialMedia\n",
       "WhatsApp    2990\n",
       "YouTube     1820\n",
       "LinkedIn     955\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['SocialMedia'].value_counts().loc['India'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SocialMedia\n",
       "WhatsApp    0.342379\n",
       "YouTube     0.208405\n",
       "LinkedIn    0.109355\n",
       "Name: proportion, dtype: float64"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['SocialMedia'].value_counts(normalize=True).loc['India'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use this to get mean median and related aggregate functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "Afghanistan     6222.0\n",
       "Albania        10818.0\n",
       "Algeria         7878.0\n",
       "Name: ConvertedComp, dtype: float64"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['ConvertedComp'].median().head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get a single country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "63016.0"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['ConvertedComp'].median().loc['Germany']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run multiple aggregate function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>median</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Afghanistan</th>\n",
       "      <td>101953.333333</td>\n",
       "      <td>6222.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Albania</th>\n",
       "      <td>21833.700000</td>\n",
       "      <td>10818.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Algeria</th>\n",
       "      <td>34924.047619</td>\n",
       "      <td>7878.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      mean   median\n",
       "Country                            \n",
       "Afghanistan  101953.333333   6222.0\n",
       "Albania       21833.700000  10818.0\n",
       "Algeria       34924.047619   7878.0"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['ConvertedComp'].agg(['mean', 'median']).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get for single country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mean      134018.564909\n",
       "median     68705.000000\n",
       "Name: Canada, dtype: float64"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['ConvertedComp'].agg(['mean', 'median']).loc['Canada']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get how many from a country uses python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "8         True\n",
       "10        True\n",
       "15       False\n",
       "50        True\n",
       "65       False\n",
       "         ...  \n",
       "77339    False\n",
       "79795     True\n",
       "83862    False\n",
       "84299     True\n",
       "86012    False\n",
       "Name: LanguageWorkedWith, Length: 9061, dtype: object"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = df['Country'] == 'India'\n",
    "df.loc[filt]['LanguageWorkedWith'].str.contains('Python')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to count the number of people who knows python we can use `.sum()` function. it counts True as 1 and flase as 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3105"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[filt]['LanguageWorkedWith'].str.contains('Python').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "# country_grp['LanguageWorkedWith'].str.contains('Python').sum()\n",
    "#AttributeError: 'SeriesGroupBy' object has no attribute 'str' \n",
    "\n",
    "# so we need to apply it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "Afghanistan     8\n",
       "Albania        23\n",
       "Algeria        40\n",
       "Name: LanguageWorkedWith, dtype: int64"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum()).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives result for all country\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get the <mark>percentage</mark> of people each country who siad they know python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "United States    20949\n",
       "India             9061\n",
       "Germany           5866\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_respondents = df['Country'].value_counts()\n",
    "country_respondents.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Country\n",
       "Afghanistan     8\n",
       "Albania        23\n",
       "Algeria        40\n",
       "Name: LanguageWorkedWith, dtype: int64"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_uses_python = country_grp['LanguageWorkedWith'].apply(lambda x: x.str.contains('Python').sum())\n",
    "country_uses_python.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # MY TEST\n",
    "# ts = pd.DataFrame([country_respondents - country_uses_python])\n",
    "# ts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to combine these two \n",
    "\n",
    "we can combine more than one series together with pandas `concat()` function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "have to use `axis='columns'` because by default it concat rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>United States</th>\n",
       "      <td>20949</td>\n",
       "      <td>10083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>India</th>\n",
       "      <td>9061</td>\n",
       "      <td>3105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Germany</th>\n",
       "      <td>5866</td>\n",
       "      <td>2451</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               count  LanguageWorkedWith\n",
       "Country                                 \n",
       "United States  20949               10083\n",
       "India           9061                3105\n",
       "Germany         5866                2451"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_df = pd.concat([country_respondents, country_uses_python], axis='columns') #sort False by default now\n",
    "python_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total Respondents</th>\n",
       "      <th>Knows Python</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>United States</th>\n",
       "      <td>20949</td>\n",
       "      <td>10083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>India</th>\n",
       "      <td>9061</td>\n",
       "      <td>3105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Germany</th>\n",
       "      <td>5866</td>\n",
       "      <td>2451</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Total Respondents  Knows Python\n",
       "Country                                       \n",
       "United States              20949         10083\n",
       "India                       9061          3105\n",
       "Germany                     5866          2451"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_df.rename(columns={'count' : 'Total Respondents', 'LanguageWorkedWith': 'Knows Python'}, inplace=True)\n",
    "python_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now to calculate the % we can create new column and put the result of calulation in it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total Respondents</th>\n",
       "      <th>Knows Python</th>\n",
       "      <th>Pct_Knows_Python</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>United States</th>\n",
       "      <td>20949</td>\n",
       "      <td>10083</td>\n",
       "      <td>48.131176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>India</th>\n",
       "      <td>9061</td>\n",
       "      <td>3105</td>\n",
       "      <td>34.267741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Germany</th>\n",
       "      <td>5866</td>\n",
       "      <td>2451</td>\n",
       "      <td>41.783157</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Total Respondents  Knows Python  Pct_Knows_Python\n",
       "Country                                                         \n",
       "United States              20949         10083         48.131176\n",
       "India                       9061          3105         34.267741\n",
       "Germany                     5866          2451         41.783157"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_df['Pct_Knows_Python'] = (python_df['Knows Python'] / python_df['Total Respondents']) * 100\n",
    "python_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total Respondents</th>\n",
       "      <th>Knows Python</th>\n",
       "      <th>Pct_Knows_Python</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Country</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Sao Tome and Principe</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Timor-Leste</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dominica</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Niger</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>100.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Turkmenistan</th>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>85.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mauritania</th>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>71.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Guinea</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>66.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bahamas</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>66.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Guyana</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>66.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Uganda</th>\n",
       "      <td>72</td>\n",
       "      <td>47</td>\n",
       "      <td>65.277778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iceland</th>\n",
       "      <td>49</td>\n",
       "      <td>30</td>\n",
       "      <td>61.224490</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Total Respondents  Knows Python  Pct_Knows_Python\n",
       "Country                                                                 \n",
       "Sao Tome and Principe                  1             1        100.000000\n",
       "Timor-Leste                            1             1        100.000000\n",
       "Dominica                               1             1        100.000000\n",
       "Niger                                  1             1        100.000000\n",
       "Turkmenistan                           7             6         85.714286\n",
       "Mauritania                             7             5         71.428571\n",
       "Guinea                                 3             2         66.666667\n",
       "Bahamas                                3             2         66.666667\n",
       "Guyana                                 3             2         66.666667\n",
       "Uganda                                72            47         65.277778\n",
       "Iceland                               49            30         61.224490"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_df.sort_values(by='Pct_Knows_Python', ascending=False, inplace=True)\n",
    "python_df.head(11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Total Respondents    391.000000\n",
       "Knows Python         182.000000\n",
       "Pct_Knows_Python      46.547315\n",
       "Name: Japan, dtype: float64"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_df.loc['Japan']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 09 - Cleaning Data - Casting Datatypes and Handling Missing Values\n",
    "\n",
    "https://youtu.be/KdmPHEnPJPs?si=xkDlFOVuQT2Q-Kaq\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#Setup for learning data\n",
    "\n",
    "people = {\n",
    "    'first': ['Corey', 'Jane', 'John', 'Chris', np.nan, None, 'NA'], \n",
    "    'last': ['Schafer', 'Doe', 'Doe', 'Schafer', np.nan, np.nan, 'Missing'], \n",
    "    'email': ['CoreyMSchafer@gmail.com', 'JaneDoe@email.com', 'JohnDoe@email.com', None, np.nan, 'Anonymous@email.com', 'NA'],\n",
    "    'age': ['33', '55', '63', '36', None, None, 'Missing']\n",
    "}\n",
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dft.replace('NA', np.nan, inplace=True)\n",
    "# dft.replace('Missing', np.nan, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>None</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "3  Chris  Schafer                     None       36\n",
       "4    NaN      NaN                      NaN     None\n",
       "5   None      NaN      Anonymous@email.com     None\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "if the people dont have their first name, last name, and age  \n",
    "we gonna drop them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can <mark>drop na values</mark> with `dropna()` methos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this drops any rows that has missing value. the bottom rows are custom missing values.\n",
    "which is actually a string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "it has some default arguments. they are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='index', how='any')\n",
    "#axis = index / columns\n",
    "#how = any /  all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>None</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "3  Chris  Schafer                     None       36\n",
       "5   None      NaN      Anonymous@email.com     None\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='index', how='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>None</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "3  Chris  Schafer                     None       36\n",
       "4    NaN      NaN                      NaN     None\n",
       "5   None      NaN      Anonymous@email.com     None\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='columns', how='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "no change, but if we change this to any, which is the default value, we lose the whole dataframe because we have one complete row missing across columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: [0, 1, 2]"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='columns', how='any').head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can choose to drop only if specific mentioned columns has missing values with `subset=['column_names']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "5   None      NaN      Anonymous@email.com     None\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='index', how='any', subset=['email'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='index', how='any', subset=['last', 'email'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`how = 'any' `--> is kinda like AND, if either first or last is missing its dropped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>None</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "      <td>NA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email      age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1   Jane      Doe        JaneDoe@email.com       55\n",
       "2   John      Doe        JohnDoe@email.com       63\n",
       "3  Chris  Schafer                     None       36\n",
       "5   None      NaN      Anonymous@email.com     None\n",
       "6     NA  Missing                       NA  Missing"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dropna(axis='index', how='all', subset=['last', 'email'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`how = 'all` --> is kinda like OR if 'last' or 'email' is missing it will not be dropped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Handling custom missing values that are not `NaN` \n",
    "\n",
    "we can <mark>replace manual Missing values to NaN</mark> with `.replace()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>None</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first     last                    email   age\n",
       "0  Corey  Schafer  CoreyMSchafer@gmail.com    33\n",
       "1   Jane      Doe        JaneDoe@email.com    55\n",
       "2   John      Doe        JohnDoe@email.com    63\n",
       "3  Chris  Schafer                     None    36\n",
       "4    NaN      NaN                      NaN  None\n",
       "5   None      NaN      Anonymous@email.com  None\n",
       "6    NaN      NaN                      NaN   NaN"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.replace('NA', np.nan, inplace=True)\n",
    "dft.replace('Missing', np.nan, inplace=True)\n",
    "dft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now if we run those again drop will wosk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To <mark>check if its `NaN` </mark> we use `.isna()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   first   last  email    age\n",
       "0  False  False  False  False\n",
       "1  False  False  False  False\n",
       "2  False  False  False  False\n",
       "3  False  False   True  False\n",
       "4   True   True   True   True\n",
       "5   True   True  False   True\n",
       "6   True   True   True   True"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.isna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sometimes with number we have to <mark>replace NaN</mark> with 0 or similiar cases\n",
    "\n",
    "in that case we can use `.fillna()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Corey</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>CoreyMSchafer@gmail.com</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chris</td>\n",
       "      <td>Schafer</td>\n",
       "      <td>MISSING</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "      <td>Anonymous@email.com</td>\n",
       "      <td>MISSING</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "      <td>MISSING</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     first     last                    email      age\n",
       "0    Corey  Schafer  CoreyMSchafer@gmail.com       33\n",
       "1     Jane      Doe        JaneDoe@email.com       55\n",
       "2     John      Doe        JohnDoe@email.com       63\n",
       "3    Chris  Schafer                  MISSING       36\n",
       "4  MISSING  MISSING                  MISSING  MISSING\n",
       "5  MISSING  MISSING      Anonymous@email.com  MISSING\n",
       "6  MISSING  MISSING                  MISSING  MISSING"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.fillna('MISSING')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#### Casting data types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "age column in test data is 'string' to find average and other stats we have to convert it to 'int'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first    object\n",
       "last     object\n",
       "email    object\n",
       "age      object\n",
       "dtype: object"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dft['age'].mean() # gives error\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark> if we have NaN values in columns the we have to convert it into 'float' datatype.\n",
    "</mark>\n",
    "since NaN is a float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "float"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we convert types with `.astype(int)` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dft['age'] = dft['age'].astype(int) #this gives error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "because age column has 'NaN' values the above gives error.  \n",
    "\n",
    "we can convert the 'NaN' to 0  \n",
    "but converting them to 0 gives problem with averages and other calculations\n",
    "\n",
    "so we can convert them to 'float', so the NaN remains as missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first     object\n",
       "last      object\n",
       "email     object\n",
       "age      float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['age'] = dft['age'].astype(float) #this works\n",
    "dft.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "46.75"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['age'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can convert the whole dataframe type too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dft.astype()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SO DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup for Real world data.\n",
    "###### uncomment when in need to avoid conflicting code autocomplete suggestions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "na_vals = ['NA', 'Missing']\n",
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent', na_values=na_vals )\n",
    "#                                              # changes NA values to `NaN` ⬆⬆⬆⬆⬆⬆  \n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "using `na_values=na_vals` when loading CSV changes NA values to `NaN`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, desktop or enterprise applications;...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I am actively looking for a job</td>\n",
       "      <td>I've never had a job</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Financial performance or funding status of the...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C++;HTML/CSS;Python</td>\n",
       "      <td>C++;HTML/CSS;JavaScript;SQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Django</td>\n",
       "      <td>Django</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Atom;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Instagram</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>No, I knew that Stack Overflow had a job board...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I am not primarily a developer, but I write co...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>Thailand</td>\n",
       "      <td>No</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Web development or web design</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>100 to 499 employees</td>\n",
       "      <td>Designer;Developer, back-end;Developer, front-...</td>\n",
       "      <td>3</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>Slightly satisfied</td>\n",
       "      <td>Slightly satisfied</td>\n",
       "      <td>Not at all confident</td>\n",
       "      <td>Not sure</td>\n",
       "      <td>Not sure</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>1-2 years ago</td>\n",
       "      <td>Interview with people in peer roles</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>I was preparing for a job search</td>\n",
       "      <td>THB</td>\n",
       "      <td>Thai baht</td>\n",
       "      <td>23000.0</td>\n",
       "      <td>Monthly</td>\n",
       "      <td>8820.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>Distracting work environment;Inadequate access...</td>\n",
       "      <td>Less than once per month / Never</td>\n",
       "      <td>Home</td>\n",
       "      <td>Average</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, but I think we should</td>\n",
       "      <td>Not sure</td>\n",
       "      <td>I have little or no influence</td>\n",
       "      <td>HTML/CSS</td>\n",
       "      <td>Elixir;HTML/CSS</td>\n",
       "      <td>PostgreSQL</td>\n",
       "      <td>PostgreSQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Other(s):</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Vim;Visual Studio Code</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Reddit</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Username</td>\n",
       "      <td>2011</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>6-10 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, I've heard of them, but I am not part of a...</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech meetups or events in your area;Courses on...</td>\n",
       "      <td>28.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   MainBranch Hobbyist  \\\n",
       "Respondent                                                               \n",
       "1                      I am a student who is learning to code      Yes   \n",
       "2                      I am a student who is learning to code       No   \n",
       "3           I am not primarily a developer, but I write co...      Yes   \n",
       "\n",
       "                        OpenSourcer  \\\n",
       "Respondent                            \n",
       "1                             Never   \n",
       "2           Less than once per year   \n",
       "3                             Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "2           The quality of OSS and closed source software ...   \n",
       "3           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment                 Country  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work          United Kingdom   \n",
       "2               Not employed, but looking for work  Bosnia and Herzegovina   \n",
       "3                               Employed full-time                Thailand   \n",
       "\n",
       "                   Student                                            EdLevel  \\\n",
       "Respondent                                                                      \n",
       "1                       No                          Primary/elementary school   \n",
       "2           Yes, full-time  Secondary school (e.g. American high school, G...   \n",
       "3                       No           Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "\n",
       "                           UndergradMajor  \\\n",
       "Respondent                                  \n",
       "1                                     NaN   \n",
       "2                                     NaN   \n",
       "3           Web development or web design   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "1           Taught yourself a new language, framework, or ...   \n",
       "2           Taken an online course in programming or softw...   \n",
       "3           Taught yourself a new language, framework, or ...   \n",
       "\n",
       "                         OrgSize  \\\n",
       "Respondent                         \n",
       "1                            NaN   \n",
       "2                            NaN   \n",
       "3           100 to 499 employees   \n",
       "\n",
       "                                                      DevType YearsCode  \\\n",
       "Respondent                                                                \n",
       "1                                                         NaN         4   \n",
       "2           Developer, desktop or enterprise applications;...       NaN   \n",
       "3           Designer;Developer, back-end;Developer, front-...         3   \n",
       "\n",
       "           Age1stCode YearsCodePro           CareerSat              JobSat  \\\n",
       "Respondent                                                                   \n",
       "1                  10          NaN                 NaN                 NaN   \n",
       "2                  17          NaN                 NaN                 NaN   \n",
       "3                  22            1  Slightly satisfied  Slightly satisfied   \n",
       "\n",
       "                        MgrIdiot  MgrMoney   MgrWant  \\\n",
       "Respondent                                             \n",
       "1                            NaN       NaN       NaN   \n",
       "2                            NaN       NaN       NaN   \n",
       "3           Not at all confident  Not sure  Not sure   \n",
       "\n",
       "                                                      JobSeek  \\\n",
       "Respondent                                                      \n",
       "1                                                         NaN   \n",
       "2                             I am actively looking for a job   \n",
       "3           I’m not actively looking, but I am open to new...   \n",
       "\n",
       "                    LastHireDate                              LastInt  \\\n",
       "Respondent                                                              \n",
       "1                            NaN                                  NaN   \n",
       "2           I've never had a job                                  NaN   \n",
       "3                  1-2 years ago  Interview with people in peer roles   \n",
       "\n",
       "           FizzBuzz                                         JobFactors  \\\n",
       "Respondent                                                               \n",
       "1               NaN                                                NaN   \n",
       "2               NaN  Financial performance or funding status of the...   \n",
       "3                No  Languages, frameworks, and other technologies ...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "1                                                         NaN            NaN   \n",
       "2           Something else changed (education, award, medi...            NaN   \n",
       "3                            I was preparing for a job search            THB   \n",
       "\n",
       "           CurrencyDesc  CompTotal CompFreq  ConvertedComp  WorkWeekHrs  \\\n",
       "Respondent                                                                \n",
       "1                   NaN        NaN      NaN            NaN          NaN   \n",
       "2                   NaN        NaN      NaN            NaN          NaN   \n",
       "3             Thai baht    23000.0  Monthly         8820.0         40.0   \n",
       "\n",
       "                                                     WorkPlan  \\\n",
       "Respondent                                                      \n",
       "1                                                         NaN   \n",
       "2                                                         NaN   \n",
       "3           There's no schedule or spec; I work on what se...   \n",
       "\n",
       "                                                WorkChallenge  \\\n",
       "Respondent                                                      \n",
       "1                                                         NaN   \n",
       "2                                                         NaN   \n",
       "3           Distracting work environment;Inadequate access...   \n",
       "\n",
       "                                  WorkRemote WorkLoc   ImpSyn CodeRev  \\\n",
       "Respondent                                                              \n",
       "1                                        NaN     NaN      NaN     NaN   \n",
       "2                                        NaN     NaN      NaN     NaN   \n",
       "3           Less than once per month / Never    Home  Average      No   \n",
       "\n",
       "            CodeRevHrs                  UnitTests PurchaseHow  \\\n",
       "Respondent                                                      \n",
       "1                  NaN                        NaN         NaN   \n",
       "2                  NaN                        NaN         NaN   \n",
       "3                  NaN  No, but I think we should    Not sure   \n",
       "\n",
       "                             PurchaseWhat               LanguageWorkedWith  \\\n",
       "Respondent                                                                   \n",
       "1                                     NaN  HTML/CSS;Java;JavaScript;Python   \n",
       "2                                     NaN              C++;HTML/CSS;Python   \n",
       "3           I have little or no influence                         HTML/CSS   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "2                               C++;HTML/CSS;JavaScript;SQL   \n",
       "3                                           Elixir;HTML/CSS   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "2                         NaN                  MySQL            Windows   \n",
       "3                  PostgreSQL             PostgreSQL                NaN   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "2                           Windows             Django                 Django   \n",
       "3                               NaN                NaN              Other(s):   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "2                         NaN                    NaN   \n",
       "3                         NaN                    NaN   \n",
       "\n",
       "                            DevEnviron        OpSys               Containers  \\\n",
       "Respondent                                                                     \n",
       "1           IntelliJ;Notepad++;PyCharm      Windows  I do not use containers   \n",
       "2                         Atom;PyCharm      Windows  I do not use containers   \n",
       "3               Vim;Visual Studio Code  Linux-based  I do not use containers   \n",
       "\n",
       "           BlockchainOrg                                       BlockchainIs  \\\n",
       "Respondent                                                                    \n",
       "1                    NaN                                                NaN   \n",
       "2                    NaN  Useful across many domains and could change ma...   \n",
       "3                    NaN                                                NaN   \n",
       "\n",
       "           BetterLife                                  ITperson OffOn  \\\n",
       "Respondent                                                              \n",
       "1                 Yes  Fortunately, someone else has that title   Yes   \n",
       "2                 Yes                                       Yes   Yes   \n",
       "3                 Yes                                       Yes   Yes   \n",
       "\n",
       "           SocialMedia              Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                                               \n",
       "1              Twitter                    Online   Username       2017   \n",
       "2            Instagram                    Online   Username       2017   \n",
       "3               Reddit  In real life (in person)   Username       2011   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "2                     Daily or almost daily   \n",
       "3                      A few times per week   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "2           Find answers to specific questions;Learn how t...   \n",
       "3           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                   SOFindAnswer                     SOTimeSaved  \\\n",
       "Respondent                                                        \n",
       "1            3-5 times per week  Stack Overflow was much faster   \n",
       "2            3-5 times per week  Stack Overflow was much faster   \n",
       "3           6-10 times per week        They were about the same   \n",
       "\n",
       "            SOHowMuchTime SOAccount                           SOPartFreq  \\\n",
       "Respondent                                                                 \n",
       "1           31-60 minutes        No                                  NaN   \n",
       "2           11-30 minutes       Yes      A few times per month or weekly   \n",
       "3                     NaN       Yes  Less than once per month or monthly   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "2           No, I knew that Stack Overflow had a job board...   \n",
       "3                                                         Yes   \n",
       "\n",
       "                                                     EntTeams         SOComm  \\\n",
       "Respondent                                                                     \n",
       "1                         No, and I don't know what those are        Neutral   \n",
       "2                         No, and I don't know what those are  Yes, somewhat   \n",
       "3           No, I've heard of them, but I am not part of a...        Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "2           Just as welcome now as I felt last year   \n",
       "3           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "2           Tech articles written by other developers;Indu...  19.0    Man   \n",
       "3           Tech meetups or events in your area;Courses on...  28.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "2             No  Straight / Heterosexual       NaN         No   \n",
       "3             No  Straight / Heterosexual       NaN        Yes   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  \n",
       "2           Appropriate in length  Neither easy nor difficult  \n",
       "3           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "problems with casting values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "1      4\n",
       "2    NaN\n",
       "3      3\n",
       "Name: YearsCode, dtype: object"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['YearsCode'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df['YearsCode'].mean()\n",
    "# will give error since the columns are Int as 'strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df['YearsCode'].astype(float).mean() \n",
    "# int gives error so we converted to float\n",
    "#still gives error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df['YearsCode'] = df['YearsCode'].astype(float) #like in video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ValueError: could not convert string to float: 'Less than 1 year'  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>TO get unique values of a series, using `.unique()` method </mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['4', nan, '3', '16', '13', '6', '8', '12', '2', '5', '17', '10',\n",
       "       '14', '35', '7', 'Less than 1 year', '30', '9', '26', '40', '19',\n",
       "       '15', '20', '28', '25', '1', '22', '11', '33', '50', '41', '18',\n",
       "       '34', '24', '23', '42', '27', '21', '36', '32', '39', '38', '31',\n",
       "       '37', 'More than 50 years', '29', '44', '45', '48', '46', '43',\n",
       "       '47', '49'], dtype=object)"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['YearsCode'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Less than 1 year'  will be replaced with 0\n",
    "'More than 50 years' replace with 51, this can affect result slightly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['YearsCode'].replace('Less than 1 year', 0, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['YearsCode'].replace('More than 50 years', 51, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['4', nan, '3', '16', '13', '6', '8', '12', '2', '5', '17', '10',\n",
       "       '14', '35', '7', 0, '30', '9', '26', '40', '19', '15', '20', '28',\n",
       "       '25', '1', '22', '11', '33', '50', '41', '18', '34', '24', '23',\n",
       "       '42', '27', '21', '36', '32', '39', '38', '31', '37', 51, '29',\n",
       "       '44', '45', '48', '46', '43', '47', '49'], dtype=object)"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['YearsCode'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now we can convert the type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent\n",
       "1         4.0\n",
       "2         NaN\n",
       "3         3.0\n",
       "4         3.0\n",
       "5        16.0\n",
       "         ... \n",
       "88377     NaN\n",
       "88601     NaN\n",
       "88802     NaN\n",
       "88816     NaN\n",
       "88863     8.0\n",
       "Name: YearsCode, Length: 88883, dtype: float64"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['YearsCode'] = df['YearsCode'].astype(float)\n",
    "df['YearsCode']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11.662114216834588"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['YearsCode'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9.0"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['YearsCode'].median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10 - Working with Dates and Time Series Data\n",
    "\n",
    "https://youtu.be/UFuo7EHI8zc?si=BXHZH4B_hKEq-gGP\n",
    "\n",
    "[Offset Aliases - Pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliases)\n",
    "\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('data/ETH_1h.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-03-13 08-PM</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-03-13 07-PM</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>119.51</td>\n",
       "      <td>132.02</td>\n",
       "      <td>117.10</td>\n",
       "      <td>129.94</td>\n",
       "      <td>7579741.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-03-13 06-PM</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.47</td>\n",
       "      <td>124.85</td>\n",
       "      <td>115.50</td>\n",
       "      <td>119.51</td>\n",
       "      <td>4898735.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2020-03-13 05-PM</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.08</td>\n",
       "      <td>127.42</td>\n",
       "      <td>121.63</td>\n",
       "      <td>124.47</td>\n",
       "      <td>2753450.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2020-03-13 04-PM</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.85</td>\n",
       "      <td>129.51</td>\n",
       "      <td>120.17</td>\n",
       "      <td>124.08</td>\n",
       "      <td>4461424.71</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Date  Symbol    Open    High     Low   Close      Volume\n",
       "0  2020-03-13 08-PM  ETHUSD  129.94  131.82  126.87  128.71  1940673.93\n",
       "1  2020-03-13 07-PM  ETHUSD  119.51  132.02  117.10  129.94  7579741.09\n",
       "2  2020-03-13 06-PM  ETHUSD  124.47  124.85  115.50  119.51  4898735.81\n",
       "3  2020-03-13 05-PM  ETHUSD  124.08  127.42  121.63  124.47  2753450.92\n",
       "4  2020-03-13 04-PM  ETHUSD  124.85  129.51  120.17  124.08  4461424.71"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas dateTime method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2020-03-13 08-PM'"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'Date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this wonr work because we are reading date as 'string' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df.loc[0, 'Date'].day_name()\n",
    "# AttributeError: 'str' object has no attribute 'day_name'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so first we have to convert to DateTime using pandas `.to_datetime()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df['Date'] = pd.to_datetime(df['Date'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This wont work either because our dates are not in default format so we have to use `format='  '` and give the proper format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Date'] = pd.to_datetime(df['Date'], format='%Y-%m-%d %I-%p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       2020-03-13 20:00:00\n",
       "1       2020-03-13 19:00:00\n",
       "2       2020-03-13 18:00:00\n",
       "3       2020-03-13 17:00:00\n",
       "4       2020-03-13 16:00:00\n",
       "                ...        \n",
       "23669   2017-07-01 15:00:00\n",
       "23670   2017-07-01 14:00:00\n",
       "23671   2017-07-01 13:00:00\n",
       "23672   2017-07-01 12:00:00\n",
       "23673   2017-07-01 11:00:00\n",
       "Name: Date, Length: 23674, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now DateTime functions work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Friday'"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'Date'].day_name()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can also <mark>parse date at time of loading CSV</mark> with `parse_dates=' '` argument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "# d_parser = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %I-%p') --> Deprecated\n",
    "\n",
    "d_parser = lambda x: datetime.strptime(x, '%Y-%m-%d %I-%p')    # --> another way but soon will be deprecated\n",
    "# dfx = pd.read_csv('data/ETH_1h.csv', parse_dates=['Date'], date_parser=d_parser) # --> futurewarning\n",
    "\n",
    "dfx = pd.read_csv('data/ETH_1h.csv', parse_dates=['Date'], date_format='%Y-%m-%d %I-%p') # New method\n",
    "#⬆ so it wont mess with our df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Tip:</b> We can now directly parse without using a parser function</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-03-13 20:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-03-13 19:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>119.51</td>\n",
       "      <td>132.02</td>\n",
       "      <td>117.10</td>\n",
       "      <td>129.94</td>\n",
       "      <td>7579741.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-03-13 18:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.47</td>\n",
       "      <td>124.85</td>\n",
       "      <td>115.50</td>\n",
       "      <td>119.51</td>\n",
       "      <td>4898735.81</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  Symbol    Open    High     Low   Close      Volume\n",
       "0 2020-03-13 20:00:00  ETHUSD  129.94  131.82  126.87  128.71  1940673.93\n",
       "1 2020-03-13 19:00:00  ETHUSD  119.51  132.02  117.10  129.94  7579741.09\n",
       "2 2020-03-13 18:00:00  ETHUSD  124.47  124.85  115.50  119.51  4898735.81"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfx.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run a DateTime methods for entire series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          Friday\n",
       "1          Friday\n",
       "2          Friday\n",
       "3          Friday\n",
       "4          Friday\n",
       "           ...   \n",
       "23669    Saturday\n",
       "23670    Saturday\n",
       "23671    Saturday\n",
       "23672    Saturday\n",
       "23673    Saturday\n",
       "Name: Date, Length: 23674, dtype: object"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date'].dt.day_name()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>DayOfWeek</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-03-13 20:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "      <td>Friday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-03-13 19:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>119.51</td>\n",
       "      <td>132.02</td>\n",
       "      <td>117.10</td>\n",
       "      <td>129.94</td>\n",
       "      <td>7579741.09</td>\n",
       "      <td>Friday</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-03-13 18:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.47</td>\n",
       "      <td>124.85</td>\n",
       "      <td>115.50</td>\n",
       "      <td>119.51</td>\n",
       "      <td>4898735.81</td>\n",
       "      <td>Friday</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  Symbol    Open    High     Low   Close      Volume  \\\n",
       "0 2020-03-13 20:00:00  ETHUSD  129.94  131.82  126.87  128.71  1940673.93   \n",
       "1 2020-03-13 19:00:00  ETHUSD  119.51  132.02  117.10  129.94  7579741.09   \n",
       "2 2020-03-13 18:00:00  ETHUSD  124.47  124.85  115.50  119.51  4898735.81   \n",
       "\n",
       "  DayOfWeek  \n",
       "0    Friday  \n",
       "1    Friday  \n",
       "2    Friday  "
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['DayOfWeek'] = df['Date'].dt.day_name()\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop(columns='DayOfWeek', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get the earliest and latest date we can us `min()` and `max()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2020-03-13 20:00:00')"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date'].max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2017-07-01 11:00:00')"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timedelta('986 days 09:00:00')"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Date'].max() - df['Date'].min()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "we can use filters too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-03-13 20:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2020-03-13 19:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>119.51</td>\n",
       "      <td>132.02</td>\n",
       "      <td>117.10</td>\n",
       "      <td>129.94</td>\n",
       "      <td>7579741.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020-03-13 18:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.47</td>\n",
       "      <td>124.85</td>\n",
       "      <td>115.50</td>\n",
       "      <td>119.51</td>\n",
       "      <td>4898735.81</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  Symbol    Open    High     Low   Close      Volume\n",
       "0 2020-03-13 20:00:00  ETHUSD  129.94  131.82  126.87  128.71  1940673.93\n",
       "1 2020-03-13 19:00:00  ETHUSD  119.51  132.02  117.10  129.94  7579741.09\n",
       "2 2020-03-13 18:00:00  ETHUSD  124.47  124.85  115.50  119.51  4898735.81"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = (df['Date'] >= '2020')\n",
    "df.loc[filt].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All rows in 2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1749</th>\n",
       "      <td>2019-12-31 23:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.33</td>\n",
       "      <td>128.69</td>\n",
       "      <td>128.14</td>\n",
       "      <td>128.54</td>\n",
       "      <td>440678.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1750</th>\n",
       "      <td>2019-12-31 22:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.38</td>\n",
       "      <td>128.69</td>\n",
       "      <td>127.95</td>\n",
       "      <td>128.33</td>\n",
       "      <td>554646.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1751</th>\n",
       "      <td>2019-12-31 21:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>127.86</td>\n",
       "      <td>128.43</td>\n",
       "      <td>127.72</td>\n",
       "      <td>128.38</td>\n",
       "      <td>350155.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1752</th>\n",
       "      <td>2019-12-31 20:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>127.84</td>\n",
       "      <td>128.34</td>\n",
       "      <td>127.71</td>\n",
       "      <td>127.86</td>\n",
       "      <td>428183.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1753</th>\n",
       "      <td>2019-12-31 19:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.69</td>\n",
       "      <td>128.69</td>\n",
       "      <td>127.60</td>\n",
       "      <td>127.84</td>\n",
       "      <td>1169847.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10504</th>\n",
       "      <td>2019-01-01 04:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.75</td>\n",
       "      <td>133.96</td>\n",
       "      <td>130.74</td>\n",
       "      <td>131.96</td>\n",
       "      <td>2791135.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10505</th>\n",
       "      <td>2019-01-01 03:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.06</td>\n",
       "      <td>130.79</td>\n",
       "      <td>130.06</td>\n",
       "      <td>130.75</td>\n",
       "      <td>503732.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10506</th>\n",
       "      <td>2019-01-01 02:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.79</td>\n",
       "      <td>130.88</td>\n",
       "      <td>129.55</td>\n",
       "      <td>130.06</td>\n",
       "      <td>838183.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10507</th>\n",
       "      <td>2019-01-01 01:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>131.62</td>\n",
       "      <td>131.62</td>\n",
       "      <td>130.77</td>\n",
       "      <td>130.79</td>\n",
       "      <td>434917.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10508</th>\n",
       "      <td>2019-01-01 00:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.53</td>\n",
       "      <td>131.91</td>\n",
       "      <td>130.48</td>\n",
       "      <td>131.62</td>\n",
       "      <td>1067136.21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8760 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Date  Symbol    Open    High     Low   Close      Volume\n",
       "1749  2019-12-31 23:00:00  ETHUSD  128.33  128.69  128.14  128.54   440678.91\n",
       "1750  2019-12-31 22:00:00  ETHUSD  128.38  128.69  127.95  128.33   554646.02\n",
       "1751  2019-12-31 21:00:00  ETHUSD  127.86  128.43  127.72  128.38   350155.69\n",
       "1752  2019-12-31 20:00:00  ETHUSD  127.84  128.34  127.71  127.86   428183.38\n",
       "1753  2019-12-31 19:00:00  ETHUSD  128.69  128.69  127.60  127.84  1169847.84\n",
       "...                   ...     ...     ...     ...     ...     ...         ...\n",
       "10504 2019-01-01 04:00:00  ETHUSD  130.75  133.96  130.74  131.96  2791135.37\n",
       "10505 2019-01-01 03:00:00  ETHUSD  130.06  130.79  130.06  130.75   503732.63\n",
       "10506 2019-01-01 02:00:00  ETHUSD  130.79  130.88  129.55  130.06   838183.43\n",
       "10507 2019-01-01 01:00:00  ETHUSD  131.62  131.62  130.77  130.79   434917.99\n",
       "10508 2019-01-01 00:00:00  ETHUSD  130.53  131.91  130.48  131.62  1067136.21\n",
       "\n",
       "[8760 rows x 7 columns]"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = (df['Date'] >= '2019') & (df['Date'] < '2020')\n",
    "df.loc[filt]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we are using 'strings' for comparision, we can also use DateTime for the same with `pd.to_datetime('2019-01-01')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1749</th>\n",
       "      <td>2019-12-31 23:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.33</td>\n",
       "      <td>128.69</td>\n",
       "      <td>128.14</td>\n",
       "      <td>128.54</td>\n",
       "      <td>440678.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1750</th>\n",
       "      <td>2019-12-31 22:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.38</td>\n",
       "      <td>128.69</td>\n",
       "      <td>127.95</td>\n",
       "      <td>128.33</td>\n",
       "      <td>554646.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1751</th>\n",
       "      <td>2019-12-31 21:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>127.86</td>\n",
       "      <td>128.43</td>\n",
       "      <td>127.72</td>\n",
       "      <td>128.38</td>\n",
       "      <td>350155.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1752</th>\n",
       "      <td>2019-12-31 20:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>127.84</td>\n",
       "      <td>128.34</td>\n",
       "      <td>127.71</td>\n",
       "      <td>127.86</td>\n",
       "      <td>428183.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1753</th>\n",
       "      <td>2019-12-31 19:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.69</td>\n",
       "      <td>128.69</td>\n",
       "      <td>127.60</td>\n",
       "      <td>127.84</td>\n",
       "      <td>1169847.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10504</th>\n",
       "      <td>2019-01-01 04:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.75</td>\n",
       "      <td>133.96</td>\n",
       "      <td>130.74</td>\n",
       "      <td>131.96</td>\n",
       "      <td>2791135.37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10505</th>\n",
       "      <td>2019-01-01 03:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.06</td>\n",
       "      <td>130.79</td>\n",
       "      <td>130.06</td>\n",
       "      <td>130.75</td>\n",
       "      <td>503732.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10506</th>\n",
       "      <td>2019-01-01 02:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.79</td>\n",
       "      <td>130.88</td>\n",
       "      <td>129.55</td>\n",
       "      <td>130.06</td>\n",
       "      <td>838183.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10507</th>\n",
       "      <td>2019-01-01 01:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>131.62</td>\n",
       "      <td>131.62</td>\n",
       "      <td>130.77</td>\n",
       "      <td>130.79</td>\n",
       "      <td>434917.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10508</th>\n",
       "      <td>2019-01-01 00:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>130.53</td>\n",
       "      <td>131.91</td>\n",
       "      <td>130.48</td>\n",
       "      <td>131.62</td>\n",
       "      <td>1067136.21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8760 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Date  Symbol    Open    High     Low   Close      Volume\n",
       "1749  2019-12-31 23:00:00  ETHUSD  128.33  128.69  128.14  128.54   440678.91\n",
       "1750  2019-12-31 22:00:00  ETHUSD  128.38  128.69  127.95  128.33   554646.02\n",
       "1751  2019-12-31 21:00:00  ETHUSD  127.86  128.43  127.72  128.38   350155.69\n",
       "1752  2019-12-31 20:00:00  ETHUSD  127.84  128.34  127.71  127.86   428183.38\n",
       "1753  2019-12-31 19:00:00  ETHUSD  128.69  128.69  127.60  127.84  1169847.84\n",
       "...                   ...     ...     ...     ...     ...     ...         ...\n",
       "10504 2019-01-01 04:00:00  ETHUSD  130.75  133.96  130.74  131.96  2791135.37\n",
       "10505 2019-01-01 03:00:00  ETHUSD  130.06  130.79  130.06  130.75   503732.63\n",
       "10506 2019-01-01 02:00:00  ETHUSD  130.79  130.88  129.55  130.06   838183.43\n",
       "10507 2019-01-01 01:00:00  ETHUSD  131.62  131.62  130.77  130.79   434917.99\n",
       "10508 2019-01-01 00:00:00  ETHUSD  130.53  131.91  130.48  131.62  1067136.21\n",
       "\n",
       "[8760 rows x 7 columns]"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = (df['Date'] >= pd.to_datetime('2019-01-01')) & (df['Date'] < pd.to_datetime('2020-01-01'))\n",
    "df.loc[filt]  # pd.datetime gives error because its a 'string' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "to use date as index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-03-13 20:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-13 19:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>119.51</td>\n",
       "      <td>132.02</td>\n",
       "      <td>117.10</td>\n",
       "      <td>129.94</td>\n",
       "      <td>7579741.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-13 18:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.47</td>\n",
       "      <td>124.85</td>\n",
       "      <td>115.50</td>\n",
       "      <td>119.51</td>\n",
       "      <td>4898735.81</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Symbol    Open    High     Low   Close      Volume\n",
       "Date                                                                   \n",
       "2020-03-13 20:00:00  ETHUSD  129.94  131.82  126.87  128.71  1940673.93\n",
       "2020-03-13 19:00:00  ETHUSD  119.51  132.02  117.10  129.94  7579741.09\n",
       "2020-03-13 18:00:00  ETHUSD  124.47  124.85  115.50  119.51  4898735.81"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index('Date').head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2020-03-13 20:00:00</td>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  Symbol    Open    High     Low   Close      Volume\n",
       "0 2020-03-13 20:00:00  ETHUSD  129.94  131.82  126.87  128.71  1940673.93"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index('Date', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-03-13 20:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>129.94</td>\n",
       "      <td>131.82</td>\n",
       "      <td>126.87</td>\n",
       "      <td>128.71</td>\n",
       "      <td>1940673.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-13 19:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>119.51</td>\n",
       "      <td>132.02</td>\n",
       "      <td>117.10</td>\n",
       "      <td>129.94</td>\n",
       "      <td>7579741.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-13 18:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>124.47</td>\n",
       "      <td>124.85</td>\n",
       "      <td>115.50</td>\n",
       "      <td>119.51</td>\n",
       "      <td>4898735.81</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Symbol    Open    High     Low   Close      Volume\n",
       "Date                                                                   \n",
       "2020-03-13 20:00:00  ETHUSD  129.94  131.82  126.87  128.71  1940673.93\n",
       "2020-03-13 19:00:00  ETHUSD  119.51  132.02  117.10  129.94  7579741.09\n",
       "2020-03-13 18:00:00  ETHUSD  124.47  124.85  115.50  119.51  4898735.81"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can filter the dates by just giving the date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df['2020']\n",
    "# wont work now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-12-31 23:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.33</td>\n",
       "      <td>128.69</td>\n",
       "      <td>128.14</td>\n",
       "      <td>128.54</td>\n",
       "      <td>440678.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 22:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>128.38</td>\n",
       "      <td>128.69</td>\n",
       "      <td>127.95</td>\n",
       "      <td>128.33</td>\n",
       "      <td>554646.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-12-31 21:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>127.86</td>\n",
       "      <td>128.43</td>\n",
       "      <td>127.72</td>\n",
       "      <td>128.38</td>\n",
       "      <td>350155.69</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Symbol    Open    High     Low   Close     Volume\n",
       "Date                                                                  \n",
       "2019-12-31 23:00:00  ETHUSD  128.33  128.69  128.14  128.54  440678.91\n",
       "2019-12-31 22:00:00  ETHUSD  128.38  128.69  127.95  128.33  554646.02\n",
       "2019-12-31 21:00:00  ETHUSD  127.86  128.43  127.72  128.38  350155.69"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['2019'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.sort_index(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sorting index solves key error in next line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### slice  ing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Symbol</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-06-02 00:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>264.78</td>\n",
       "      <td>268.99</td>\n",
       "      <td>264.11</td>\n",
       "      <td>267.28</td>\n",
       "      <td>763788.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-02 01:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>267.28</td>\n",
       "      <td>269.20</td>\n",
       "      <td>266.53</td>\n",
       "      <td>267.68</td>\n",
       "      <td>582283.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-06-02 02:00:00</th>\n",
       "      <td>ETHUSD</td>\n",
       "      <td>267.68</td>\n",
       "      <td>268.23</td>\n",
       "      <td>265.00</td>\n",
       "      <td>266.03</td>\n",
       "      <td>820874.62</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Symbol    Open    High     Low   Close     Volume\n",
       "Date                                                                  \n",
       "2019-06-02 00:00:00  ETHUSD  264.78  268.99  264.11  267.28  763788.60\n",
       "2019-06-02 01:00:00  ETHUSD  267.28  269.20  266.53  267.68  582283.93\n",
       "2019-06-02 02:00:00  ETHUSD  267.68  268.23  265.00  266.03  820874.62"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['2019-06-02' : '2020-03'].head(3) # wont work if index not sorted\n",
    "# df.loc['2019-06' : '2020-03'] #both works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get average closing price of ethereum for all rows for these dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2019-06-02 00:00:00    267.28\n",
       "2019-06-02 01:00:00    267.68\n",
       "2019-06-02 02:00:00    266.03\n",
       "Name: Close, dtype: float64"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['2019-06-02' : '2020-03']['Close'].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "198.57806587960937"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['2019-06-02' : '2020-03']['Close'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TO get highs for each day when our data is hourly basis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "274.75"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc['2019-06-02']['High'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resampling data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get the same like weeks and other periods by using `.resample()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2017-07-01    279.99\n",
       "2017-07-02    293.73\n",
       "2017-07-03    285.00\n",
       "2017-07-04    282.83\n",
       "Freq: D, Name: High, dtype: float64"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['High'].resample('D').max().head(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can save it to variable and get max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "274.75"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "highs = df['High'].resample('D').max()\n",
    "highs['2019-06-02']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this will be useful in case we need to plot them  \n",
    "we need matplotlib to do that"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='Date'>"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAHBCAYAAAB314FOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB1LUlEQVR4nO3dd3wUdfoH8M+WbHo2hRQCSQi99xZFpESKwMnJeaKconJwKqjYxYIeFk5URBD11BP0J1gPLJyCSBUIxUDo0iG0JJC2qVvn98fuTHazm7BJZrPZzef9euVldmZ2ZxKH7LPP9/l+H4UgCAKIiIiIfIjS2xdAREREVFcMYIiIiMjnMIAhIiIin8MAhoiIiHwOAxgiIiLyOQxgiIiIyOcwgCEiIiKfwwCGiIiIfI7a2xfgKRaLBZcuXUJ4eDgUCoW3L4eIiIjcIAgCSkpKkJiYCKWy5jyL3wYwly5dQlJSkrcvg4iIiOrh/PnzaN26dY37/TaACQ8PB2D9BURERHj5aoiIiMgdOp0OSUlJ0vt4Tfw2gBGHjSIiIhjAEBER+ZhrlX+wiJeIiIh8DgMYIiIi8jkMYIiIiMjnMIAhIiIin8MAhoiIiHwOAxgiIiLyOQxgiIiIyOcwgCEiIiKfwwCGiIiIfA4DGCIiIvI5DGCIiIjI5zCAISIiIp/DAIY8ymCyIPNcIcwWwduXQkREfoQBDHnUnFUHMen9HXjn1+PevhQiIvIjdQ5gtm7digkTJiAxMREKhQLfffddjcfef//9UCgUWLRokcP2goICTJkyBREREYiMjMS0adNQWlrqcMyBAwdwww03ICgoCElJSViwYEFdL5WagP/uvQAAWLzxpJevhIiI/EmdA5iysjL06tULS5curfW41atXY+fOnUhMTHTaN2XKFBw+fBjr16/HmjVrsHXrVsyYMUPar9PpMGrUKKSkpCAzMxNvvPEGXnrpJXz44Yd1vVwiIiLyQ+q6PmHs2LEYO3ZsrcdcvHgRDz30ENatW4dx48Y57Dt69CjWrl2LPXv2oH///gCAJUuW4Oabb8abb76JxMRErFixAgaDAZ988gk0Gg26deuGrKwsLFy40CHQISIiouZJ9hoYi8WCu+66C08++SS6devmtD8jIwORkZFS8AIA6enpUCqV2LVrl3TM0KFDodFopGNGjx6NY8eOobCwUO5LJiIiIh9T5wzMtbz++utQq9V4+OGHXe7PyclBXFyc40Wo1YiOjkZOTo50TGpqqsMx8fHx0r6oqCin19Xr9dDr9dJjnU7XoJ+D5KFRK2EwWbx9GURE5GdkzcBkZmbinXfewfLly6FQKOR86WuaP38+tFqt9JWUlNSo5yfXwgNlj5GJiIjkDWB+++035OXlITk5GWq1Gmq1GufOncPjjz+ONm3aAAASEhKQl5fn8DyTyYSCggIkJCRIx+Tm5jocIz4Wj6luzpw5KC4ulr7Onz8v549G9SAIAkIZwBARkQfIGsDcddddOHDgALKysqSvxMREPPnkk1i3bh0AIC0tDUVFRcjMzJSet3HjRlgsFgwaNEg6ZuvWrTAajdIx69evR6dOnVwOHwFAYGAgIiIiHL7Ie0xmC8Yt3obsgnJpW5ne5MUrIiIif1Lnj8elpaU4ebJqTY8zZ84gKysL0dHRSE5ORkxMjMPxAQEBSEhIQKdOnQAAXbp0wZgxYzB9+nR88MEHMBqNmDVrFiZPnixNub7zzjvxz3/+E9OmTcPTTz+NQ4cO4Z133sHbb7/dkJ+VGtGRyzocuexYh7Q3uxA3dIj10hUREZE/qXMG5vfff0efPn3Qp08fAMBjjz2GPn36YO7cuW6/xooVK9C5c2eMHDkSN998M4YMGeKwxotWq8Uvv/yCM2fOoF+/fnj88ccxd+5cTqH2IUoXNVAHLhR74UqIiMgf1TkDM2zYMAiC+31tzp4967QtOjoaK1eurPV5PXv2xG+//VbXy6MmQqV0DmAqjWYvXAkREfkj9kIij7C4CHINZk6nJiIieTCAIY8wmZ0DGFfbiIiI6oMBDHmE0UW2xcQMDBERyYQBDHmEq+EiAzMwREQkEwYw5BFGl0NIzMAQEZE8GMCQR7gKVkwWZmCIiEgeDGDII1zVwHAWEhERyYUBDHmEq3oXDiEREZFcGMCQRxhNzsGKq7oYIiKi+mAAQx5hsrgKYJiBISIieTCAIY9wPYTEDAwREcmDAQx5hOshJGZgiIhIHgxgyCNcBStGTqMmIiKZMIAhj3C15gtnIRERkVwYwJBHGDiEREREHsQAhjzCdTNHDiEREZE8GMCQR3AlXiIi8iQGMOQRrps5MgNDRETyYABDHuFyCMnF4nZERET1wQCGPMLlEJKLwl4iIqL6YABDHuEqWHE1tZqIiKg+GMCQR+hdBTCsgSEiIpkwgCGPcJWBMZgtEAQGMURE1HAMYMgjxAxMaotQPH5TR2k7R5GIiEgODGDII8QMzJOjO2Hq9W2k7ZyJREREcmAAQx6hN5kBAIFqJdRKhbTdzBQMERHJgAEMeYQ4hKRRK6GyC2A4E4mIiOTAAIY8QhxCClSroFZW3WaciURERHJgAEMeUT0Do7AlYVgDQ0REcmAAQx6hlzIw1ltMrINhDQwREcmBAQx5hMFWxKuxBTBiHQyHkIiISA4MYMgjnDMw1v8yA0NERHJgAEOyEwQBBnNVDQxgl4FhDQwREcmAAQzJzmgWIHYMCFSrAAABKjGAYQaGiIgajgEMyU7MvgBVQ0isgSEiIjkxgCHZ6Y1m6XuNijUwREQkPwYwJDsxAxOgUkBpy7xU1cA4BjCCILBDNRER1RkDGJKdOExkvwKvWqyBsRteEgQBt3+4ExPf24HCMkPjXiQREfk0BjAkO6MtSBGDFsD1Qna6ShN2nynA/vNF+DTjbKNeIxER+TYGMCQ7cZgoQFV1e6ls2Rj7IST7bExRubGRro6IiPxBnQOYrVu3YsKECUhMTIRCocB3330n7TMajXj66afRo0cPhIaGIjExEXfffTcuXbrk8BoFBQWYMmUKIiIiEBkZiWnTpqG0tNThmAMHDuCGG25AUFAQkpKSsGDBgvr9hNToxCEk+y7UrjIw9sFMpV3hLxER0bXUOYApKytDr169sHTpUqd95eXl2Lt3L1544QXs3bsXq1atwrFjx/CnP/3J4bgpU6bg8OHDWL9+PdasWYOtW7dixowZ0n6dTodRo0YhJSUFmZmZeOONN/DSSy/hww8/rMePSI1NXKwuwD6AcbEOjNEuA1NuYABDRETuU9f1CWPHjsXYsWNd7tNqtVi/fr3DtnfffRcDBw5EdnY2kpOTcfToUaxduxZ79uxB//79AQBLlizBzTffjDfffBOJiYlYsWIFDAYDPvnkE2g0GnTr1g1ZWVlYuHChQ6BDTZNRLOK1G0JSK52LeO3XhKlgBoaIiOrA4zUwxcXFUCgUiIyMBABkZGQgMjJSCl4AID09HUqlErt27ZKOGTp0KDQajXTM6NGjcezYMRQWFro8j16vh06nc/gi7xCHidR2GRhX06jt2wpwCImIiOrCowFMZWUlnn76adxxxx2IiIgAAOTk5CAuLs7hOLVajejoaOTk5EjHxMfHOxwjPhaPqW7+/PnQarXSV1JSktw/DrnJ5HIWkvNCdkb7DAyHkIiIqA48FsAYjUb89a9/hSAIeP/99z11GsmcOXNQXFwsfZ0/f97j5yTXjJZa1oFxmIXEISQiIqqfOtfAuEMMXs6dO4eNGzdK2RcASEhIQF5ensPxJpMJBQUFSEhIkI7Jzc11OEZ8LB5TXWBgIAIDA+X8MaiezJba1oGxq4Gx+54ZGCIiqgvZMzBi8HLixAn8+uuviImJcdiflpaGoqIiZGZmSts2btwIi8WCQYMGScds3boVRmPV2iDr169Hp06dEBUVJfclk8ykIl4XNTD2w0b22RhmYIiIqC7qHMCUlpYiKysLWVlZAIAzZ84gKysL2dnZMBqN+Mtf/oLff/8dK1asgNlsRk5ODnJycmAwWJeK79KlC8aMGYPp06dj9+7d2L59O2bNmoXJkycjMTERAHDnnXdCo9Fg2rRpOHz4ML766iu88847eOyxx+T7ycljTC5nIbmqgbHLwDCAISKiOqjzENLvv/+O4cOHS4/FoGLq1Kl46aWX8MMPPwAAevfu7fC8TZs2YdiwYQCAFStWYNasWRg5ciSUSiUmTZqExYsXS8dqtVr88ssvmDlzJvr164cWLVpg7ty5nELtI8ShoWvOQmIRLxER1VOdA5hhw4bV2j3Ync7C0dHRWLlyZa3H9OzZE7/99ltdL4+aAJcZGFXtNTB6kwUWiyB1ryYiIqoNeyGR7FyuxGv7vrjCKK35Yl8PA3AYiYiI3McAhmQnDhOpHIaQrLfa0k2nMHj+BlgsgsMQEsAAhoiI3McAhmQnBib23ag1dlOqi8qNyC2pdBhCAlgHQ0RE7mMAQ7ITZxfZZ2ACA1QOx5y9Wu40hMR2AkRE5C6PLGRHzZvUC8ku6xKodoyVz+aXoXq9LoeQiIjIXczAkOzEGpgAu1YC1QOYy0UVzkW8HEIiIiI3MYAh2UlDSA4ZGMchpEqTRWr6KPpk+xnPXxwREfkFBjAkO7OUgbGvgXG81Qwmi8OidgCw7nAuyg0mz18gERH5PAYwJDuji4Xsqg8h6U1mpyEkAE5BDRERkSsMYEh24tCQfSuB6kNIeqPzEBIAmF0ENURERNUxgCHZmdyYhaQ3WWB0kW0xWpyDGiIiouoYwJDsqpo52g0hBTgHMC4zMBxCIiIiNzCAIdlJzRxrG0IymV3Wu1RvL0BEROQKAxiSXdUQUm1FvBZpunWv1lppOzMwRETkDgYwJDtxaCiglnVg9CaLtHBdepd4RIYEWJ/LAIaIiNzAAIZkZ3TRjdrVOjDltgAmWKOShpuqN3gkIiJyhQEMyc7s5jowYgATolFLwQ5rYIiIyB1s5kiyE7MoAbUU8eYUV+L0lTIAQIhGJc1YYg0MERG5gxkYkp24wq79EFJEsGOsXG7XuDFYo5LWjGENDBERuYMBDMlO6oVkN4QUoqk52RccoJKCHWZgiIjIHQxgSHZSN2q7DAwALLmjj8vjQ+yLeF0sbkdERFQdAxiSnUnKwDgGMBN6JWLPc+lOxwdrVFDZamA4hERERO5gAEOykxayUzrfXtWnUwPW4SU1h5CIiKgOGMCQ7MRhIFW1DAzgPJ0asA0hsYiXiIjqgAEMyU5cyyXARQZGo3LeZr+QnZkL2RERkRsYwJDspG7ULjIwCoXCKQsTYjcLyciF7IiIyA0MYEh2VTUwzgEMAGjsApjP7hsItUrJheyIiKhOGMCQ7EwuWgnYsw9serSydqKWWgkwgCEiIjcwgCHZSUNINWRg7PsdhQZaF7gTp1yzBoaIiNzBAIZkV5WBcR3AGO2CFHE4iRkYIiKqCwYwJDtxJV5X68AArjtOswaGiIjqggEMyc5cw0q8IldZFs5CIiKiumAAQ7IzWpy7UV8L14EhIqK6YABDshNX4g2oYRaSK6yBISKiumAAQ7KyWASIMUhNs5BcEadcmzmEREREbmAAQ7Kyz6DUVMTripoZGCIiqgMGMCQrk10NS03TqF2pGkJiDQwREV0bAxiSlX0GpT5FvMzAEBGROxjAkKzySw3S9zUV8baLDQUA9EmOlLYFBagAABUGs+cujoiI/EadA5itW7diwoQJSExMhEKhwHfffeewXxAEzJ07Fy1btkRwcDDS09Nx4sQJh2MKCgowZcoUREREIDIyEtOmTUNpaanDMQcOHMANN9yAoKAgJCUlYcGCBXX/6ajR/fPHw9L3NWVglt87EA8Ma4cP/tZP2hYTpgHgGAARERHVpM4BTFlZGXr16oWlS5e63L9gwQIsXrwYH3zwAXbt2oXQ0FCMHj0alZWV0jFTpkzB4cOHsX79eqxZswZbt27FjBkzpP06nQ6jRo1CSkoKMjMz8cYbb+Cll17Chx9+WI8fkRrT5mNXrnlMUnQInh7TGfERQdK2FmGBAIArpXqPXRsREfkPdV2fMHbsWIwdO9blPkEQsGjRIjz//PO45ZZbAACfffYZ4uPj8d1332Hy5Mk4evQo1q5diz179qB///4AgCVLluDmm2/Gm2++icTERKxYsQIGgwGffPIJNBoNunXrhqysLCxcuNAh0KGmJ1EbhEvFldc+sBoxA3OVAQwREblB1hqYM2fOICcnB+np6dI2rVaLQYMGISMjAwCQkZGByMhIKXgBgPT0dCiVSuzatUs6ZujQodBoNNIxo0ePxrFjx1BYWOjy3Hq9HjqdzuGLGl+F0VrDsnb2DXV6XqwtA3O1hAEMERFdm6wBTE5ODgAgPj7eYXt8fLy0LycnB3FxcQ771Wo1oqOjHY5x9Rr256hu/vz50Gq10ldSUlLDfyCqE73JjMJyIwAgLjzoGkc7EoeQdJUm6E0s5CUiotr5zSykOXPmoLi4WPo6f/68ty+p2blqK8ANUCkQFRJQp+dqgwMg1vwW24IgIiKimsgawCQkJAAAcnNzHbbn5uZK+xISEpCXl+ew32QyoaCgwOEYV69hf47qAgMDERER4fBFjatMbwIAhAWqoVC4vwYMACiVCmiDrUFPUQUDGCIiqp2sAUxqaioSEhKwYcMGaZtOp8OuXbuQlpYGAEhLS0NRUREyMzOlYzZu3AiLxYJBgwZJx2zduhVGY9Ub2fr169GpUydERUXJeckkIzGACdHUuTYcABAZYq15KixznkptsQjYcvyKdA4iImre6hzAlJaWIisrC1lZWQCshbtZWVnIzs6GQqHA7Nmz8corr+CHH37AwYMHcffddyMxMRETJ04EAHTp0gVjxozB9OnTsXv3bmzfvh2zZs3C5MmTkZiYCAC48847odFoMG3aNBw+fBhfffUV3nnnHTz22GOy/eAkP3ERuhCNql7PjwypOQPz0o+HMfWT3XhnwwmnfURE1PzU+aPy77//juHDh0uPxaBi6tSpWL58OZ566imUlZVhxowZKCoqwpAhQ7B27VoEBVUVda5YsQKzZs3CyJEjoVQqMWnSJCxevFjar9Vq8csvv2DmzJno168fWrRogblz53IKdRNX3sAAJsqWgSkqd87AfJZxDgDw4dbTePbmLvW8QiIi8hd1DmCGDRsGQai5X41CocC8efMwb968Go+Jjo7GypUraz1Pz5498dtvv9X18siLygwNHEISa2CqFfGazFUNHjvEhdXz6oiIyJ/4zSwk8r6GDyHZamCqBTAllVV1L8o6FgcTEZF/YgBDsikTA5jA+mVgggKst2P1dWCK7WpiSio5Q4mIiBjAkIwqxCGkgPplYNS27tUms+MQpc4uaMnR1b1NARER+R8GMCQbqYg3sJ4BjG0lO5OlWgBTUTWEZBGA5dvP1PMKiYjIXzCAIdk0dBaSWmULYOyKdgHHDAwAbHKj4zUREfk3BjAkm/IGzkISMzDmahmYzzLOOjwOq2eNDRER+Q8GMCSbSqM1cxJU3xoYpfV2NNoFMEazBTtPFzgcJ3a8JiKi5osBDMnGYLIGMBp1/W4rcQjJbKkaQrJvK9C9lbW/lThdm4iImi8GMCQbg612JVBVzwBGzMDYzUISO1wDwKzhHQAA5czAEBE1ewxgSDbi+i31zsC4qIHJL9MDADrFhyM8yFr7UskMDBFRs8cAhmQj1xCS0W4WUoFtCCkmTCPV1rAGhoiIGMCQbKQApr5DSLbn2WdgxCGk6FCNND2bAQwRETGAIdnoG5qBEReys6uBETtTR4VoECxmYDiERETU7DGAIdlIRbwNDGCMdrOQKm3ZlmCNCsF2GZjaOqITEZH/YwBDspFvGnVVcCKtLaNWSgGM2SI4zFQiIqLmh0uakmwaPoRUNY36XH4ZFm84iVNXSgEAgQEqaQgJsGZh6nseIiLyfQxgSDZiBqahQ0hmiwX3Ld+DU1fKpH1BASoEqJRQKxUwWQRUGMzQBgc0/KKJiMgn8SMsyaZqFlJ9mzlab0eTWXAIXgAgKMC6L9TWB6lUbwIRETVfDGBINmIRb32HdlTiLCSLc31LkNoaFIUxgCEiIjCAIZmYLYJUfFvfACZAJU6jtjjtExexEwOYMr0Js7/ch1vf2y7NVCIiouaDAQzJQhw+AjyUgZGGkKyBzNVSPb7LuoS92UX46eDlep2PiIh8FwMYkoXYBwmofxFvgFgD4zKAsWVggqyFuztPF0j7dp8pcDqeiIj8GwMYkoWYgVEoqmYT1VXVSryuhpCst2qYLQPzxe5saV/W+aJ6nY+IiHwXAxiqk7ySSpdL+evt+iApFPUNYGrOwATainjLXZz7eG4JSiqN9TonERH5JgYw5LY8XSUGvroBQ17f6LRPDDrq28gRqFqJ11VRrjiElKvTO+2zCMCaA6yDISJqThjAkNt2n7XWmuSXGZz2icM+KlX9si9A1RCSiwSMNISkq3DMtIiFv99nXaz3eYmIyPcwgCG3BdhlV6o3UxQzMOIwUH2oa8nehGis06dfmdjdYfudA5MBWIt6R7+91aGYmIiI/BcDGHJbgF12RWyyKDKZxQCm/hkYVQ3PTdQGITpUAwAY3jkOH93dX9rXo5UWY7olAACO5ZbgeE5pvc9PRES+gwEMuU1ll10pMziuhGuyWAMadQOGkAJqeO78ST0dHrfUBknfa0MCsOTOPtLjC4Xl9T4/ERH5DgYw5Daj3WJ15XrHoZqqIaSGBDCub8eIIMeeo2I2BgC0wQEIUCnxp16JAIALhRX1Pj8REfkOBjDkNqPd+izVexFJQ0gNmIVUUwATGx7o8DgqpCqAEbM2raOCATADQ0TUXDCAIbcZ7AKYL/dkO+yThpAakIFxZc7YzmgdFeKwLVhT1e26TUwogKphJVfTrImIyP8wgCG36e2GkD7LOOewTxpCakANTHVKBfCPG9u53LfjmRH49bGhiAmzZmeiQ63/LXAxxZuIiPyP+tqHEFnZN2ysThxCUjVgGnV1K/4+uMZ9iZHBDo+jQq09kgrKGcAQETUHzMCQ24zVehSZ7VacM9uGkAJkHEJKaxfj9rExzMAQETUrDGDIbdUzMOV2U6mNUgZG3hoYd4kZmKJyg0Ng1Rj2ZRfijXV/1JqhIiIieXEIidzmHMCYER5kDRzEoKGmmUSeJs5MsghAcYXRYaq1JxlMFvz5vR0AgI7x4bild6tGOS8RUXPHDAy5rfoQkn1naHGftzIwASolwm3rxRQ2Yh3M77b+UACgqzTVciQREcmJAQy5Te8UwFS9YVdlYLwTwABAqK1fUoWh8fohXSqulL7nEBIRUeORPYAxm8144YUXkJqaiuDgYLRr1w4vv/yyQ/M/QRAwd+5ctGzZEsHBwUhPT8eJEyccXqegoABTpkxBREQEIiMjMW3aNJSWss+NN7kaQhIZLd6tgQGAENv6MOWNGMDklVQFMGV6ZmCIiBqL7AHM66+/jvfffx/vvvsujh49itdffx0LFizAkiVLpGMWLFiAxYsX44MPPsCuXbsQGhqK0aNHo7Ky6s1gypQpOHz4MNavX481a9Zg69atmDFjhtyXS3VQWwBjNou9kLyX1BMXuKvep8mT8uwWzqu+OjEREXmO7EW8O3bswC233IJx48YBANq0aYMvvvgCu3fvBmDNvixatAjPP/88brnlFgDAZ599hvj4eHz33XeYPHkyjh49irVr12LPnj3o39/aeXjJkiW4+eab8eabbyIxMVHuyyY3ONXA2L1hiwvZNXQadXSoBgVlBmjqEQh5YwjJPgPDAIaIqPHI/nH5uuuuw4YNG3D8+HEAwP79+7Ft2zaMHTsWAHDmzBnk5OQgPT1deo5Wq8WgQYOQkZEBAMjIyEBkZKQUvABAeno6lEoldu3aJfclk5tqy8CYLPIsZPd/0wZiSPsW+Ob+tDo/N9gLQ0j2rQs4hERE1Hhkz8A888wz0Ol06Ny5M1QqFcxmM1599VVMmTIFAJCTkwMAiI+Pd3hefHy8tC8nJwdxcXGOF6pWIzo6WjqmOr1eD72+6s1Ep9PJ9jORlaGWIl6TbV9Di3i7JWrx+d8H1eu5Yg1MRSMOIeXqWANDROQNsmdgvv76a6xYsQIrV67E3r178emnn+LNN9/Ep59+KvepHMyfPx9arVb6SkpK8uj5miO90TGAKSw3St+bmkARb1UNTONkYARBQF4Ja2CIiLxB9gDmySefxDPPPIPJkyejR48euOuuu/Doo49i/vz5AICEhAQAQG5ursPzcnNzpX0JCQnIy8tz2G8ymVBQUCAdU92cOXNQXFwsfZ0/f17uH63ZqzBaA4NWtj5E9sv2i72QvLWQHVBVA9NYQ0jFFUaHYbUyfeMNXRERNXeyv9uUl5dDWa0OQqVSwWLrlZOamoqEhARs2LBB2q/T6bBr1y6kpVnrHtLS0lBUVITMzEzpmI0bN8JisWDQINfDC4GBgYiIiHD4Inl8m3kBe7MLqwKYKGsAY79gXFPIwDT2EJJ99gUA8kv1NRxJRERyk70GZsKECXj11VeRnJyMbt26Yd++fVi4cCHuu+8+AIBCocDs2bPxyiuvoEOHDkhNTcULL7yAxMRETJw4EQDQpUsXjBkzBtOnT8cHH3wAo9GIWbNmYfLkyZyB1MgyzxXiiW/2AwA6J4QDqCkDI06j9v4QUmNlYK7aApjIkAAUlRtxWVcJg8kCjZrrQxIReZrsAcySJUvwwgsv4MEHH0ReXh4SExPxj3/8A3PnzpWOeeqpp1BWVoYZM2agqKgIQ4YMwdq1axEUFCQds2LFCsyaNQsjR46EUqnEpEmTsHjxYrkvl67hQmG59L3eNlziMoCxZWDUTSAD01gBjLgycavIYOiNFlQYzbhUVIE2LUIb5fxERM2Z7AFMeHg4Fi1ahEWLFtV4jEKhwLx58zBv3rwaj4mOjsbKlSvlvjyqI7GuBKgKWBLtAhhBEKBQKGCyDRGqGziNuiGCpRqYxhlCMtoCOo1aiaToYBzPLUV2QTkDGCKiRsBcN9XKfkiouMI66yhBGwgAuFxcie4vrsP2k1elXkjezMCENnIGpmrxPiWSo0MAAD8dvOzQNoOIiDyDAQzVylWDwhZhgdL3ZQYzpny8C0bbLCRvthJo7CEko13dT+soawDz5Z7z2HEqv1HOT0TUnDGAoVpVX7wOAGLsAhiROEPJmwWswY08jdpoN3W8tW1mFgD8ctj1YotERCQfBjBUK1cZmKiQAKdtRy9ZVz62fyNvbI09jdpot/pwiF2tUMtI7/0OiIiaCwYwVCu9iwAmSK1y2nb6ahkAINWLBayNPYRU1T5BiYl9qqb3s6UAEZHnMYChWlXPwAQFKKGspVBXLGb1BjELkleid5j+7Sn2dT8hGjVmDW8PANBVGGt7GhERyYABDNWqegATHOCcfRF1S4xAUC37PU3MwADAnR95vmu5sVoDy/AgawB16JIOn+88J83MIiIi+cm+Dgz5l+pFvNGhmhqP/b9p9esiLZdguwAmu8DzGRj7adQAEBFsrQ3KPFeIzHOFUCoUuHNQssevg4ioOWIGhmqlNzrWk3SMD3d53Nu396o1uGkMIdWyP1/tyfbo+cTslLpaBkb04/5LHj0/EVFzxgCGaqWvloHpUC2AaRUZjC1PDsPE3q0a87Jcqr4GzdP/PejR84mrD4sduCOCHGdnNUYWiIiouWIAQ7WqXgOTZJsm/dzNXRCgUmDhX3shJSYUCoX3VuC1Z98NOynas9OZTdI6MNZzhlXLwBTZdesmIiJ5sQaGalU9gIkKsQ4TTR/aFnelpXi1aNeVHc+MwIajeXh29UEoPRxUGcyOGRj7vlGAdZVidqcmIvIM/mWlWlVfBybSbhG7pha8AEB8RBAGtY0GABSWeTYDYqrWPsF+FpTonQ3HccOCjTiZV+LRayEiam4YwFCtqmdgtMHOq/A2NdG2LJGu0iRNdfYE8bU1tiGk0EDnhObSTadwvqACc78/7LHrICJqjhjAUK2cAhgXbQSamojgAIilMEXlnltUrnoDS1cZGNGJvFKPXQcRUXPEAIZqVT2D4QsZGJVSIU3pztVVyva67248gWFvbMLl4goAdt2obdFSYC21LkXlBli4sB0RkWwYwFCtzILjm26giz5ITVG72DAAwNHLOtle881fjuNsfjmeX30IQNU0arFIt7aZWEazgKtletmuhYiouWMAQ7Xy1eXwu7SMAAAcvSx/8exvJ64CsBtCUjr/MxreKdZp2+Ui+bJBRETNHQMYqpWYgLn3+jbY/swI715MHbSNtXbFvlRUIftrG8wW7Dqdj4uF1tcW14GxF6By/qd1uZgBDBGRXLgODNVKzMD0SY5Cq0jPLgwnJ7EexVOzkG7/cKf0vatgxVXiqpAL2xERyYYZGKqVWAOjaiIr7bpLDCqqN6P0BLWLDExLbRAm9W0NAGgRFggAqDCYnY4jIqL6YQBDtRJsAYzSt+KXqgDG5PkAJthuQb+P7+6PMd0S8Piojljwl57YOWckRnS21sNUGBnAEBHJhUNIVCtxCEnpYxGMRuYhJEGouZi5dVSI9H1613ikd42XHidog6QAp5IBDBGRbJiBoVqJtRy+NoSkUYkBjDyzqGobirpW08gg2wJ3HEIiIpIPAxiqlUUcQvKxO0XuIaTqPaHshWhqT2SKGRgOIRERycfH3paosUlDSL6WgZF5CElvdHydzgnh0KiUmNAr8ZrPZQBDRCQ/1sBQraQhJB+rgRHXZpFrFpLe5Bh8dGkZga9mpCE86Nr/hII1rIEhIpIbAxiqlcVHMzCeHkKael0btxtbihmYctbAEBHJhgEM1cos+GYAI/dCduIQUnSoBv97eAhaat1f1C+YRbxERLJjDQzVSizi9b0hJHlnIYlDSKGBqjoFLwA4jZqIyAMYwFCtqoaQvHwhdRSg9swQUn26cbOIl4hIfgxgqFbSEJKPRTD2Rby1LULnLnH4RxyaqgtxHRjWwBARyYcBDNXKYktg+FwNjKoqU2Jy1Vmxjn48cAkAkNoitM7PDbWtE8MAhohIPgxgqFYWX23mqK66XjmGkfZlFwEA/to/qc7PDQ20BlOlelODr4OIiKwYwFCtfH0lXqDhM5EEQUBOcSUAIDk65BpHOwsLtGZgDCaLbLOiiIiaOx97W6LGZvbRISS1UgHxkhu6mF2J3iQV4MZHBNX5+aGBVasVlFXLwlhkGN4iImqOGMBQrXx1GrVCoYBYu3vkkq5Br5Vry75EBKmlNV3qIkCllFob2A8jfbLtDLq9uA7/l3G2QddHRNQcMYChWll8dCE7ew99sa9Bz99+8ioAIEFb9+yLSBxGKtNXFfL+eOASKoxmvPD9YVlmShERNScMYKhWZh9dB8ZeSWXDimdXZ1lnIKW1jan3a7gq5LVvEHm11FDv1yYiao4YwFCtxBoNXxtCAoDnx3UBAIQHNqxjhq7CCAC4uUfLer+GOBNq+me/S9vKDVXBzIBXf8XpK6X1fn0ioubGIwHMxYsX8be//Q0xMTEIDg5Gjx498PvvVX+4BUHA3Llz0bJlSwQHByM9PR0nTpxweI2CggJMmTIFERERiIyMxLRp01Bayj/wjU2sMfXFIaQ7ByUDsBbhFtuCkPoQA43QBgRCuTo9AKCgzCC1FCiqdk1LN52q9+sTETU3sgcwhYWFuP766xEQEICff/4ZR44cwVtvvYWoqCjpmAULFmDx4sX44IMPsGvXLoSGhmL06NGorKyUjpkyZQoOHz6M9evXY82aNdi6dStmzJgh9+XSNfjqSrwAEKJRI8rWMVqcBl0f4gJ09SngrbqWqucevayDxSI4BVX/3XsBu07n1/scRETNiewBzOuvv46kpCQsW7YMAwcORGpqKkaNGoV27doBsGZfFi1ahOeffx633HILevbsic8++wyXLl3Cd999BwA4evQo1q5di48//hiDBg3CkCFDsGTJEnz55Ze4dOmS3JdMtZCGkHwwAwMA2mBrAFNSWf8MjJgxCWlAALP83oHS90cvl0BXaZRmSf362I3Svm22gmEiIqqd7AHMDz/8gP79++O2225DXFwc+vTpg48++kjaf+bMGeTk5CA9PV3aptVqMWjQIGRkZAAAMjIyEBkZif79+0vHpKenQ6lUYteuXS7Pq9frodPpHL6o4apmIXn5QuopPEgMYOpXyGs0W6SO1mJTxvoYmBqNqWkpAIDzheUoKrcGVKEaFdrHheHe69sAAM7ll9f7HEREzYnsAczp06fx/vvvo0OHDli3bh0eeOABPPzww/j0008BADk5OQCA+Ph4h+fFx8dL+3JychAXF+ewX61WIzo6Wjqmuvnz50Or1UpfSUl1X/KdHAmCUFUD46MRjDh9uaSey/jb9y9qyBASACTZVvE9X1CO7AJroBJvm5o9KNU6w+lcflmDzkFE1FzIHsBYLBb07dsXr732Gvr06YMZM2Zg+vTp+OCDD+Q+lYM5c+aguLhY+jp//rxHz9cc2C9N4qtDSGFB1gBm95l8rD3kOvitjTh8pFIqoFE17J9L6yhbAFNYgQ1HcwEAHePCAVS1KLhQWNGgcxARNReyBzAtW7ZE165dHbZ16dIF2dnZAICEhAQAQG5ursMxubm50r6EhATk5eU57DeZTCgoKJCOqS4wMBAREREOX9QwZrsIxhdnIQFVU6g/35mN+z/PROa5wjo9XyrgDVBB0cDfQVJ0MABg//kifJpxDgDQMT4MABARbL1ONnwkInKP7AHM9ddfj2PHjjlsO378OFJSrOP/qampSEhIwIYNG6T9Op0Ou3btQlpaGgAgLS0NRUVFyMzMlI7ZuHEjLBYLBg0aJPclUw3Mdn16fK2Zo0jMwIgyTtWtSFacQt3Q4SOgagjJ3rDO1qHSIFt9jd5k4aq8RERukP1t6dFHH8XOnTvx2muv4eTJk1i5ciU+/PBDzJw5E4C1R83s2bPxyiuv4IcffsDBgwdx9913IzExERMnTgRgzdiMGTMG06dPx+7du7F9+3bMmjULkydPRmJiotyXTDVwGELy0RqY8GoBTF6Jvk7PrzA0fAaSKCIoQJoVBQD/N20g+iZblxcIsisQ1pvYsZqI6FoatkSpCwMGDMDq1asxZ84czJs3D6mpqVi0aBGmTJkiHfPUU0+hrKwMM2bMQFFREYYMGYK1a9ciKKiq18yKFSswa9YsjBw5EkqlEpMmTcLixYvlvlyqhT8MIYUFBjg8rmuNidiFuiEzkOzFRwRK6790ig+Xtgepqz5LVBrNDgENERE5kz2AAYDx48dj/PjxNe5XKBSYN28e5s2bV+Mx0dHRWLlypScuj9zkMITkowGMWFsiKq3jdGo5FrGz1zkhAsdzSxEepEZseKC0Xa1SQq1UwGQRUGlkBoaI6Fo8EsCQfxAcMjBevJAGSK5Wd6Kr44J2cg4hAcDLE7sjNjwQPVtrnYqCgwJUKNWbpJlPRERUMwYwVCP7DIyv1sC0jQ1zeFzXWT5VQ0jy/FPRBgfghfFdXe4LClCiVA9UmhjAEBFdi4/OLaHGINbAKBRo8BRib2kZEeTwuK4BjNxDSLUJVFvPwSEkIqJrYwBDNRJ8uBO1SKlU4OkxndEvxTrbp6TSVKdpyhW2adQhjVBUGxSgtJ2TGRgiomthAEM1Mvt4I0fRA8Pa4dP7rM0UzXUskm3MDIw484hDSERE18YAhmokNXL0g7vEPoNy6kqp28+rkKETtbukxexs58w8V4D7lu9hfyQiIhf84K2JPMViS1T48hCSyL4Z5RPf7Hf7eRUGedeBqY04hCRmiB7+Igsb/8jDjW9sxu4zBR4/PxGRL2EAQzUSi3h9fQipuj9yStw6rrjCiN1nrYFDYwwhiYHiil3WPklX7FYN/ungZY+fn4jIlzCAIZcsFgFzvz8EwDF74cs2PH6j9H1+ae0tBSoMZox6ewtOX7EO34RoPL/iQEGZAQCw56y14WRKTNUaNrqKuq1fQ0Tk7xjAkEu/HMnBbyesjQ/9JH5Bu9gwpLYIBQAcvqSr9dhPtp9Brq4qyAnWeP6fyt8Gp0jfXynR46pdkHW5uNLj5yci8iUMYMilq6UG6XtfXcTOla6JEQBqD2AyTuXjjXWOHdXlWsiuNrf2bSV9P+DVX1FYXpV1ydUxgCEisscAhlyyL3vx1UXsXOlmC2COXK45gFm194LTtuo9lTxBXMjOlRwGMEREDhjAEABg7aEch+nFCvhP0GIvKcpaV5JXS0CgDQ5w2tY/Jdpj11Qbja1LdbnB7NDagYiouWMAQ9h+8iru/zwTI9/aIm2zT7r40xunGJwU11IUW33G0df/SJMCicbWIa6ql1O5wb02CIIg4OlvD+DZ1Qc9dVlERF7HAIbwu23Wiz371fbVflQDIwYwtc3qse+X9PqkHhiY6p3sCwC0aREqFVG722Jgx6l8fPX7eazclY3CMsO1n0BE5IMYwBDMFuel9fV2y9kHqPznNnEnA6OrqApgbu3b2uPXVJvWkcEItU3hLnMzgNl64or0va6S06+JyD/5zzsT1ZvJxRCRfb8gf5qFJAYwZQYzjGbXPZFKbG/6r/65u9eDt3ZxYQgJtA5plbnZSds+62IfjBER+RMGMORQ4yJ+X2ms+rSvVvlPABNhV6Bb0zCSmLUID3Iu5vW02PBAh8djuydIi+hVGN3LwBTZTb9mBoaI/JXn54ZSk6errPqUXm4wITwowKEjcoA/dHO0USkVCA9Uo0RvQnGFETFhgU7HiFmL8KDG/+fx7f1p+CHrEtrHhSEpOgThQQFSI8maMjBrD+UgQKXAyC7xAIAiu8CMK/gSkb9iANPMXS6uwBe7s6XHFUazNYAx+GcGBrBmYcQAxhUx0xEW2Pj/PFJiQvHQyA4O28QamHIXNTA5xZW4//NMAMDvz6ejRVggisrthpCYgSEiP+U/H62pzkr1JqTN3+iwTZzpYl8Do/ajIl7g2oW8elsAE+ilqdPViTUw9gFMXkkllm46ifVHcqRtm49Zi3cdhpBYA0NEfooZmGbs0x1nnbY9u/og7hiY7DCE1N22eq2/uGYAY7IGb7WtjNuYxCEk+3VgHv96v9SrSrT/fBEm9W3lOIRUacScVQfx08HLGNYpFotu7+1XKysTUfPVND5iklecyC1x2rb9ZD5mrdwnFfEGqBR4emznxr40j7rWWjBVAUzT+OchDiFdLbE2dyzVm5yCFwA4X1iOSqMFBlNV9uxkXim+2J2N4gojvs+6hNNXyxrnoomIPKxp/IUmr7hQWFHjvjO2N7r5t/ZEhBdm43jStTMwtiGkgKbxz2NAG+tCeqv2XYTBZKmxDcL5gnIUVTguXLftpGOgc+QaXbiJiHxF0/gLTV4h1lSolQqkxIQ47Duea+2L5KovkK/ThtQcwJgtAoxm61TypjKENKFXIlqEaXChsALfZ11EYbnr1XXPF1agoNrKuyWVjjUwj3+9H4cuFnvsWomIGgsDmGZMHCZaOX0wokM1Lo/xywCmlgyM/fBLUxlCCtaoMMm2IvCBC8UoKLNed6/WWqz8+yB8OWMwAOu1Z+eXAwDCq82gCg9SQ6kADGYL7l2+x2GdHyIiX9Q0/kKTV4gZmBCNCg+P7ICgACX6p0Q5NHL0xwBGXMzOfraOyL6FQlMJYACgva2p45mrZdhwNBcAEBWqwXXtW2Bw2xhpzRqxxqVDfBj6JkdKz//33/rhP1MHAACulOhxIrcURES+jLOQmjFxVktQgArDO8Uh8/mbEBygwoR3t+GwrVbCHwOY2DBrtulKqd5pn1jAq1IqmtT08dQWoQCsM43EuhaN3fXFhGpQUmnCqSvWwCQ6VIOP7u4Po1lw6KQ9uG00dp4uwPHcEvRorW3En4CISF5N5y80NTpxrRdxmm5ooBpKpUJ6swSAiGD/i3ETI4MBABddFDHrjU1rBpKoje3/SYndarx/5FTNIhOHAE9fsWZgtMEaKBQKh+AFADrGhwMAjruYgUZE5Eua1l9pajQmswUGWzPD4ADHYtUWdsvrV9/nD1rZApi8Er3DkBFQNYQU1MR+7phQjVNrg0fsVuyNDrX+PzuVZ83ARIa4zpwlR1uLtS8Xu57JRETkKxjANFP2jQGDNY5v1nERVQGMPy56Fh2qkTIsucWOw0iVTTQDo1A4ZsbiIwLx5z6tpMcxtgyMmKFpHRXs8nXEZpFXSpyHz4iIfEnT+itNjUZsGaBQOL9Z3zkwGQkRQbitX2tvXJrHKRQKxEcEAbAuyW9PWgOmiQUwABwCmOvatYBSWRVcprWLcTi2TUwoXBGza1dd1P8QEfmSpvdXmhqFNAMpQOWUZYkM0WDHMyPwxm29vHFpjUIcjimt1uG5qbURsDdlUIr0fUttkMO+W3onYkj7FtLj6uv6iKQMDAMYIvJxDGCaKXEIKVjjukjX/tO9PxI7TX/82xkIgiBtb2qr8NobmBqNz6cNwuhu8fhLteyYQqHAnJs7Q6NWokWYBq2jXAcwYgamqNzosOYNEZGv8b8pJuQWMQMTrGl6b9SNQQxgtp28iu0n8zGkQwsUlRtw3/LfAThOUW5KhnRogSEdWrjc1y1Ri61PDodCAafZR6LI4ABo1EoYTBZcLq5ASg1DTURETV3T/CtNHieuxBoS0Dxj2DC7GT2nr1pn7ny49bS0TRxq8TUJ2iCpvscVpVKBDrZF8Y5e5lRqIvJdDGCaKTEDE6RperUejUFlN0RWaTTDYhFQaLcyb1K06yEYf9ClZQQA4Mgl9kQiIt/FAKaZqjBWFfE2R/b1H6/99Aee+Ha/w6J9IX4c2A1oEwUA+HxXNi4X19yRnIioKWMA00xV2NoI+PMbdW2qF7Cu2nsRhy/qpMcJtQzD+Lo/9WqF+IhAFJQZ8PzqQ96+HCKievF4APOvf/0LCoUCs2fPlrZVVlZi5syZiImJQVhYGCZNmoTc3FyH52VnZ2PcuHEICQlBXFwcnnzySZhMJpA8mvsQkt7FDJxDdkMqt/b1zzVwAOvChW/d1hsAsOGPPHy646xXr4eIqD48GsDs2bMH//73v9GzZ0+H7Y8++ih+/PFHfPPNN9iyZQsuXbqEW2+9VdpvNpsxbtw4GAwG7NixA59++imWL1+OuXPnevJym5XmPoTUx65Ts0jsTv3GX3rWOIvHX1zfPgYD20QDAF796ShKKp07cxMRNWUe+ytdWlqKKVOm4KOPPkJUVJS0vbi4GP/5z3+wcOFCjBgxAv369cOyZcuwY8cO7Ny5EwDwyy+/4MiRI/j888/Ru3dvjB07Fi+//DKWLl0Kg8HgqUtuViqkadTNM4C5/8Z2NfYLigrRNPLVND6FQoGv/jEYoRoVDCYLctgbiYh8jMcCmJkzZ2LcuHFIT0932J6ZmQmj0eiwvXPnzkhOTkZGRgYAICMjAz169EB8fLx0zOjRo6HT6XD48GGX59Pr9dDpdA5fVLPmHsAEBajw5OhOLvf56hTqulIoFGhh+1mLK5iBISLf4pEA5ssvv8TevXsxf/58p305OTnQaDSIjIx02B4fH4+cnBzpGPvgRdwv7nNl/vz50Gq10ldSUpIMP0ndCYKAmSv2YsrHO6UgoSkqF1fibaZDSACQGOnc8FCjUqJzy3AvXI13RAZbs1BF5QxgiMi3yB7AnD9/Ho888ghWrFiBoKDGm8kxZ84cFBcXS1/nz59vtHPbe+rbA/jfwcvYfjIfy5twcWSl2AupmWZgAGBwaozTth6ttU2yD5KnRNgCGGZgiMjXyB7AZGZmIi8vD3379oVarYZarcaWLVuwePFiqNVqxMfHw2AwoKioyOF5ubm5SEhIAAAkJCQ4zUoSH4vHVBcYGIiIiAiHr8ZWbjDhm8wL0uNjOU13GKuqlUDzXIkXsA6fvTC+q8O2vi6Ke/1ZpK3ep4gBDBH5GNkDmJEjR+LgwYPIysqSvvr3748pU6ZI3wcEBGDDhg3Sc44dO4bs7GykpaUBANLS0nDw4EHk5eVJx6xfvx4RERHo2rWr0zmbiktFjouC5Zc13YJjDiFZTRuSiqV39pUe902OquVo/6O1Ld7HDAwR+RrZP36Hh4eje/fuDttCQ0MRExMjbZ82bRoee+wxREdHIyIiAg899BDS0tIwePBgAMCoUaPQtWtX3HXXXViwYAFycnLw/PPPY+bMmQgMbLoFlheLHGdyXC1tugGMzvaGFdqMh5BEI7vE4d7r26C4wojhneO8fTmNKjLYmoEpLm+69yoRkSteGT94++23oVQqMWnSJOj1eowePRrvvfeetF+lUmHNmjV44IEHkJaWhtDQUEydOhXz5s3zxuW6TczAxIUHIq9Ej6ulei9fkTNBELDjVD6yzhcBANrbGvs1Z0EBKrw4oZu3L8Mr4rXWOrVzBeVevhIiorpplABm8+bNDo+DgoKwdOlSLF26tMbnpKSk4KeffvLwlcnrsi2A6dlai1+P5qGgzACLRYDSrnGgtx26qMOUj3cBsM64aRvLAKY569FKCwA4cKEYgiBAoWg69yoRUW38e7nRRnbVVvPSMd46DddsEZpcbcG5gjLp+/E9Wzp0Zabmp3NCOAJUChSUGXDkctMtOiciqo4BjIwKbDUvCdogaG3TU5vaMFKhLcga3ikWb97Wy8tXQ94WFKDCqG7WmX3/zbzo5auh5upkXinuWbYbmecKvX0p5EMYwMiowFYIGRWiQUyYtTiyqRXyFpRZM0IJ2uAmNbRF3iP2RMrRVVzjSCLPmP3VPmw+dgWT3t/h7UshH8IARkYFtuxGTKgGLUKts6Xyy5pYBsYWZEWHuu4DRM1PeJC1FK6kkt3eyTsuFbEXF9UdAxgZicMzUaEatAi3ZWBK5A1g8kv1sFiEej+/0C5LRAQAYYHWAEbHAIa8RLwHieqCAYxMTl8plRauiw7VIEbKwMgzhGQyW/DEN/vR75Vf8cr/jtb7dQrsrpEIAMKDrNm40sqmVXBOzQcDGKoPBjANcDKvFLO/3IeTeSVYsSsbANClZQTiwgOlGpglG09i+8mrDT7Xy2uO4Ftbm4JdZ/Lr/To5xdZUbYuwprsgIDUuDiGRt4UFVQUwRrPFi1dCvoQBTAPMWrkX32Vdwu3/3onTV0oBAFMGJUOhUDgECOK6Kw3x86GqLtxiEFIX+aV63P3JbpzIs15nSkxIg6+J/EOELQPDAIa8JVBd9VZU0IRbsFDTwgCmAcRgIL/MgE3HrgAA2saGAgBahDkO0ey3rXxbX6X6qjeX/DIDKm29jNz16k9HsfX4FelxYmRwg66H/IeYgakwmvnpl7xCb6y675ra2lnUdDGAaYC4cOdhmNQWYgDjuO+bzPP1Po/ZIkjdo0XVG0dey8ELxdL3LcICEaDi/3qysk/flzILQ15Qbqy673QMYMhNfBdrgOoBTLvYULTUWjMbMdUCmJzi+s9GKjNU/eMWh34uuhHA5OoqMeyNTXhz3THk6KzDTmGBasxO71DvayH/E6BSIjLEOoy052yBl6+GmiP7D2jMwJC7GMA0gDh7Q7Tqgeul72OqDSFdacCKvGW24aMAlQJtYqwZHncyMKv3XcTZ/HK8u+mkVN+w89mR+NvglHpfC/mnyQOSAQD/t/McAEBXaYQg1H+6PlFdVNgFMA99sQ+Xi7moIl0bA5g62HQsz2FGUbldZuSe69pAG1IV0IQHqhFl97gh68GIaf2wQLVUu3LRjYWfTuSWOjwOD1JzuiK5dMfAJADAjlP5+HTHWfT+5y94b/MpL18VNQdGs8WhgLzcYMY7v56QHheUGeo8ZE7NAwMYNxjNFvx24gruXbYHUz7eJRXUimnP2ekd8ML4rg7PUSgU+O3pEVjz0BAA1iGfl344jJN5JW6dUxAEnMwrgd5kls4XGqhG6yhrAHPUjcZ71Zvzce0XqklKTCiiQgJgtgh48YfDsAjA+wxgyMMKygwY/fZWh0kKAHDF9oGv0mjGhCXbMPKtLdJMTyIRAxg3fLLtDO76z27p8eGL1oLYCttMoCHtW7js6hwWqEb7uDDp8fIdZzFr5b4az/H0twek2UWbj11B+sKt6PT8WiyyfRoJC1RLna7XH8nFpmN5+HTHWZyy/cMu1ZtgMFmr+QVBwIWCcgDWbBAA/P2GtvX8DVBzEFotO8chJPK0L3Zn4/TVMqftYhPcz3eew8WiClQYzVi2/WwjXx01dRxPcMMb6445PL79w51Iig7G+QJrWjNYo6rxuUEBKrSKDJaKbv/Icc7ACIKAeWuOAAACA5SYd0t3HLCbNbTFNv05LFCNkZ3jML5nS6w5cBn3LtsDALi+fQxaaoPxbeYFBAeocHOPlrildyJKbJ9q1j06FLm6SvRJjqrvr4CagerDi+VGM/QmMwLVNd/fRA1x+kpV8GL/d/JqqQGCIODj385I+zmMRNUxA+OGZBeLvonBCwCEaGqPA4e0byF9rw0OQKXRjFV7L0gLNhWWV1Xdf5ZxDl/vOe+yiC08SA2lUoGX/tTNYfv2k/lYZ1vorsJoxn/3XsDdn1gzRlEhAUiMDGbwQtcUHlQ9AwOXq0iXG0zMzpAsLhZZs8RPjemEbU8Px7J7B9i2V+Dxr/dLsycBoKCcC9yRIwYw1yAIAvJ01nTm+keHujwmtJYMDADcfV0KNLZ1VzRqJVbtvYjHvt6PQa/9CqPZ4hSsPPXfA/hyj3XdmBGd4wBY12655/pU6Xtxu6hE73r9jk4J4bVeG5Eo2EUg/tzqQw4zRPZlF6L7i+vw8pr69+MiEl0otP7tG9AmGgqFAoNTY6R9q/ZddDi2qJzTq8kRA5hrKK4wSgVmSdEhSGsb43RMbUNIANAtUYvfnh4OwFq0djzXOoxkNAv4+VAOHvh8b43Pvff6Ntj65HBse3o4buwYK21/67ZeuHNQssMS3NVpVEq8PqlnrddGJDKYnFd3vlxciSe/3Y+jl3UY8Oqv+PN7O2ARgE+2n8GmY3leuEryFxaLILVFaWWbXenqb2lytDUDzhYDVJ3fBzCnrrg366cmYjFZRJAaQQEq/GtSD0wekIR37+yDbokRuHNQstN6MK7EhGqgUFhX1c0rqUqLPvzFPmTbim0BoFdrLdK7xEuPO8WHIzkmBEEBjv+wo0I1eO3PPbDr2ZH4+5BUBAUo8Y8bq4p0/3FjWxx7ZQxSbOvGEF2L/WJi/7Qbplxz4DJ+3H9JmhkieumHwzCx9QDVU1GFESaLdSjSfuXyaUNSHY4b37MlAOvaRLzffI/Z4rnhZr8v4n1u9WH89ESrej8/v9Qa9YtTkFNiQvEvW1ZjfM9Et19HrVIiUWstUtt/vtjlMV/OGIzBbWNwubgCaqUCt/ZthbiIoFpfNzJEg+fHd8WzN3eBUqnAv7ecBgD0T7GmZIncZR/AjO2RgBd/OCw9PnjR+Z49l1+Onw7l4E+93P93QP5vwdo/EBqoxszh7Ws9rqCs6sOhxi6T/MzYzuidFInjuSUorjDi4ZEd8N7mUxAEa0a8+irn1DSZzBY89MU+bDt5FW/e1gsxoRr0bxMt6zn8PoA5cqkYgiDU+828sNwxgGmI9nFhuFhU4bINwOnXbobSNhW7pTYYH9zVr06vLT533eyh2H+hCOld4q7xDCJH9rUuLUId3yR+O+FYzNs5IRx/5JRgzf5LDGBIkp1fLi2AeN/1qbUOr4sfDqsHJAEqJSZUu6diQjXILzPgQmEFAxgfsfXEFfxsm1zyj//LBAD894E09EuRL4jx+yEkiwCstf0S6yO/TAxgGv6PpoPdmjDVKV2sI1MfnRLC8df+Scy+UJ3NGmH9xDypb2solQr8bXCy0zHibXVLb2tWs/oCZOT/jGYL/vTuNjzweabDdl2lER9srVr80H6o3J7FIuBYTond39Zrfzjs3koLANh/oaieV02NLSu7yGnb8dxSGM0W2WYx+n0AAwDvbDhR719YgfgpQYYMTIf4mgMYIm+bPCAJ62YPxb8m9QAAvDKxh1RAKdo5ZyTWzR6KtrHW2ipx4UVqPk7mleLAhWL8fCgHusqqmUGL1p/Ayl3Z0uNfDufC4qL+4dOMsxi9aCtmrrROXnAngOmdFAkA+CHrEqfw+wCzRZCyL/a+zbyAPvPW48lvD8hynmYRwPyRU1KvLIwgCPjlSC4AIDpMniEkoqZKoVCgU0I4AlRVfxY+uac/nh7TGVEhAZh3SzfERwShU0K4VFReYWRRZXNjH7Seyqta3v/bzPMOx73601F89Ntpp+eLdXpiHOLOh8NJfVsjUK3E7+cKsTe7sD6XTY1o5+l8nMhzbv2Qea4QpXoTvs28IEsg2iwCGADY8Efdp3yeL6iQihcn1KFgtybtYx3XZLmunXVKNgMbaqrax4XjgWHtsG/uKNyd1kbaHmwLYPTMwDQ7xRVVWZeTtjep4gojdJXOw4kfbzvjkIW5UFjusDgdAAxMvXZNRHJMCMZ0TwAA6UMlNV3i7N3Y8JpLL3J19W9wLPL7AEYsht1ztqDOzz2Tb13mukNcGLomRjT4WrQhARjV1TpF+v4b2+HDu/vj+XFdsNy2+iSRrwgKsP7p4BBS82MfqDz57QE88c1+7DydDwBI1AY5LOdwpUQv1boUlxvxwneHAAD9UqIwMDUaPVtrcXOPlm6dt3+KdTXxsy56J1HTItbG9UmKlIb/qhPXQ2sIv5+F1NGW3bhQWAGzRXDZdLEm52wBTJsW8q2l8uHd/R0es8Ei+aJgaQiJAUxzY5+BAax1Dd9mXgAAtIsLwyMjO2BASjSe+u8BFJQZkKurRGx4ID7edhqbjln7uk0bkup24CKKCLaut1XiItNDTUup7f9RWJAab9/eG/87eBlPVat7sb6/xrp4tvv8PgMTExYItVLhtICcO87lWxeYS4l27oVE1JyJNTCVtdTAlOpN/LTsh3QVNS/p3z4uDCEaNdK7xiMx0rqG1YXCchhMFmm4acqg5DoHL0BVry4GME2f+P8oPFCN0EA1/to/CUvv7IuNj98oLVRov4Brffl9AKNSKhAZYo3cX7JbmMsd4tLVcRFcd4DIXqBtCKnCaHZZjKc3mfGX93dg5MIt2H++qJGvjjypegbGnv3invHh1gDm/s/3YtTbW3DokrWecGQ916gSVzwvqWRPpKZOHEIKs2sQO65nS7SNDUNSlLVthH1DZKPZgpd+OIzvqvW/uha/D2AAa2t2AFhnN61vx8mr+NvHu2r9hCh+0ohwo1UAUXMSbNfaQm9yzsJsPnYFf+SUwGwR8OFW55ko5LuKa2mq2C+lqut9vLZqFfGz+eXSG1Z925swA+M7xP9HYYHO751JthEN+wzMl7uzsXzHWcz+KqtO52kWAcz9N7aTvhdXwX3kqyxsO3kVE5Zsq/F54hoH4tgrEVnZ9+ZyVcgrNukDgLWHcxwek2+zz8ColQrcNTgFAPCPoY71fLf2aYUAlWPNYaI2CElR9RuSr8rAmLgWTBNXqrfeI/YZGFGqrab09NVSKaGw60zVJJtyg/sBqt8X8QLAk6M74as92SgsN+L01TIkRYfAaGsKVqI3wWCyOPTiEOkqrL9IZmCIHAWolFArFTBZBJd1MPadg80WAQcuFCEmLA7lejO0Ifz35MvEAObpMZ3x1/6tEaxRYXjnWNzY0XFoqH+baGx8fBhK9Sb89d8ZMJot+Ghqf5d/a90hZmAMZgv0JotTg9vq8nSVuFxciV41zIIhzxGHkMIDnUOMlJhQaNRKVBotOF9YjpSYUJy+UjUScqGwAgnB7p2nWWRgVEoFBtiaSIkzi7rZTYvu+PzPOOBiieqqDEyziPOI6qSqkNc5A2MfwADWP0r3LtuDga/96vLfGvkOMYDp3DIcMWGBCNGoMaJzvMsZnknRIejSMgI/zBqCnx8Zim6J2nqfN0yjllpZ5Ooq8dYvx3DqivNiaQCQX6rHzYt/w8T3tuNYTsOn61LdFJTV/N6pUirQPtY6OzjrfBFK9SactPv/mJ3vfnFvswhggKpCXLEexlBt3P7Jbw5Ibb/PXC3Dq/87gsu2tDczMETOxOGBJRtPOu0TAxjxDWfemiPYdvIq9CYLXv3fUQiCgHKDSfpAQb5D/GCnrcPQemqLUGnooL6USoV0zhvf2IwlG09i/GLXJQA/H8rB1VIDBAH47cSVBp2X6qbCYMbRyzoAzou3ikZ0tmbrHvkyC91fXOfwfpxbh9nCzSaAEZsxLt5wAtn55VLa+6/9W0OjVuJYbgl+3H8JAPCvn4/io9/OSM9lDQyRM7FhqPjHShAE7DlbgOJyI/LLrKtsdnfxiXvXmQKkzvkJXeeuw7A3N2P1vguNd9HUYGIGpi4BjFwm2pqIiiqMZpcLop20W8a+eid18hy9yYzB8zcAsH54SYp2PRY0bUhqjav01jbLrbpmE8C0sOtlNPadrdCbrGnvib1b4ZGRHQAASzaeAACcyHVMS4a7KEQiau5W/H0QAOuwrCAI+HznOdz2QQZ6zfsFO09bi/LELsIAcN/1qbi1j+MbkCAAj361HwcvFNfp3BaLgKJyw7UPJFmZLYI0w8QrAUy1+wcARr29FW2e+R8+3XEWgLUL9nLb94A1A3OpqMLpefW1L7sQ//r5j1pnYzVXn2w7KwUgY7olSB9yqosK1eDpMZ0dtokBTV1+r80mgLHveFpmMKNMbw1gAgNUmDIoGQBw6koZKgxmxEdUTf+79/o2Ds3tiMiqXWwYVEoFygxm5Or0eGPdMadjbh+QJH2fHB3s8g0IAF7+3xEs3nBCWpL+Wl784TD6vrweW49zeKAx2a/B4o0AJrWWKdgv/nAYmecKMPnfO6VtQQFKWATgyz3na3xeXc39/jA+2HIKNyzYiFwdZ9fZ+3KPtRv5Yzd1xDuT+9R67K19WuH5cV3w/pS+WHbPANxtm81WlwxMs0kthFarhr5UbI3IA9VKRIZoEB6oRonehItFFdIvcPm9AzCsU/0WXSLydxq1EinRITh9tQwZp686NfNTKRXo0UqL6Tek4tejeRjfKxEtwgKxcvogxIYFok2LUBy4UIRJ72dg95kC7LZNpTz7r3E1nlNvMmP84m1Sp9tX/3cUQzs2bDlycp/YgC88SO2VD3bakACEalQoM5ixbvZQjF601WH/pPczpO8fu6kjUmJC8MiXWVi84QRCNSpMv6EtlHVoJ1PdpaKqBr+6ShM+yziLJ0d3vsazmodcXSXO5ZdDoQDuub7NNWebKZUKh1Y6FwqtxbtF3szAzJ8/HwMGDEB4eDji4uIwceJEHDvm+MmssrISM2fORExMDMLCwjBp0iTk5jp2GM3Ozsa4ceMQEhKCuLg4PPnkkzCZ6r+AUatIx7E4cRkBcSZFK9vqgPYBjDc+YRD5kra22QSr9lpX0OwYX9VZPTpUA5VSgefGdcWmJ4ahRZg1RXxduxboEB+OAJVS2maveoG9ve0nr0rBCwAcyy3Bol+Ps6lkI/kjx1rv1DHedXFmY1j36FD8/MgN6JQQjsdv6uhy9tNHd/fHwyM7SMWiADD/5z+w7nBOg869cP1xh8ffZ12SluRo7sSGyV0SIuo18UUbYh0lKapwf2hY9gBmy5YtmDlzJnbu3In169fDaDRi1KhRKCurmm3w6KOP4scff8Q333yDLVu24NKlS7j11lul/WazGePGjYPBYMCOHTvw6aefYvny5Zg7d269r6tjfDjevdM5pSV21U20BThTP9ktLXbH4l2i2rWLtab0M05Zh37axITikZEdEB2qwQN2C0jWxH5oV3SmltWxXX06W/TrCYx8awuulurdvWyqB0EQsOOk9f9z5wTvBTCto6xTswHgoZEdcOq1m/HVjMHS/ufHdcFNXeMBWBe/m35DqrTv7V+P1xogX4vYFuPt23uhRZgGFworsP5Ibu1Paib22DKoA1Oj6/X8SNv7bfUlGGojewCzdu1a3HPPPejWrRt69eqF5cuXIzs7G5mZmQCA4uJi/Oc//8HChQsxYsQI9OvXD8uWLcOOHTuwc6d17PKXX37BkSNH8Pnnn6N3794YO3YsXn75ZSxduhQGQ/0L98b3TER6tT4cYgamk4t/kMzAENWuva3bu8m2BIE2OACP3tQRe1+4CfcNSa3tqQCAsEC102qtoxdtRfrCLXh34wmczCvB8u1nkJ1fDotFkJY2AICVfx+Eib2tvXcuFlXg7Wqfjkle7285ha9+t9aS9God6d2LqWZQ2xgceGkUzsy/2WFYAgCeG9cVO+eMRKBaieO5pdhztgC/nbhS5yn8giDgQqH1w22fpCip79Nuu1Vkm7N9tuCuf5uo2g+sQUtb64njuaXYe86936nHBzGLi63jhdHR1qgsMzMTRqMR6enp0jGdO3dGcnIyMjKs45cZGRno0aMH4uPjpWNGjx4NnU6Hw4fr1pCxug7VUp9iAHNbv9ZOx3L9F6La3dgpFvYZ/LoG/QqFwmUW5mReKd785TjSF27FSz8ewdA3NqHtsz9JhcIPjWiP69q3wKLJffD5NOtsqJ8OXq7/D0LX9N/Mqunuo7rF13Kkd0QEBdQ46yVBGyRlBpZtP4u7/rMbYxb9htM1LITnytVSAyqMZigU1oy92Pcp81xhva/5QmE5rpT4R+ZQ7G0kfqipq/ZxYfhTL2tQuGCdex9GPBrAWCwWzJ49G9dffz26d+8OAMjJyYFGo0FkZKTDsfHx8cjJyZGOsQ9exP3iPlf0ej10Op3DlyuD28Y4PA6yFRq1jQ3DZ/cNxOQBSVApFejSMqLeS14TNRdx4UEY2aXq32pkPdoERIVUBTB3DEyCOzWWraOqatq6tLR+KCksN7IewYPibN2lJ/RKRGSIc9DZ1CXYZpf+etQ65FNhNGPEW1uQ6eanfTFj0zIiCBq1UnqjrmmK9pUSPe76zy4Mf3Ozy0CpsMyAIa9vQvrCLU69ncRFVX1FcYVRGt5NjHSzD0A1CoUCL4zvCoUCOHTRvWUVPDoLaebMmTh06BC2bau5YaJc5s+fj3/+85/XPM6+yFClVEBtV0k/tGMshnaMxYsTuqGGQJ6IqrljYJJUB1CfYdebusbjj5wSdG8VgSdHd8a0IW2ta0IIwIY/cpEYGYznVh/EKVu/lAFtonBzj5bS8yNDNFAorIX5heUG6Y2W5CWuwHtrX9dT4Zu6BK3r+2LS+xmICgnAT4/cgJbamt9892UXAQC62dY2ipKKTo0QBMEp+/Pq/45Ii+h9uec8nr25i8P+/baWGsUVRoxfsg3P3dwF17VvgbfXH8fHv53GI+kd0CoyBMM7xyJE03QnDBeXG9Fr3i/S44aMXMSGB6J7ohb7T7vXTsBjv5VZs2ZhzZo12Lp1K1q3rhqeSUhIgMFgQFFRkUMWJjc3FwkJCdIxu3fvdng9cZaSeEx1c+bMwWOPPSY91ul0SEpKcjrO/tNeTVFusKb2JmFEVKVdbNWHgvoUvj8+qhNmDm8vDefaDynd2tf6t+Pfd/XH5mN5+NvgFKcmfiqlAtEhGuSXGZBfygCmJifzShETqkGUiyE7d/j67Ez79b0AYPoNqdKK64XlRryx9hje+msvl8NQlUYzVuw6BwDobxs6ErONZouAEr3J6Y17z9mqoaXvsy4iJlSDqde1ke5f+waGhy/pcNcnu7Hkjj54Z4N1QdXXfvpD2n/HwCQ8elPHJndvWywCvsmUb40dwLp67/7T7h0r+xiJIAiYNWsWVq9ejY0bNyI11bGQr1+/fggICMCGDRukbceOHUN2djbS0tIAAGlpaTh48CDy8vKkY9avX4+IiAh07drV5XkDAwMRERHh8OXKtTqYElHd2C9RUGGo33Tma/27bB8Xhr/f0LbG48Sgpy4zGJqTU1dKkb5wC/q8vF56I64rXw9gxIa+gHUI8rlxXXHy1bEY2936oXjVvovYfaYAZouAwjIDjGYL5n5/CGPf+Q3PrT6Es/nlCFQrMa6nNfsXFKCSZrEWlRnx3b6LuPW97Th7tQzFFUZpNitgXT9n/s9/OKwQbL8cAGANhB5csdfltX+x+zwGvroBd3y402ExQW976Mt9eOV/R6XHi++offE6dwQHuJ9XkT2AmTlzJj7//HOsXLkS4eHhyMnJQU5ODioqrP8ztVotpk2bhsceewybNm1CZmYm7r33XqSlpWHwYOtUuFGjRqFr16646667sH//fqxbtw7PP/88Zs6cicBA1/0T6iOQNS5EDWY/DGvfOqAxiQHMx7+dZvdhO7m6Sqw7nINfDldN9X1u9aE61VhcKqrA0k0npRYCvjq5oVNCOD74W1+0iQnBPde1AWC9d9+b0leqZ/n9XCFGL9qKga/9ivuW78FnGedw9LIO/91rLWCeMigFraNCpNcUM/oXisox+6ss7M0uQvrCLXh29UGX1/B91iXpd3/F1rRwTLcE3N6/arQgRKPC59MGIcQ2EmBfFJtxOh/vbnJunupp5QaT0zIFepMZ/ztQVTj/8d39pSLchgjWuP++LPsQ0vvvvw8AGDZsmMP2ZcuW4Z577gEAvP3221AqlZg0aRL0ej1Gjx6N9957TzpWpVJhzZo1eOCBB5CWlobQ0FBMnToV8+bNk/VaWaRLJI9tTw/H+YIKrwUw7eLCsOtMATYdu4KtJ64ia+5NCPfRN1o5TXp/By4UViCs2krkRy/r0L2VFt9nXcTrP/+B7q20aBsbhodGtHdatXzi0u3Is5sp46sZGAAY070lxnRv6bBNoVDgxo6xOJlXird+OQYxtqveBFKpAMb3cnxuZIgGl4srcedHu6RtJosgvbEvvqMPggNU2H7yKpbvOIujl3Vo9+xP6N4qAocuWieaTOzTCsM7W+svz+aXYUTnOHRpGYHNTw6DSqFATFggfjtxBXf9x1pW8eHW0xjbvSV6J0XK+aupkSAImPrJbvx+rhDz/9wDkwdaW++sPVQ1oeahEe0xrJM8K2LXpd5H9gCmejW1K0FBQVi6dCmWLl1a4zEpKSn46aef5Lw0J4FqDicRyaF1VIjDJ9PG9mh6R6zcZe3DYrYI+PlgDv46wLkGrrkR1y0p1TuuYi6uXfLIl1kAgEvFlQByoVLCYWn8i0UVDsFLeKDaLz/4tYmx3rvVE1MD2kThyxlpKNWbUGk0O9XRRFYL5m7qGo+zV8twIq8UceGBGNs9AQEqJW7qGo/eSZF4/Jv9MFsEKXgBrI2GA9UqaWhKZF/vckOHWPzx8hiMXrQV5/LLMW35HuyYM8Lj72F6kxkLfzku1fMs2XgS43slIlCtxHxbjc6dg5Lx+KhOsp0zuA5lHv53J7ohwtZd+oYOLbx8JUQkh9jwQIfutvYL3jVX1ZvitdQG4ZGRHQAA89YcwfglzrNDv9x9Hhbbu/ihi8W45d3tDvvvtDW+9TedWzrWTD52U0fc1q81/n1Xf6iUCmiDA5yCFwDo3qrqeZ3iw/HSn7phxd8H4ZWJ3bFy+iCHflET+7TCu3f0QZ/kSIfXiHHRTsOVoAAVVj94PZQKIL/MgHP57s3UaYgnvzmAf2+tqqi9WFSB7i+uw5KNJ5Gjq0SLMA2eH9ellleou5A6TKJplgHMqgevw4PD2uGlCd28fSlEJJP7b2yLB4ZZ2xcUlvtuMe+3mRcwZ9XBBq8FUn3tkX8Mbeuy8eUrE7vjh1nXA7C+Me48kw9BEPDs6oMOdQ9Pj+mMJ0fL90m7KemfEoUJvRKhVirw1YzBeHhkB7xxWy+Xiyzae3BYe/RJjsSMoW2x7tGhaBUZjLiIIPxtcAraxzmv7j62R0usfvB6vDyxu7TtWuewFx2qQQ/bKsin8txfhK++fth/yeX2xbaZUncMTJZ9inddZgE33cnlHtQ+LhxPjWEHUSJ/olAoEGN7M2hoACPOZqr+5rLj5FVcKdVjXI+WDsXLcinVm/DEN/sBAGO6J+DGBnTazqu2wuvkgckIVCsxeUAS1hy4jBs7xqJnay1u698agWoVbu3bCqv2XsSGo3n4ZNtZHLhQtZjYn/u0koJDf6RQKLB4cm+Y/tqrTl22o0I1WP3g9XU+Xxe71jXiiIC72sWGYv/5IpyqwyrC9WG/KGTP1losur039mYXSfcnAPxtcIrs563LEFKzDGCIyD+JK8Q2ZDq1wWRB35fXAwBOvDoWASolisoNeH/zKSmdvumPPCya3PApo9VtPX5F+r6sWt1KXdn/DtY/OlSagv6vST3x8sTuTm/UPVtpsWrvRfxn2xmH7f1SovDYTR0bdC2+QKFQOPXl8pR+KVF4aER7JGiDamx/UJOuLSOwChexL7sI5wvKMW/NESRHh2DO2M6yBtXihwCFAlj94PVQKRVoGxsGBYA5qw7itv6tXQ6rNZRXi3iJiLwlyra4mKuu1e6yHzbJK9GjVWQwlm0/61AL8F3WJcRFBGHO2M51fgOqzR+Xq4o7G7qmjfj82/q1duoB5yrL4GoG2efTBmEIawVlp1Ao6l34KvZ02ngsDxsWVK2Vll1QjtQWobhzYDLOXC1Dt8QIxDUgwMgvtWUhQzRQ2fX3mNSvNW7u0VJaA0dudamBYQBDRH4jSoYhJPvgJ1dXiVCNCgdsy74D1nU5TuaV4sOtp9G2Rag0rbQhzuWX4Z0NJ7Bq70VpmzsBjKsl7EXSG1CYezUWfZOjpNVpgwNU+L9pA9HfbvE3ahq6toxAh7gwp4XwxHYeH9oC7YggNf738A1Iiq7b7MBcXSXydHqst/WMclWj48nV6lkDQ0TNUqxtRkdeiR5mi+DwydFdRXbBz/ubT0lvDADw7p19cEOHWNz9n13Yf6EYu84UyBLAPPxlFvafL3LYdq0A5sXvD2H1vov4YdYQtGkR6rBv/ZFcfLLdOhQU42aRqFKpwHPjuuK5cV1hsQhQ1uN3R56nVinx9T/S8Pg3+7Hxjzw8MaojWmqD8bhdbQoA6CpNuOntLRjfMxFTBiWjT3LUNV/70MVi3PWfXSi0C+Jj3AyA5cJp1ETULCVGBiNQrYTBZEF2Qf2mmRbYBTD2wQsAtIkJhTY4AFNtK7leqVYoWx9Gs0Xqvmu/6uoP+y9JQUxeSSU2H8uT1nMRBAGfZpyDrtKEYW9uxh85OmkNrvVHcjH9s9+l14kOrfvq5QxemraoUA0+vrs/1s6+AQ8Ma49J/Vrj4Euj8MHf+uHN23rhm/vT0FIbhEqjBd9mXsCt7+/AjzXMKBIZzRbcs2yPQ/ACAKO7ue4/6ClRIRr0S7l2sAUwA0NEfkSlVKB9XBgOX9Lhv5kX8EQ9pv1W/wNuL9m24FkLW6an+vLq9XG+oBxmi4DgABXWPzoU6w7n4v7PM1FQZsCwNzbh2Zu7YMnGk7hYVIGYUA0evakjhrR3rEsZs+g3aNRKjOwch5/tVkjVBge4/WZAvkWpVKBzQtU6NOFBARjTvSrY2PTEMPx24io+33kOW45fwVd7zmNCr0QYTBZYBAFBASp8uPUUjl4uwYsTumLL8SsO93Na2xgsv29Aoy/4mhwTgk/vGwjto9c+lgEMEfmVLi0jcPiSDp9lnMXjozrWuci2yMXQTVigGrPTO0h9gOwDmHP5ZUiMDK7T9Ft7Z65auxK3aREKhUKBEZ3jkN4lDr8ezYOu0oRnVlX11ckvM+D57w4h1EWdgMFkcQhe/vvAdejaMsKj9QrUdAUFqHBT13i01AZhy/Er2HbyKia9vwN5JZUo05vx4oSuUsfr1fuqaq9mDW+PvimR6Jcc3eRXq2cAQ0R+Zc7Yzvg28wJ0lSZcKq506JbtjvxqAUyn+HCsnX2DQyDUItxaF3C11IAb39iMzgnh+O8D1zn1EXLH8VxrMWa7WGsdi0atxEd394fZImDOqoP4JtPaSPChEe3RIiwQi349jsJyI0I0Knx230D0bB2J47kl+D7rIr7LuoQrJXqs+PsgZl4IANAxPhxBAUpUGi3IPFcobRfbSNjrnBCOB4e3k31xOk/xjaskInJTTFggOsWH41huCR7/OguhGjViwwMx/9YebmVjLhZZ+wfdMTAJFwor8IyLqdIxoYHSmwIA/JFTgm4vrsOIznFYfEcfp+aJ1W06loc9ZwqgViqwbMdZAEDXxKrhAIVCAbVKgQV/6YnzheXYm12EiX1aoV1sGCb2aYVVey9gUGqM9JzurbTo3kqLR9I74vSVUvS0rdZKpFEr8e4dffHEt/tdLi/QLjYU5QYziiuM+Oju/j4TvACAQnCn+6IP0ul00Gq1KC4uRkRExLWfQER+47GvsrDKLi0OAM+M7YypaW1qHVKpNJrR+YW1AIBP7umPEZ3jazx2xa5zeG71Iaftt/dPwqt/7l7jomJLNpzAW+uPO23/9L6BLlferTSaUao3ScNWRPVhMlvw0W/WmWlF5Qb8e+tpRIUE4IdZQxAdqkGZweTQQNKb3H3/ZgBDRH7naqke/V/51Wl729hQrJs91GW9islswa3v75CW0F87+waHIklXdp3Oh95kwcY/8rDclkkBrGtwpHeNxz3XtXHIhgiCgNQ5Pzm9znXtYvDJPQOk1XKJPE0QBJgsQr1rtzzJ3fdv38kVERG5qUWYtTv162v/cNh++koZvvn9gsuuyqeulDn0/3GndmZQ2xgAwNCOsXhuXBekL9yCc/nl0FWasGrvRaw7lIN1jw5F6yjr7CVdRVV7gDdv64WRneOkxfeIGlNjtk7wFAYwROSX/jG0rRTAjOvZEn2SIvHK/47itZ+OYmjHFlJQIfojp2oZ/3/+qRvCbTOO3BWgUmL1g9cjv1SPbSev4qOtp3GpuBJDXt8EAOjVWoubulqHpKJDNfhLv9YN+fGImr2mlzsiIpKBUqnAsnsGoFdrLR6/qSPuvT4V/VOiUKo3Ydn2s07Hi4vJTRmULC1UV1fRoRp0iA/Hvden4q2/9nbYt/9CMd78xVr7EhlSt+CIiJwxgCEivzW8cxy+nzUEbWPDoFIqMH1oWwDA2kM5qF7+t/EPa2M8sVleQw1MjUZStHUYKjIkALHhVUW4nRPCa3oaEbmJQ0hE1GwM7RCL4AAVLhZVoMvctfj2/uvQvZUWBy4U4dSVMmhUSgzvHCfLuVRKBf57/3UoN5ilXkVHLunw08HL+FPvRFnOQdScMQNDRM1GsEaFYZ2sU5UrjRY8/d8DsFgEqQv0zT0SpNV25RAXEeTQaLFrYgSeGN0JHeOZgSFqKGZgiKhZeX58V0SGaPDF7mwcvqRD73m/wGyxDifJlX0hIs9jBoaImpVWkcGYf2sPvDShKwBAV2lCmcEMwLqiLRH5BgYwRNQs3XN9KtbNHgqV0roWRqvIYKTGhF7jWUTUVHAIiYiarU4J4Vh2zwB89ft5zBzWHkqlby/sRdScMIAhomZtaMdYDHXRg4iImjYOIREREZHPYQBDREREPocBDBEREfkcBjBERETkcxjAEBERkc9hAENEREQ+hwEMERER+RwGMERERORzGMAQERGRz2EAQ0RERD6HAQwRERH5HAYwRERE5HMYwBAREZHPYQBDREREPkft7QvwFEEQAAA6nc7LV0JERETuEt+3xffxmvhtAJOfnw8ASEpK8vKVEBERUV3l5+dDq9XWuN9vA5jo6GgAQHZ2dq2/gJoMGDAAe/bsqff5G/J8Xzy3TqdDUlISzp8/j4iIiEY9ty/+vrz5XG8/3xef66v3t6/eY756bl+97obe33Jfd3FxMZKTk6X38Zr4bQCjVFrLe7Rabb3+h6hUqnr/oWro83353BEREY1+bl/9ffnqPdLQ5/vic0W+dn/76j3mq+f21esW1ff+9tR1i+/jNWERbw1mzpzptef78rkbor7n9tXfl6/eIw19vi8+Vw7euL999R7z1XP76nU3lLeuWyFcq0rGR+l0Omi1WhQXFzc4KqVr4++b/Bnvb/JnTe3+dvd6/DYDExgYiBdffBGBgYHevpRmgb9v8me8v8mfNbX7293r8dsMDBEREfkvv83AEDU3CoUC3333nbcvg8gjeH9TdQxgiJqoe+65BxMnTvT2ZRB5BO9vaqhmF8AwiiciIvJ9Ph/AMIpvHPw9e1ebNm2waNEih229e/fGSy+95JXr8Te8v72L97fn+PO97fMBDBERETU/fhXAMIpvHGvXrsWQIUMQGRmJmJgYjB8/HqdOnZL2nz17FgqFAqtWrcLw4cMREhKCXr16ISMjw4tXTeQe3t/kr/zt3varAIYaR1lZGR577DH8/vvv2LBhA5RKJf785z/DYrE4HPfcc8/hiSeeQFZWFjp27Ig77rgDJpPJS1dN5B7e3+Sv/O3e9tteSOQ5kyZNcnj8ySefIDY2FkeOHEH37t2l7U888QTGjRsHAPjnP/+Jbt264eTJk+jcuXOjXq8/UCqVTq3ljUajl67Gv/H+bny8vxuHv93bzMBQnZ04cQJ33HEH2rZti4iICLRp0waAtfO3vZ49e0rft2zZEgCQl5fXaNfpT2JjY3H58mXpsU6nw5kzZ7x4Rf6L93fj4/3dOPzt3varDAyj+MYxYcIEpKSk4KOPPkJiYiIsFgu6d+8Og8HgcFxAQID0vUKhAACnVCW5Z8SIEVi+fDkmTJiAyMhIzJ07FyqVytuX5Zd4fzc+3t+Nw9/ubb8KYBjFe15+fj6OHTuGjz76CDfccAMAYNu2bV6+Kv9ksVigVlv/ic6ZMwdnzpzB+PHjodVq8fLLL/Pe9gDe342H93fj8sd7268CGEbxnhcVFYWYmBh8+OGHaNmyJbKzs/HMM894+7L8Ul5eHtq3bw8AiIiIwJdffumwf+rUqQ6P2das4Xh/Nx7e343LH+9tn6+BqR7F33jjjRg/fjzGjRuHiRMnol27dl6+Qv8g/p6VSiW+/PJLZGZmonv37nj00UfxxhtvePvy/EphYSHWrFmDzZs3Iz093duX0yzw/m48vL8blz/f2z7fjXrMmDFo37493n33XW9fil/j77nx/PnPf8aePXswdepUvPLKK9IYNHkO7+/Gw/u7cfnzve2zQ0iFhYXYvn07Nm/ejPvvv9/bl+O3+HtufKtXr/b2JTQbvL8bH+/vxtEc7m2fDWDuu+8+7NmzB48//jhuueUWb1+O3+LvmfwZ72/yV83h3vb5ISQiIiJqfny+iJeIiIiaHwYwRERE5HMYwBAREZHPYQBDkvnz52PAgAEIDw9HXFwcJk6ciGPHjjkcU1lZiZkzZyImJgZhYWGYNGkScnNzHY55+OGH0a9fPwQGBqJ3794uz7Vu3ToMHjwY4eHhiI2NxaRJk3D27FkP/WTU3DXmvf3111+jd+/eCAkJQUpKis+vtUFNnxz39/79+3HHHXcgKSkJwcHB6NKlC9555x2nc23evBl9+/ZFYGAg2rdvj+XLl3v6x6sRAxiSbNmyBTNnzsTOnTuxfv16GI1GjBo1CmVlZdIxjz76KH788Ud888032LJlCy5duoRbb73V6bXuu+8+3H777S7Pc+bMGdxyyy0YMWIEsrKysG7dOly9etXl6xDJobHu7Z9//hlTpkzB/fffj0OHDuG9997D22+/7ZdrcFDTIcf9nZmZibi4OHz++ec4fPgwnnvuOcyZM8fh3j1z5gzGjRuH4cOHIysrC7Nnz8bf//53rFu3rlF/XolAVIO8vDwBgLBlyxZBEAShqKhICAgIEL755hvpmKNHjwoAhIyMDKfnv/jii0KvXr2ctn/zzTeCWq0WzGaztO2HH34QFAqFYDAY5P9BiKrx1L19xx13CH/5y18cti1evFho3bq1YLFY5P0hiGrQ0Ptb9OCDDwrDhw+XHj/11FNCt27dHI65/fbbhdGjR8v8E7iHGRiqUXFxMQAgOjoagDVCNxqNDst/d+7cGcnJycjIyHD7dfv16welUolly5bBbDajuLgY//d//4f09HSHLqhEnuKpe1uv1yMoKMhhW3BwMC5cuIBz587JcOVE1ybX/V1cXCy9BgBkZGQ4tX8YPXp0nf6NyIkBDLlksVgwe/ZsXH/99ejevTsAICcnBxqNBpGRkQ7HxsfHIycnx+3XTk1NxS+//IJnn30WgYGBiIyMxIULF/D111/L+SMQueTJe3v06NFYtWoVNmzYAIvFguPHj+Ott94CAFy+fFm2n4GoJnLd3zt27MBXX32FGTNmSNtycnIQHx/v9Bo6nQ4VFRXy/iBuYABDLs2cOROHDh1y6hArh5ycHEyfPh1Tp07Fnj17sGXLFmg0GvzlL39hx1nyOE/e29OnT8esWbMwfvx4aDQaDB48GJMnTwYAKJX8c0ueJ8f9fejQIdxyyy148cUXMWrUKBmvTl78F0VOZs2ahTVr1mDTpk1o3bq1tD0hIQEGgwFFRUUOx+fm5iIhIcHt11+6dCm0Wi0WLFiAPn36YOjQofj888+xYcMG7Nq1S64fg8iJp+9thUKB119/HaWlpTh37hxycnIwcOBAAEDbtm1l+RmIaiLH/X3kyBGMHDkSM2bMwPPPP++wLyEhwWlmXm5uLiIiIhAcHCzvD+MGBjAkEQQBs2bNwurVq7Fx40akpqY67O/Xrx8CAgKwYcMGaduxY8eQnZ2NtLQ0t89TXl7u9GlUpVIBsKY/ieTWWPe2SKVSoVWrVtBoNPjiiy+QlpaG2NjYBv8cRK7IdX8fPnwYw4cPx9SpU/Hqq686nSctLc3hNQBg/fr19fo3IguvlA5Tk/TAAw8IWq1W2Lx5s3D58mXpq7y8XDrm/vvvF5KTk4WNGzcKv//+u5CWliakpaU5vM6JEyeEffv2Cf/4xz+Ejh07Cvv27RP27dsn6PV6QRAEYcOGDYJCoRD++c9/CsePHxcyMzOF0aNHCykpKQ7nIpJLY93bV65cEd5//33h6NGjwr59+4SHH35YCAoKEnbt2tWoPy81L3Lc3wcPHhRiY2OFv/3tbw6vkZeXJx1z+vRpISQkRHjyySeFo0ePCkuXLhVUKpWwdu3aRv15RQxgSALA5deyZcukYyoqKoQHH3xQiIqKEkJCQoQ///nPwuXLlx1e58Ybb3T5OmfOnJGO+eKLL4Q+ffoIoaGhQmxsrPCnP/1JOHr0aCP9pNTcNNa9feXKFWHw4MFCaGioEBISIowcOVLYuXNnI/6k1BzJcX+/+OKLLl8jJSXF4VybNm0SevfuLWg0GqFt27YO52hs7EZNREREPoc1MERERORzGMAQERGRz2EAQ0RERD6HAQwRERH5HAYwRERE5HMYwBAREZHPYQBDREREPocBDBEREfkcBjBE5DX33HMPFAoFFAoFAgICEB8fj5tuugmffPJJnfpiLV++HJGRkZ67UCJqchjAEJFXjRkzBpcvX8bZs2fx888/Y/jw4XjkkUcwfvx4mEwmb18eETVRDGCIyKsCAwORkJCAVq1aoW/fvnj22Wfx/fff4+eff8by5csBAAsXLkSPHj0QGhqKpKQkPPjggygtLQUAbN68Gffeey+Ki4ulbM5LL70EANDr9XjiiSfQqlUrhIaGYtCgQdi8ebN3flAikhUDGCJqckaMGIFevXph1apVAAClUonFixfj8OHD+PTTT7Fx40Y89dRTAIDrrrsOixYtQkREBC5fvozLly/jiSeeAADMmjULGRkZ+PLLL3HgwAHcdtttGDNmDE6cOOG1n42I5MFmjkTkNffccw+Kiorw3XffOe2bPHkyDhw4gCNHjjjt+/bbb3H//ffj6tWrAKw1MLNnz0ZRUZF0THZ2Ntq2bYvs7GwkJiZK29PT0zFw4EC89tprsv88RNR41N6+ACIiVwRBgEKhAAD8+uuvmD9/Pv744w/odDqYTCZUVlaivLwcISEhLp9/8OBBmM1mdOzY0WG7Xq9HTEyMx6+fiDyLAQwRNUlHjx5Famoqzp49i/Hjx+OBBx7Aq6++iujoaGzbtg3Tpk2DwWCoMYApLS2FSqVCZmYmVCqVw76wsLDG+BGIyIMYwBBRk7Nx40YcPHgQjz76KDIzM2GxWPDWW29BqbSW7X399dcOx2s0GpjNZodtffr0gdlsRl5eHm644YZGu3YiahwMYIjIq/R6PXJycmA2m5Gbm4u1a9di/vz5GD9+PO6++24cOnQIRqMRS5YswYQJE7B9+3Z88MEHDq/Rpk0blJaWYsOGDejVqxdCQkLQsWNHTJkyBXfffTfeeust9OnTB1euXMGGDRvQs2dPjBs3zks/MRHJgbOQiMir1q5di5YtW6JNmzYYM2YMNm3ahMWLF+P777+HSqVCr169sHDhQrz++uvo3r07VqxYgfnz5zu8xnXXXYf7778ft99+O2JjY7FgwQIAwLJly3D33Xfj8ccfR6dOnTBx4kTs2bMHycnJ3vhRiUhGnIVEREREPocZGCIiIvI5DGCIiIjI5zCAISIiIp/DAIaIiIh8DgMYIiIi8jkMYIiIiMjnMIAhIiIin8MAhoiIiHwOAxgiIiLyOQxgiIiIyOcwgCEiIiKfwwCGiIiIfM7/Axzu8QDVRnSsAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "highs.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop(columns='Symbol', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-07-01 11:00:00</th>\n",
       "      <td>279.98</td>\n",
       "      <td>279.99</td>\n",
       "      <td>272.1</td>\n",
       "      <td>275.01</td>\n",
       "      <td>679358.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-07-01 12:00:00</th>\n",
       "      <td>275.01</td>\n",
       "      <td>275.01</td>\n",
       "      <td>271.0</td>\n",
       "      <td>274.83</td>\n",
       "      <td>824362.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-07-01 13:00:00</th>\n",
       "      <td>274.83</td>\n",
       "      <td>274.93</td>\n",
       "      <td>265.0</td>\n",
       "      <td>268.79</td>\n",
       "      <td>3010787.99</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       Open    High    Low   Close      Volume\n",
       "Date                                                          \n",
       "2017-07-01 11:00:00  279.98  279.99  272.1  275.01   679358.87\n",
       "2017-07-01 12:00:00  275.01  275.01  271.0  274.83   824362.87\n",
       "2017-07-01 13:00:00  274.83  274.93  265.0  268.79  3010787.99"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To resample multiple columns at once  \n",
    "ex: resample by day but different \n",
    "\n",
    "can do this by resampling entire data frame with `.resample()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-07-02</th>\n",
       "      <td>268.066486</td>\n",
       "      <td>271.124595</td>\n",
       "      <td>264.819730</td>\n",
       "      <td>268.202162</td>\n",
       "      <td>2.185035e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-07-09</th>\n",
       "      <td>261.337024</td>\n",
       "      <td>262.872917</td>\n",
       "      <td>259.186190</td>\n",
       "      <td>261.062083</td>\n",
       "      <td>1.337349e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-07-16</th>\n",
       "      <td>196.193214</td>\n",
       "      <td>199.204405</td>\n",
       "      <td>192.722321</td>\n",
       "      <td>195.698393</td>\n",
       "      <td>2.986756e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Open        High         Low       Close        Volume\n",
       "Date                                                                    \n",
       "2017-07-02  268.066486  271.124595  264.819730  268.202162  2.185035e+06\n",
       "2017-07-09  261.337024  262.872917  259.186190  261.062083  1.337349e+06\n",
       "2017-07-16  196.193214  199.204405  192.722321  195.698393  2.986756e+06"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.resample('W').mean().head(3)\n",
    "#wont work if the 'DayOfWeek' & 'Symbol' column from earlier is not dropped since they are non-numeric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but  not all columns make sense with mean.  \n",
    "for high we need max, for low we need min  \n",
    "so we can resample using `.agg()` method  \n",
    "also accepts a map of columns and aggregation functions we need to do on those columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Close</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017-07-02</th>\n",
       "      <td>268.202162</td>\n",
       "      <td>293.73</td>\n",
       "      <td>253.23</td>\n",
       "      <td>8.084631e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-07-09</th>\n",
       "      <td>261.062083</td>\n",
       "      <td>285.00</td>\n",
       "      <td>231.25</td>\n",
       "      <td>2.246746e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-07-16</th>\n",
       "      <td>195.698393</td>\n",
       "      <td>240.33</td>\n",
       "      <td>130.26</td>\n",
       "      <td>5.017750e+08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Close    High     Low        Volume\n",
       "Date                                                \n",
       "2017-07-02  268.202162  293.73  253.23  8.084631e+07\n",
       "2017-07-09  261.062083  285.00  231.25  2.246746e+08\n",
       "2017-07-16  195.698393  240.33  130.26  5.017750e+08"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.resample('W').agg({'Close': 'mean', 'High': 'max', 'Low': 'min', 'Volume': 'sum'}).head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "![Alt text](image.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11 - Reading/Writing Data to Different Sources - Excel, JSON, SQL, Etc\n",
    "\n",
    "https://youtu.be/N6hyN6BW6ao?si=kFLFwe_sntgN_jim\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>No</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>Bosnia and Herzegovina</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, desktop or enterprise applications;...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I am actively looking for a job</td>\n",
       "      <td>I've never had a job</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Financial performance or funding status of the...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C++;HTML/CSS;Python</td>\n",
       "      <td>C++;HTML/CSS;JavaScript;SQL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Windows</td>\n",
       "      <td>Django</td>\n",
       "      <td>Django</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Atom;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Instagram</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>No, I knew that Stack Overflow had a job board...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>19.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist  \\\n",
       "Respondent                                                    \n",
       "1           I am a student who is learning to code      Yes   \n",
       "2           I am a student who is learning to code       No   \n",
       "\n",
       "                        OpenSourcer  \\\n",
       "Respondent                            \n",
       "1                             Never   \n",
       "2           Less than once per year   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "2           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment                 Country  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work          United Kingdom   \n",
       "2               Not employed, but looking for work  Bosnia and Herzegovina   \n",
       "\n",
       "                   Student                                            EdLevel  \\\n",
       "Respondent                                                                      \n",
       "1                       No                          Primary/elementary school   \n",
       "2           Yes, full-time  Secondary school (e.g. American high school, G...   \n",
       "\n",
       "           UndergradMajor                                           EduOther  \\\n",
       "Respondent                                                                     \n",
       "1                     NaN  Taught yourself a new language, framework, or ...   \n",
       "2                     NaN  Taken an online course in programming or softw...   \n",
       "\n",
       "           OrgSize                                            DevType  \\\n",
       "Respondent                                                              \n",
       "1              NaN                                                NaN   \n",
       "2              NaN  Developer, desktop or enterprise applications;...   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "2                NaN         17          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant                          JobSeek  \\\n",
       "Respondent                                                     \n",
       "1               NaN     NaN                              NaN   \n",
       "2               NaN     NaN  I am actively looking for a job   \n",
       "\n",
       "                    LastHireDate LastInt FizzBuzz  \\\n",
       "Respondent                                          \n",
       "1                            NaN     NaN      NaN   \n",
       "2           I've never had a job     NaN      NaN   \n",
       "\n",
       "                                                   JobFactors  \\\n",
       "Respondent                                                      \n",
       "1                                                         NaN   \n",
       "2           Financial performance or funding status of the...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "1                                                         NaN            NaN   \n",
       "2           Something else changed (education, award, medi...            NaN   \n",
       "\n",
       "           CurrencyDesc  CompTotal CompFreq  ConvertedComp  WorkWeekHrs  \\\n",
       "Respondent                                                                \n",
       "1                   NaN        NaN      NaN            NaN          NaN   \n",
       "2                   NaN        NaN      NaN            NaN          NaN   \n",
       "\n",
       "           WorkPlan WorkChallenge WorkRemote WorkLoc ImpSyn CodeRev  \\\n",
       "Respondent                                                            \n",
       "1               NaN           NaN        NaN     NaN    NaN     NaN   \n",
       "2               NaN           NaN        NaN     NaN    NaN     NaN   \n",
       "\n",
       "            CodeRevHrs UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                  \n",
       "1                  NaN       NaN         NaN          NaN   \n",
       "2                  NaN       NaN         NaN          NaN   \n",
       "\n",
       "                         LanguageWorkedWith  \\\n",
       "Respondent                                    \n",
       "1           HTML/CSS;Java;JavaScript;Python   \n",
       "2                       C++;HTML/CSS;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "2                               C++;HTML/CSS;JavaScript;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "2                         NaN                  MySQL            Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "2                           Windows             Django                 Django   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "2                         NaN                    NaN   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "2                         Atom;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg                                       BlockchainIs  \\\n",
       "Respondent                                                                    \n",
       "1                    NaN                                                NaN   \n",
       "2                    NaN  Useful across many domains and could change ma...   \n",
       "\n",
       "           BetterLife                                  ITperson OffOn  \\\n",
       "Respondent                                                              \n",
       "1                 Yes  Fortunately, someone else has that title   Yes   \n",
       "2                 Yes                                       Yes   Yes   \n",
       "\n",
       "           SocialMedia Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                                  \n",
       "1              Twitter       Online   Username       2017   \n",
       "2            Instagram       Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "2                     Daily or almost daily   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "2           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "2           3-5 times per week  Stack Overflow was much faster  11-30 minutes   \n",
       "\n",
       "           SOAccount                       SOPartFreq  \\\n",
       "Respondent                                              \n",
       "1                 No                              NaN   \n",
       "2                Yes  A few times per month or weekly   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "2           No, I knew that Stack Overflow had a job board...   \n",
       "\n",
       "                                       EntTeams         SOComm  \\\n",
       "Respondent                                                       \n",
       "1           No, and I don't know what those are        Neutral   \n",
       "2           No, and I don't know what those are  Yes, somewhat   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "2           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "2           Tech articles written by other developers;Indu...  19.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "2             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  \n",
       "2           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CSV & TSV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>to write back to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>I code primarily as a hobby</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, back-end;Engineer, site reliability</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...</td>\n",
       "      <td>Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...</td>\n",
       "      <td>Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...</td>\n",
       "      <td>AWS;Docker;Heroku;Linux;MacOS;Slack</td>\n",
       "      <td>Android;Arduino;AWS;Docker;Google Cloud Platfo...</td>\n",
       "      <td>Express;Flask;React.js;Spring</td>\n",
       "      <td>Django;Express;Flask;React.js;Vue.js</td>\n",
       "      <td>Hadoop;Node.js;Pandas</td>\n",
       "      <td>Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...</td>\n",
       "      <td>Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development;Testing;Production;Outside of work...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>SIGH</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Handle</td>\n",
       "      <td>2012</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>Less than once per week</td>\n",
       "      <td>Stack Overflow was slightly faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, definitely</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>India</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10,000 or more employees</td>\n",
       "      <td>Data or business analyst;Data scientist or mac...</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>INR</td>\n",
       "      <td>Indian rupee</td>\n",
       "      <td>950000.0</td>\n",
       "      <td>Yearly</td>\n",
       "      <td>13293.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A few days each month</td>\n",
       "      <td>Home</td>\n",
       "      <td>Far above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>C#;Go;JavaScript;Kotlin;Python;R;SQL</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server;MyS...</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Linux;Raspberry Pi;Windows</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>.NET;Node.js;Pandas;Torch/PyTorch</td>\n",
       "      <td>.NET;Node.js;TensorFlow;Torch/PyTorch</td>\n",
       "      <td>Android Studio;Eclipse;IPython / Jupyter;Notep...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not at all</td>\n",
       "      <td>Useful for immutable record keeping outside of...</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>Neither</td>\n",
       "      <td>Screen Name</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Multiple times per day</td>\n",
       "      <td>Find answers to specific questions;Get a sense...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Student</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>I've never had a job</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Industry that I'd be working in;Languages, fra...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...</td>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;C#;Go;HTM...</td>\n",
       "      <td>MariaDB;MySQL;Oracle;SQLite</td>\n",
       "      <td>MariaDB;MongoDB;Microsoft SQL Server;MySQL;Ora...</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Google Cloud Platform;iOS;Linux;MacOS;...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Drupal;jQuer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>.NET;.NET Core;Node.js;TensorFlow;Unity 3D;Unr...</td>\n",
       "      <td>Atom;NetBeans;Notepad++;Sublime Text;Vim</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>What?</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>More than 10 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, I've heard of them, but I am not part of a...</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist  \\\n",
       "Respondent                                                    \n",
       "8                      I code primarily as a hobby      Yes   \n",
       "10                  I am a developer by profession      Yes   \n",
       "15          I am a student who is learning to code      Yes   \n",
       "\n",
       "                           OpenSourcer  \\\n",
       "Respondent                               \n",
       "8              Less than once per year   \n",
       "10          Once a month or more often   \n",
       "15                               Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "8           OSS is, on average, of HIGHER quality than pro...   \n",
       "10          OSS is, on average, of HIGHER quality than pro...   \n",
       "15          OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                                    Employment Country         Student  \\\n",
       "Respondent                                                               \n",
       "8           Not employed, but looking for work   India             NaN   \n",
       "10                          Employed full-time   India              No   \n",
       "15          Not employed, but looking for work   India  Yes, full-time   \n",
       "\n",
       "                                                      EdLevel  \\\n",
       "Respondent                                                      \n",
       "8                    Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "10                Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "15          Secondary school (e.g. American high school, G...   \n",
       "\n",
       "                                               UndergradMajor  \\\n",
       "Respondent                                                      \n",
       "8           Computer science, computer engineering, or sof...   \n",
       "10                                                        NaN   \n",
       "15                                                        NaN   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "8           Taught yourself a new language, framework, or ...   \n",
       "10                                                        NaN   \n",
       "15          Taken an online course in programming or softw...   \n",
       "\n",
       "                             OrgSize  \\\n",
       "Respondent                             \n",
       "8                                NaN   \n",
       "10          10,000 or more employees   \n",
       "15                               NaN   \n",
       "\n",
       "                                                      DevType YearsCode  \\\n",
       "Respondent                                                                \n",
       "8              Developer, back-end;Engineer, site reliability         8   \n",
       "10          Data or business analyst;Data scientist or mac...        12   \n",
       "15                                                    Student         3   \n",
       "\n",
       "           Age1stCode YearsCodePro              CareerSat  \\\n",
       "Respondent                                                  \n",
       "8                  16          NaN                    NaN   \n",
       "10                 20           10  Slightly dissatisfied   \n",
       "15                 13          NaN                    NaN   \n",
       "\n",
       "                           JobSat            MgrIdiot MgrMoney MgrWant  \\\n",
       "Respondent                                                               \n",
       "8                             NaN                 NaN      NaN     NaN   \n",
       "10          Slightly dissatisfied  Somewhat confident      Yes     Yes   \n",
       "15                            NaN                 NaN      NaN     NaN   \n",
       "\n",
       "                                                      JobSeek  \\\n",
       "Respondent                                                      \n",
       "8                                                         NaN   \n",
       "10          I’m not actively looking, but I am open to new...   \n",
       "15          I’m not actively looking, but I am open to new...   \n",
       "\n",
       "                    LastHireDate LastInt FizzBuzz  \\\n",
       "Respondent                                          \n",
       "8                            NaN     NaN      NaN   \n",
       "10                 3-4 years ago     NaN       No   \n",
       "15          I've never had a job     NaN      NaN   \n",
       "\n",
       "                                                   JobFactors  \\\n",
       "Respondent                                                      \n",
       "8                                                         NaN   \n",
       "10          Languages, frameworks, and other technologies ...   \n",
       "15          Industry that I'd be working in;Languages, fra...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "8                                                         NaN            NaN   \n",
       "10                                                        NaN            INR   \n",
       "15          Something else changed (education, award, medi...            NaN   \n",
       "\n",
       "            CurrencyDesc  CompTotal CompFreq  ConvertedComp  WorkWeekHrs  \\\n",
       "Respondent                                                                 \n",
       "8                    NaN        NaN      NaN            NaN          NaN   \n",
       "10          Indian rupee   950000.0   Yearly        13293.0         70.0   \n",
       "15                   NaN        NaN      NaN            NaN          NaN   \n",
       "\n",
       "                                                     WorkPlan WorkChallenge  \\\n",
       "Respondent                                                                    \n",
       "8                                                         NaN           NaN   \n",
       "10          There's no schedule or spec; I work on what se...           NaN   \n",
       "15                                                        NaN           NaN   \n",
       "\n",
       "                       WorkRemote WorkLoc             ImpSyn  \\\n",
       "Respondent                                                     \n",
       "8                             NaN     NaN                NaN   \n",
       "10          A few days each month    Home  Far above average   \n",
       "15                            NaN     NaN                NaN   \n",
       "\n",
       "                                            CodeRev  CodeRevHrs  \\\n",
       "Respondent                                                        \n",
       "8                                               NaN         NaN   \n",
       "10          Yes, because I see value in code review         4.0   \n",
       "15                                              NaN         NaN   \n",
       "\n",
       "                                UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                           \n",
       "8                                     NaN         NaN          NaN   \n",
       "10          Yes, it's part of our process         NaN          NaN   \n",
       "15                                    NaN         NaN          NaN   \n",
       "\n",
       "                                           LanguageWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...   \n",
       "10                              C#;Go;JavaScript;Python;R;SQL   \n",
       "15          Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...   \n",
       "\n",
       "                                       LanguageDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...   \n",
       "10                       C#;Go;JavaScript;Kotlin;Python;R;SQL   \n",
       "15          Assembly;Bash/Shell/PowerShell;C;C++;C#;Go;HTM...   \n",
       "\n",
       "                                           DatabaseWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...   \n",
       "10          Elasticsearch;MongoDB;Microsoft SQL Server;MyS...   \n",
       "15                                MariaDB;MySQL;Oracle;SQLite   \n",
       "\n",
       "                                       DatabaseDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...   \n",
       "10                 Elasticsearch;MongoDB;Microsoft SQL Server   \n",
       "15          MariaDB;MongoDB;Microsoft SQL Server;MySQL;Ora...   \n",
       "\n",
       "                             PlatformWorkedWith  \\\n",
       "Respondent                                        \n",
       "8           AWS;Docker;Heroku;Linux;MacOS;Slack   \n",
       "10                                Linux;Windows   \n",
       "15                                Linux;Windows   \n",
       "\n",
       "                                       PlatformDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Android;Arduino;AWS;Docker;Google Cloud Platfo...   \n",
       "10                         Android;Linux;Raspberry Pi;Windows   \n",
       "15          Android;Google Cloud Platform;iOS;Linux;MacOS;...   \n",
       "\n",
       "                                           WebFrameWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8                               Express;Flask;React.js;Spring   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "15                                                        NaN   \n",
       "\n",
       "                                       WebFrameDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8                        Django;Express;Flask;React.js;Vue.js   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "15          Angular/Angular.js;ASP.NET;Django;Drupal;jQuer...   \n",
       "\n",
       "                           MiscTechWorkedWith  \\\n",
       "Respondent                                      \n",
       "8                       Hadoop;Node.js;Pandas   \n",
       "10          .NET;Node.js;Pandas;Torch/PyTorch   \n",
       "15                                        NaN   \n",
       "\n",
       "                                       MiscTechDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...   \n",
       "10                      .NET;Node.js;TensorFlow;Torch/PyTorch   \n",
       "15          .NET;.NET Core;Node.js;TensorFlow;Unity 3D;Unr...   \n",
       "\n",
       "                                                   DevEnviron        OpSys  \\\n",
       "Respondent                                                                   \n",
       "8           Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...  Linux-based   \n",
       "10          Android Studio;Eclipse;IPython / Jupyter;Notep...      Windows   \n",
       "15                   Atom;NetBeans;Notepad++;Sublime Text;Vim  Linux-based   \n",
       "\n",
       "                                                   Containers BlockchainOrg  \\\n",
       "Respondent                                                                    \n",
       "8           Development;Testing;Production;Outside of work...           NaN   \n",
       "10                                                        NaN    Not at all   \n",
       "15                                                Development           NaN   \n",
       "\n",
       "                                                 BlockchainIs BetterLife  \\\n",
       "Respondent                                                                 \n",
       "8           Useful across many domains and could change ma...        Yes   \n",
       "10          Useful for immutable record keeping outside of...         No   \n",
       "15                                                        NaN        Yes   \n",
       "\n",
       "           ITperson  OffOn SocialMedia              Extraversion   ScreenName  \\\n",
       "Respondent                                                                      \n",
       "8              SIGH    Yes     YouTube  In real life (in person)       Handle   \n",
       "10              Yes    Yes     YouTube                   Neither  Screen Name   \n",
       "15              Yes  What?     YouTube  In real life (in person)          NaN   \n",
       "\n",
       "           SOVisit1st             SOVisitFreq  \\\n",
       "Respondent                                      \n",
       "8                2012    A few times per week   \n",
       "10                NaN  Multiple times per day   \n",
       "15               2018   Daily or almost daily   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "8           Find answers to specific questions;Learn how t...   \n",
       "10          Find answers to specific questions;Get a sense...   \n",
       "15          Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                           SOFindAnswer                         SOTimeSaved  \\\n",
       "Respondent                                                                    \n",
       "8               Less than once per week  Stack Overflow was slightly faster   \n",
       "10                   3-5 times per week            They were about the same   \n",
       "15          More than 10 times per week            They were about the same   \n",
       "\n",
       "            SOHowMuchTime SOAccount                           SOPartFreq  \\\n",
       "Respondent                                                                 \n",
       "8           11-30 minutes       Yes  Less than once per month or monthly   \n",
       "10                    NaN       Yes      A few times per month or weekly   \n",
       "15                    NaN       Yes  Less than once per month or monthly   \n",
       "\n",
       "           SOJobs                                           EntTeams  \\\n",
       "Respondent                                                             \n",
       "8             Yes                No, and I don't know what those are   \n",
       "10            Yes                No, and I don't know what those are   \n",
       "15            Yes  No, I've heard of them, but I am not part of a...   \n",
       "\n",
       "                     SOComm                             WelcomeChange  \\\n",
       "Respondent                                                              \n",
       "8           Yes, definitely     A lot more welcome now than last year   \n",
       "10            Yes, somewhat  Somewhat less welcome now than last year   \n",
       "15            Yes, somewhat   Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "8           Tech articles written by other developers;Indu...  24.0    Man   \n",
       "10          Tech articles written by other developers;Tech...   NaN    NaN   \n",
       "15          Tech articles written by other developers;Indu...  20.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "8             No  Straight / Heterosexual       NaN        NaN   \n",
       "10           NaN                      NaN       NaN        Yes   \n",
       "15            No                      NaN       NaN        Yes   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "8           Appropriate in length  Neither easy nor difficult  \n",
       "10                       Too long                   Difficult  \n",
       "15                       Too long  Neither easy nor difficult  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt = df['Country'] == 'India'\n",
    "india_df = df[filt]\n",
    "india_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we use `.to_csv()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "india_df.to_csv('data/india_df.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark> Tab delimited file\n",
    "\n",
    "here we need a give an custom separator but use the same `.to_csv()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "india_df.to_csv('data/india_df.tsv', sep='\\t') # \\t means Tab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>reading a tsv file</mark> we just need to use say the separator while reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent', sep='\\t' )`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EXCEL\n",
    "we need the following packages\n",
    "`pip install xlwt openpyxl xlrd`\n",
    "- xlwt - for old .xls files\n",
    "- openpyxl - for newer .xlsx\n",
    "- xlrd - for reading Excel files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark> Write Excel files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "india_df.to_excel('data/india_df.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to read or write to specific sheets we need to include sheet argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark> Read excel file</mark>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = pd.read_excel('data/india_df.xlsx', index_col='Respondent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>I code primarily as a hobby</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Developer, back-end;Engineer, site reliability</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...</td>\n",
       "      <td>Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...</td>\n",
       "      <td>Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...</td>\n",
       "      <td>AWS;Docker;Heroku;Linux;MacOS;Slack</td>\n",
       "      <td>Android;Arduino;AWS;Docker;Google Cloud Platfo...</td>\n",
       "      <td>Express;Flask;React.js;Spring</td>\n",
       "      <td>Django;Express;Flask;React.js;Vue.js</td>\n",
       "      <td>Hadoop;Node.js;Pandas</td>\n",
       "      <td>Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...</td>\n",
       "      <td>Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development;Testing;Production;Outside of work...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>SIGH</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Handle</td>\n",
       "      <td>2012</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>Less than once per week</td>\n",
       "      <td>Stack Overflow was slightly faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, definitely</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>India</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10,000 or more employees</td>\n",
       "      <td>Data or business analyst;Data scientist or mac...</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>INR</td>\n",
       "      <td>Indian rupee</td>\n",
       "      <td>950000.0</td>\n",
       "      <td>Yearly</td>\n",
       "      <td>13293.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A few days each month</td>\n",
       "      <td>Home</td>\n",
       "      <td>Far above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>C#;Go;JavaScript;Kotlin;Python;R;SQL</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server;MyS...</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Linux;Raspberry Pi;Windows</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>.NET;Node.js;Pandas;Torch/PyTorch</td>\n",
       "      <td>.NET;Node.js;TensorFlow;Torch/PyTorch</td>\n",
       "      <td>Android Studio;Eclipse;IPython / Jupyter;Notep...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not at all</td>\n",
       "      <td>Useful for immutable record keeping outside of...</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>Neither</td>\n",
       "      <td>Screen Name</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Multiple times per day</td>\n",
       "      <td>Find answers to specific questions;Get a sense...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>Yes, full-time</td>\n",
       "      <td>Secondary school (e.g. American high school, G...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taken an online course in programming or softw...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Student</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>I've never had a job</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Industry that I'd be working in;Languages, fra...</td>\n",
       "      <td>Something else changed (education, award, medi...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...</td>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;C#;Go;HTM...</td>\n",
       "      <td>MariaDB;MySQL;Oracle;SQLite</td>\n",
       "      <td>MariaDB;MongoDB;Microsoft SQL Server;MySQL;Ora...</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Google Cloud Platform;iOS;Linux;MacOS;...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Drupal;jQuer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>.NET;.NET Core;Node.js;TensorFlow;Unity 3D;Unr...</td>\n",
       "      <td>Atom;NetBeans;Notepad++;Sublime Text;Vim</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>What?</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018</td>\n",
       "      <td>Daily or almost daily</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>More than 10 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, I've heard of them, but I am not part of a...</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>20.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist  \\\n",
       "Respondent                                                    \n",
       "8                      I code primarily as a hobby      Yes   \n",
       "10                  I am a developer by profession      Yes   \n",
       "15          I am a student who is learning to code      Yes   \n",
       "\n",
       "                           OpenSourcer  \\\n",
       "Respondent                               \n",
       "8              Less than once per year   \n",
       "10          Once a month or more often   \n",
       "15                               Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "8           OSS is, on average, of HIGHER quality than pro...   \n",
       "10          OSS is, on average, of HIGHER quality than pro...   \n",
       "15          OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                                    Employment Country         Student  \\\n",
       "Respondent                                                               \n",
       "8           Not employed, but looking for work   India             NaN   \n",
       "10                          Employed full-time   India              No   \n",
       "15          Not employed, but looking for work   India  Yes, full-time   \n",
       "\n",
       "                                                      EdLevel  \\\n",
       "Respondent                                                      \n",
       "8                    Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "10                Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "15          Secondary school (e.g. American high school, G...   \n",
       "\n",
       "                                               UndergradMajor  \\\n",
       "Respondent                                                      \n",
       "8           Computer science, computer engineering, or sof...   \n",
       "10                                                        NaN   \n",
       "15                                                        NaN   \n",
       "\n",
       "                                                     EduOther  \\\n",
       "Respondent                                                      \n",
       "8           Taught yourself a new language, framework, or ...   \n",
       "10                                                        NaN   \n",
       "15          Taken an online course in programming or softw...   \n",
       "\n",
       "                             OrgSize  \\\n",
       "Respondent                             \n",
       "8                                NaN   \n",
       "10          10,000 or more employees   \n",
       "15                               NaN   \n",
       "\n",
       "                                                      DevType YearsCode  \\\n",
       "Respondent                                                                \n",
       "8              Developer, back-end;Engineer, site reliability         8   \n",
       "10          Data or business analyst;Data scientist or mac...        12   \n",
       "15                                                    Student         3   \n",
       "\n",
       "           Age1stCode YearsCodePro              CareerSat  \\\n",
       "Respondent                                                  \n",
       "8                  16          NaN                    NaN   \n",
       "10                 20           10  Slightly dissatisfied   \n",
       "15                 13          NaN                    NaN   \n",
       "\n",
       "                           JobSat            MgrIdiot MgrMoney MgrWant  \\\n",
       "Respondent                                                               \n",
       "8                             NaN                 NaN      NaN     NaN   \n",
       "10          Slightly dissatisfied  Somewhat confident      Yes     Yes   \n",
       "15                            NaN                 NaN      NaN     NaN   \n",
       "\n",
       "                                                      JobSeek  \\\n",
       "Respondent                                                      \n",
       "8                                                         NaN   \n",
       "10          I’m not actively looking, but I am open to new...   \n",
       "15          I’m not actively looking, but I am open to new...   \n",
       "\n",
       "                    LastHireDate LastInt FizzBuzz  \\\n",
       "Respondent                                          \n",
       "8                            NaN     NaN      NaN   \n",
       "10                 3-4 years ago     NaN       No   \n",
       "15          I've never had a job     NaN      NaN   \n",
       "\n",
       "                                                   JobFactors  \\\n",
       "Respondent                                                      \n",
       "8                                                         NaN   \n",
       "10          Languages, frameworks, and other technologies ...   \n",
       "15          Industry that I'd be working in;Languages, fra...   \n",
       "\n",
       "                                                 ResumeUpdate CurrencySymbol  \\\n",
       "Respondent                                                                     \n",
       "8                                                         NaN            NaN   \n",
       "10                                                        NaN            INR   \n",
       "15          Something else changed (education, award, medi...            NaN   \n",
       "\n",
       "            CurrencyDesc  CompTotal CompFreq  ConvertedComp  WorkWeekHrs  \\\n",
       "Respondent                                                                 \n",
       "8                    NaN        NaN      NaN            NaN          NaN   \n",
       "10          Indian rupee   950000.0   Yearly        13293.0         70.0   \n",
       "15                   NaN        NaN      NaN            NaN          NaN   \n",
       "\n",
       "                                                     WorkPlan WorkChallenge  \\\n",
       "Respondent                                                                    \n",
       "8                                                         NaN           NaN   \n",
       "10          There's no schedule or spec; I work on what se...           NaN   \n",
       "15                                                        NaN           NaN   \n",
       "\n",
       "                       WorkRemote WorkLoc             ImpSyn  \\\n",
       "Respondent                                                     \n",
       "8                             NaN     NaN                NaN   \n",
       "10          A few days each month    Home  Far above average   \n",
       "15                            NaN     NaN                NaN   \n",
       "\n",
       "                                            CodeRev  CodeRevHrs  \\\n",
       "Respondent                                                        \n",
       "8                                               NaN         NaN   \n",
       "10          Yes, because I see value in code review         4.0   \n",
       "15                                              NaN         NaN   \n",
       "\n",
       "                                UnitTests PurchaseHow PurchaseWhat  \\\n",
       "Respondent                                                           \n",
       "8                                     NaN         NaN          NaN   \n",
       "10          Yes, it's part of our process         NaN          NaN   \n",
       "15                                    NaN         NaN          NaN   \n",
       "\n",
       "                                           LanguageWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...   \n",
       "10                              C#;Go;JavaScript;Python;R;SQL   \n",
       "15          Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...   \n",
       "\n",
       "                                       LanguageDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...   \n",
       "10                       C#;Go;JavaScript;Kotlin;Python;R;SQL   \n",
       "15          Assembly;Bash/Shell/PowerShell;C;C++;C#;Go;HTM...   \n",
       "\n",
       "                                           DatabaseWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...   \n",
       "10          Elasticsearch;MongoDB;Microsoft SQL Server;MyS...   \n",
       "15                                MariaDB;MySQL;Oracle;SQLite   \n",
       "\n",
       "                                       DatabaseDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...   \n",
       "10                 Elasticsearch;MongoDB;Microsoft SQL Server   \n",
       "15          MariaDB;MongoDB;Microsoft SQL Server;MySQL;Ora...   \n",
       "\n",
       "                             PlatformWorkedWith  \\\n",
       "Respondent                                        \n",
       "8           AWS;Docker;Heroku;Linux;MacOS;Slack   \n",
       "10                                Linux;Windows   \n",
       "15                                Linux;Windows   \n",
       "\n",
       "                                       PlatformDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Android;Arduino;AWS;Docker;Google Cloud Platfo...   \n",
       "10                         Android;Linux;Raspberry Pi;Windows   \n",
       "15          Android;Google Cloud Platform;iOS;Linux;MacOS;...   \n",
       "\n",
       "                                           WebFrameWorkedWith  \\\n",
       "Respondent                                                      \n",
       "8                               Express;Flask;React.js;Spring   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "15                                                        NaN   \n",
       "\n",
       "                                       WebFrameDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8                        Django;Express;Flask;React.js;Vue.js   \n",
       "10          Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "15          Angular/Angular.js;ASP.NET;Django;Drupal;jQuer...   \n",
       "\n",
       "                           MiscTechWorkedWith  \\\n",
       "Respondent                                      \n",
       "8                       Hadoop;Node.js;Pandas   \n",
       "10          .NET;Node.js;Pandas;Torch/PyTorch   \n",
       "15                                        NaN   \n",
       "\n",
       "                                       MiscTechDesireNextYear  \\\n",
       "Respondent                                                      \n",
       "8           Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...   \n",
       "10                      .NET;Node.js;TensorFlow;Torch/PyTorch   \n",
       "15          .NET;.NET Core;Node.js;TensorFlow;Unity 3D;Unr...   \n",
       "\n",
       "                                                   DevEnviron        OpSys  \\\n",
       "Respondent                                                                   \n",
       "8           Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...  Linux-based   \n",
       "10          Android Studio;Eclipse;IPython / Jupyter;Notep...      Windows   \n",
       "15                   Atom;NetBeans;Notepad++;Sublime Text;Vim  Linux-based   \n",
       "\n",
       "                                                   Containers BlockchainOrg  \\\n",
       "Respondent                                                                    \n",
       "8           Development;Testing;Production;Outside of work...           NaN   \n",
       "10                                                        NaN    Not at all   \n",
       "15                                                Development           NaN   \n",
       "\n",
       "                                                 BlockchainIs BetterLife  \\\n",
       "Respondent                                                                 \n",
       "8           Useful across many domains and could change ma...        Yes   \n",
       "10          Useful for immutable record keeping outside of...         No   \n",
       "15                                                        NaN        Yes   \n",
       "\n",
       "           ITperson  OffOn SocialMedia              Extraversion   ScreenName  \\\n",
       "Respondent                                                                      \n",
       "8              SIGH    Yes     YouTube  In real life (in person)       Handle   \n",
       "10              Yes    Yes     YouTube                   Neither  Screen Name   \n",
       "15              Yes  What?     YouTube  In real life (in person)          NaN   \n",
       "\n",
       "           SOVisit1st             SOVisitFreq  \\\n",
       "Respondent                                      \n",
       "8                2012    A few times per week   \n",
       "10                NaN  Multiple times per day   \n",
       "15               2018   Daily or almost daily   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "8           Find answers to specific questions;Learn how t...   \n",
       "10          Find answers to specific questions;Get a sense...   \n",
       "15          Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                           SOFindAnswer                         SOTimeSaved  \\\n",
       "Respondent                                                                    \n",
       "8               Less than once per week  Stack Overflow was slightly faster   \n",
       "10                   3-5 times per week            They were about the same   \n",
       "15          More than 10 times per week            They were about the same   \n",
       "\n",
       "            SOHowMuchTime SOAccount                           SOPartFreq  \\\n",
       "Respondent                                                                 \n",
       "8           11-30 minutes       Yes  Less than once per month or monthly   \n",
       "10                    NaN       Yes      A few times per month or weekly   \n",
       "15                    NaN       Yes  Less than once per month or monthly   \n",
       "\n",
       "           SOJobs                                           EntTeams  \\\n",
       "Respondent                                                             \n",
       "8             Yes                No, and I don't know what those are   \n",
       "10            Yes                No, and I don't know what those are   \n",
       "15            Yes  No, I've heard of them, but I am not part of a...   \n",
       "\n",
       "                     SOComm                             WelcomeChange  \\\n",
       "Respondent                                                              \n",
       "8           Yes, definitely     A lot more welcome now than last year   \n",
       "10            Yes, somewhat  Somewhat less welcome now than last year   \n",
       "15            Yes, somewhat   Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "8           Tech articles written by other developers;Indu...  24.0    Man   \n",
       "10          Tech articles written by other developers;Tech...   NaN    NaN   \n",
       "15          Tech articles written by other developers;Indu...  20.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "8             No  Straight / Heterosexual       NaN        NaN   \n",
       "10           NaN                      NaN       NaN        Yes   \n",
       "15            No                      NaN       NaN        Yes   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "8           Appropriate in length  Neither easy nor difficult  \n",
       "10                       Too long                   Difficult  \n",
       "15                       Too long  Neither easy nor difficult  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>write to JSON</mark>\n",
    "\n",
    "we need to use`.to_json()` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "india_df.to_json('data/india_df.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "by default it saves as 'dictionary like' JSON where all the responses are within a 'MainBranch'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to save it as 'list-like'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "india_df.to_json('data/india_df_list.json', orient='records', lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`lines=True` make each of the records in new line.  \n",
    "also the json is now saved like list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>Read JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we need to use the same arguments (setting) we used to write them when reading them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "json = pd.read_json('data/india_df_list.json', orient='records', lines=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I code primarily as a hobby</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per year</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Not employed, but looking for work</td>\n",
       "      <td>India</td>\n",
       "      <td>None</td>\n",
       "      <td>Bachelor’s degree (BA, BS, B.Eng., etc.)</td>\n",
       "      <td>Computer science, computer engineering, or sof...</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>None</td>\n",
       "      <td>Developer, back-end;Engineer, site reliability</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...</td>\n",
       "      <td>Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...</td>\n",
       "      <td>Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...</td>\n",
       "      <td>AWS;Docker;Heroku;Linux;MacOS;Slack</td>\n",
       "      <td>Android;Arduino;AWS;Docker;Google Cloud Platfo...</td>\n",
       "      <td>Express;Flask;React.js;Spring</td>\n",
       "      <td>Django;Express;Flask;React.js;Vue.js</td>\n",
       "      <td>Hadoop;Node.js;Pandas</td>\n",
       "      <td>Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...</td>\n",
       "      <td>Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...</td>\n",
       "      <td>Linux-based</td>\n",
       "      <td>Development;Testing;Production;Outside of work...</td>\n",
       "      <td>None</td>\n",
       "      <td>Useful across many domains and could change ma...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>SIGH</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>In real life (in person)</td>\n",
       "      <td>Handle</td>\n",
       "      <td>2012</td>\n",
       "      <td>A few times per week</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>Less than once per week</td>\n",
       "      <td>Stack Overflow was slightly faster</td>\n",
       "      <td>11-30 minutes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Less than once per month or monthly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, definitely</td>\n",
       "      <td>A lot more welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>24.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a developer by profession</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Once a month or more often</td>\n",
       "      <td>OSS is, on average, of HIGHER quality than pro...</td>\n",
       "      <td>Employed full-time</td>\n",
       "      <td>India</td>\n",
       "      <td>No</td>\n",
       "      <td>Master’s degree (MA, MS, M.Eng., MBA, etc.)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>10,000 or more employees</td>\n",
       "      <td>Data or business analyst;Data scientist or mac...</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Slightly dissatisfied</td>\n",
       "      <td>Somewhat confident</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>I’m not actively looking, but I am open to new...</td>\n",
       "      <td>3-4 years ago</td>\n",
       "      <td>None</td>\n",
       "      <td>No</td>\n",
       "      <td>Languages, frameworks, and other technologies ...</td>\n",
       "      <td>None</td>\n",
       "      <td>INR</td>\n",
       "      <td>Indian rupee</td>\n",
       "      <td>950000.0</td>\n",
       "      <td>Yearly</td>\n",
       "      <td>13293.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>There's no schedule or spec; I work on what se...</td>\n",
       "      <td>None</td>\n",
       "      <td>A few days each month</td>\n",
       "      <td>Home</td>\n",
       "      <td>Far above average</td>\n",
       "      <td>Yes, because I see value in code review</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Yes, it's part of our process</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>C#;Go;JavaScript;Kotlin;Python;R;SQL</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server;MyS...</td>\n",
       "      <td>Elasticsearch;MongoDB;Microsoft SQL Server</td>\n",
       "      <td>Linux;Windows</td>\n",
       "      <td>Android;Linux;Raspberry Pi;Windows</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>Angular/Angular.js;ASP.NET;Django;Express;Flas...</td>\n",
       "      <td>.NET;Node.js;Pandas;Torch/PyTorch</td>\n",
       "      <td>.NET;Node.js;TensorFlow;Torch/PyTorch</td>\n",
       "      <td>Android Studio;Eclipse;IPython / Jupyter;Notep...</td>\n",
       "      <td>Windows</td>\n",
       "      <td>None</td>\n",
       "      <td>Not at all</td>\n",
       "      <td>Useful for immutable record keeping outside of...</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>YouTube</td>\n",
       "      <td>Neither</td>\n",
       "      <td>Screen Name</td>\n",
       "      <td>None</td>\n",
       "      <td>Multiple times per day</td>\n",
       "      <td>Find answers to specific questions;Get a sense...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>They were about the same</td>\n",
       "      <td>None</td>\n",
       "      <td>Yes</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Yes, somewhat</td>\n",
       "      <td>Somewhat less welcome now than last year</td>\n",
       "      <td>Tech articles written by other developers;Tech...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Too long</td>\n",
       "      <td>Difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       MainBranch Hobbyist                 OpenSourcer  \\\n",
       "0     I code primarily as a hobby      Yes     Less than once per year   \n",
       "1  I am a developer by profession      Yes  Once a month or more often   \n",
       "\n",
       "                                          OpenSource  \\\n",
       "0  OSS is, on average, of HIGHER quality than pro...   \n",
       "1  OSS is, on average, of HIGHER quality than pro...   \n",
       "\n",
       "                           Employment Country Student  \\\n",
       "0  Not employed, but looking for work   India    None   \n",
       "1                  Employed full-time   India      No   \n",
       "\n",
       "                                       EdLevel  \\\n",
       "0     Bachelor’s degree (BA, BS, B.Eng., etc.)   \n",
       "1  Master’s degree (MA, MS, M.Eng., MBA, etc.)   \n",
       "\n",
       "                                      UndergradMajor  \\\n",
       "0  Computer science, computer engineering, or sof...   \n",
       "1                                               None   \n",
       "\n",
       "                                            EduOther  \\\n",
       "0  Taught yourself a new language, framework, or ...   \n",
       "1                                               None   \n",
       "\n",
       "                    OrgSize  \\\n",
       "0                      None   \n",
       "1  10,000 or more employees   \n",
       "\n",
       "                                             DevType YearsCode Age1stCode  \\\n",
       "0     Developer, back-end;Engineer, site reliability         8         16   \n",
       "1  Data or business analyst;Data scientist or mac...        12         20   \n",
       "\n",
       "  YearsCodePro              CareerSat                 JobSat  \\\n",
       "0         None                   None                   None   \n",
       "1           10  Slightly dissatisfied  Slightly dissatisfied   \n",
       "\n",
       "             MgrIdiot MgrMoney MgrWant  \\\n",
       "0                None     None    None   \n",
       "1  Somewhat confident      Yes     Yes   \n",
       "\n",
       "                                             JobSeek   LastHireDate LastInt  \\\n",
       "0                                               None           None    None   \n",
       "1  I’m not actively looking, but I am open to new...  3-4 years ago    None   \n",
       "\n",
       "  FizzBuzz                                         JobFactors ResumeUpdate  \\\n",
       "0     None                                               None         None   \n",
       "1       No  Languages, frameworks, and other technologies ...         None   \n",
       "\n",
       "  CurrencySymbol  CurrencyDesc  CompTotal CompFreq  ConvertedComp  \\\n",
       "0           None          None        NaN     None            NaN   \n",
       "1            INR  Indian rupee   950000.0   Yearly        13293.0   \n",
       "\n",
       "   WorkWeekHrs                                           WorkPlan  \\\n",
       "0          NaN                                               None   \n",
       "1         70.0  There's no schedule or spec; I work on what se...   \n",
       "\n",
       "  WorkChallenge             WorkRemote WorkLoc             ImpSyn  \\\n",
       "0          None                   None    None               None   \n",
       "1          None  A few days each month    Home  Far above average   \n",
       "\n",
       "                                   CodeRev  CodeRevHrs  \\\n",
       "0                                     None         NaN   \n",
       "1  Yes, because I see value in code review         4.0   \n",
       "\n",
       "                       UnitTests PurchaseHow PurchaseWhat  \\\n",
       "0                           None        None         None   \n",
       "1  Yes, it's part of our process        None         None   \n",
       "\n",
       "                                  LanguageWorkedWith  \\\n",
       "0  Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...   \n",
       "1                      C#;Go;JavaScript;Python;R;SQL   \n",
       "\n",
       "                              LanguageDesireNextYear  \\\n",
       "0  Bash/Shell/PowerShell;C;C++;Elixir;Erlang;Go;P...   \n",
       "1               C#;Go;JavaScript;Kotlin;Python;R;SQL   \n",
       "\n",
       "                                  DatabaseWorkedWith  \\\n",
       "0  Cassandra;Elasticsearch;MongoDB;MySQL;Oracle;R...   \n",
       "1  Elasticsearch;MongoDB;Microsoft SQL Server;MyS...   \n",
       "\n",
       "                              DatabaseDesireNextYear  \\\n",
       "0  Cassandra;DynamoDB;Elasticsearch;Firebase;Mong...   \n",
       "1         Elasticsearch;MongoDB;Microsoft SQL Server   \n",
       "\n",
       "                    PlatformWorkedWith  \\\n",
       "0  AWS;Docker;Heroku;Linux;MacOS;Slack   \n",
       "1                        Linux;Windows   \n",
       "\n",
       "                              PlatformDesireNextYear  \\\n",
       "0  Android;Arduino;AWS;Docker;Google Cloud Platfo...   \n",
       "1                 Android;Linux;Raspberry Pi;Windows   \n",
       "\n",
       "                                  WebFrameWorkedWith  \\\n",
       "0                      Express;Flask;React.js;Spring   \n",
       "1  Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "\n",
       "                              WebFrameDesireNextYear  \\\n",
       "0               Django;Express;Flask;React.js;Vue.js   \n",
       "1  Angular/Angular.js;ASP.NET;Django;Express;Flas...   \n",
       "\n",
       "                  MiscTechWorkedWith  \\\n",
       "0              Hadoop;Node.js;Pandas   \n",
       "1  .NET;Node.js;Pandas;Torch/PyTorch   \n",
       "\n",
       "                              MiscTechDesireNextYear  \\\n",
       "0  Ansible;Apache Spark;Chef;Hadoop;Node.js;Panda...   \n",
       "1              .NET;Node.js;TensorFlow;Torch/PyTorch   \n",
       "\n",
       "                                          DevEnviron        OpSys  \\\n",
       "0  Atom;IntelliJ;IPython / Jupyter;PyCharm;Visual...  Linux-based   \n",
       "1  Android Studio;Eclipse;IPython / Jupyter;Notep...      Windows   \n",
       "\n",
       "                                          Containers BlockchainOrg  \\\n",
       "0  Development;Testing;Production;Outside of work...          None   \n",
       "1                                               None    Not at all   \n",
       "\n",
       "                                        BlockchainIs BetterLife ITperson  \\\n",
       "0  Useful across many domains and could change ma...        Yes     SIGH   \n",
       "1  Useful for immutable record keeping outside of...         No      Yes   \n",
       "\n",
       "  OffOn SocialMedia              Extraversion   ScreenName SOVisit1st  \\\n",
       "0   Yes     YouTube  In real life (in person)       Handle       2012   \n",
       "1   Yes     YouTube                   Neither  Screen Name       None   \n",
       "\n",
       "              SOVisitFreq                                          SOVisitTo  \\\n",
       "0    A few times per week  Find answers to specific questions;Learn how t...   \n",
       "1  Multiple times per day  Find answers to specific questions;Get a sense...   \n",
       "\n",
       "              SOFindAnswer                         SOTimeSaved  SOHowMuchTime  \\\n",
       "0  Less than once per week  Stack Overflow was slightly faster  11-30 minutes   \n",
       "1       3-5 times per week            They were about the same           None   \n",
       "\n",
       "  SOAccount                           SOPartFreq SOJobs  \\\n",
       "0       Yes  Less than once per month or monthly    Yes   \n",
       "1       Yes      A few times per month or weekly    Yes   \n",
       "\n",
       "                              EntTeams           SOComm  \\\n",
       "0  No, and I don't know what those are  Yes, definitely   \n",
       "1  No, and I don't know what those are    Yes, somewhat   \n",
       "\n",
       "                              WelcomeChange  \\\n",
       "0     A lot more welcome now than last year   \n",
       "1  Somewhat less welcome now than last year   \n",
       "\n",
       "                                        SONewContent   Age Gender Trans  \\\n",
       "0  Tech articles written by other developers;Indu...  24.0    Man    No   \n",
       "1  Tech articles written by other developers;Tech...   NaN   None  None   \n",
       "\n",
       "                 Sexuality Ethnicity Dependents           SurveyLength  \\\n",
       "0  Straight / Heterosexual      None       None  Appropriate in length   \n",
       "1                     None      None        Yes               Too long   \n",
       "\n",
       "                   SurveyEase  \n",
       "0  Neither easy nor difficult  \n",
       "1                   Difficult  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-box alert-warning\"> ** NEED AN SQL DB FOR THIS SECTION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we need to install sequelalchemy library `sqlalchemy` for this. its popular ORM (object relational mapper) to connect SQL with python \n",
    "\n",
    "`pip install sqlalchemy`\n",
    "\n",
    "for postgres need `pscopg2`\n",
    "\n",
    "`pip install pscopg2-binary `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sqlalchemy import create_engine\n",
    "import psycopg2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = create_engine('postgresql://dbuser:dbpass@localhost:5432/sample_db')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "database name 'sample_table'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>writing to table in database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "india_df.to_sql('sample_table', engine, if_exists='replace')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>reading from table in database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_df = pd.read_sql('sample_table', engine, index_col='Respondent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark> to run a specific SQL query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_df = pd.read_sql_query('SELECT * FROM sample_table', engine, index_col='Respondent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading data from URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "posts_df = pd.read_json('https://raw.githubusercontent.com/CoreyMSchafer/code_snippets/master/Python/Flask_Blog/snippets/posts.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "posts_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}