Data_100_Final_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GRu-1wYkvvlq"
   },
   "source": [
    "# Predicting NBA Performance Given NCAA Statistics of Basketball Players"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O-QuWitKv1Bt"
   },
   "source": [
    "The goal of this project is to understand how the performance of a NBA player can be predicted from his performance during NCAA, background and biological features.\n",
    "\n",
    "This notebook is divided in 3 sections. First, we will understand the data available and transform it such that it is easier to work with. Then we will explore the question of how good a basketball player will be based on NCAA performance. Finally, we will predict which position he will be playing in using the same information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NCuXZCilxoa7"
   },
   "source": [
    "We import all necessary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ou93phnKxrRN"
   },
   "outputs": [],
   "source": [
    "# General packages\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from math import *\n",
    "import re\n",
    "import warnings\n",
    "from statistics import mode\n",
    "from scipy.stats import binom\n",
    "\n",
    "# Sklearn functionalities\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.dummy import DummyRegressor\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2CoRkjucxTyP"
   },
   "source": [
    "## 1. Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SLVqwnTQxYPD"
   },
   "source": [
    "There are 5 available dataset with basketball data. We explore each of them separately and determine if they are useful for our analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cLp2c226xjCP"
   },
   "source": [
    "### 2012-18_officialBoxScore\n",
    "\n",
    "This dataset has one row for each officer and game for games between seasons 2012-13 and 2017-18. At each row, in addition to the information from the office, there is information about the genral statistics from the two teams that were playing, as well as information about the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 320
    },
    "colab_type": "code",
    "id": "rXCfQDssvzJ9",
    "outputId": "e05e9f0b-1237-490d-bcab-e9c2590429fa"
   },
   "outputs": [],
   "source": [
    "official_box_score = pd.read_csv('2012-18_officialBoxScore.csv')\n",
    "official_box_score.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UXEJSqSfzNot"
   },
   "source": [
    "As we are interested in the performance of individual players and not team performance, this dataset is of little interest. Therefore, we will not clean it, even if some data could be encoded in a better way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ifeQ6J80YNRB"
   },
   "source": [
    "### 2012-18_teamBoxScore\n",
    "\n",
    "This dataframe has two rows for each game that has been played. All of them contain information about the game and performance of both teams. On one of the rows one team is considered thefeatured team and the other one the oponent. The information of this table is the same as the one described before, but organized using a different granularity. Using the same reasons as before, this dataset will not be explored in detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 320
    },
    "colab_type": "code",
    "id": "vr4Z-8L3ZC4F",
    "outputId": "90504e25-71c0-4d1b-b4e4-d77f4698b0e3"
   },
   "outputs": [],
   "source": [
    "team_box_score = pd.read_csv('2012-18_teamBoxScore.csv')\n",
    "team_box_score.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s7PbU-JKZJhY"
   },
   "source": [
    "### 2012-18_standings\n",
    "\n",
    "This dataset contains information regarding the general standing of teams compared to the rest. As it might be useful to determine the importance of each team, it will be lightly cleaned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 292
    },
    "colab_type": "code",
    "id": "bxUnS9y7Z2UG",
    "outputId": "7582b14a-30a2-47fd-c9c1-23fadf468fd5"
   },
   "outputs": [],
   "source": [
    "standings = pd.read_csv('2012-18_standings.csv')\n",
    "standings.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nBRJIXkLZ560"
   },
   "source": [
    "This dataset contains redundant information and messy columns. In order to solve this issues, we will\n",
    "\n",
    "1. Columns 'stk', 'stkType' and 'sktTot' refer to the strike of wins or losses that a team has. In order to sumarize this information, a single number will describe the strike. It will be positive if it is a winning strike and negative otherwise.\n",
    "\n",
    "2. Column 'rankOrd' is redundant and should be dropped.\n",
    "\n",
    "3. Column 'stDate' contains more than one piece of information per cell. That information should be splitted in multiple cells.\n",
    "\n",
    "To address the latter, we define the following function and create three more columns with the day, month and year of the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 292
    },
    "colab_type": "code",
    "id": "Y0QvIinIaoGt",
    "outputId": "6a331e12-0826-4cec-c49e-b24105d703a0"
   },
   "outputs": [],
   "source": [
    "# Cleaning function\n",
    "def cleanDate(df, column, day_name, month_name, year_name):\n",
    "  '''\n",
    "  Args:\n",
    "    df (dataframe): A data frame with a column that contains year, month and day\n",
    "    column (character): name of the column that contains that information\n",
    "    day_name (character): name of the column that will contain day information\n",
    "    day_month (character): name of the column that will contain month information\n",
    "    day_year (character): name of the column that will contain year information\n",
    "\n",
    "  Returns:\n",
    "    None\n",
    "  '''\n",
    "  data = pd.to_datetime(df[column])\n",
    "  df[day_name] = data.dt.day \n",
    "  df[month_name] = data.dt.month\n",
    "  df[year_name] = data.dt.year\n",
    "\n",
    "# Cleaning the dataset\n",
    "standings['stk'] = standings['stkTot'] * [-1 if x == 'loss' else 1 for x in standings['stkType']]\n",
    "standings = standings.drop(columns = ['stkType', 'stkTot', 'rankOrd'])\n",
    "cleanDate(standings, 'stDate', 'stDay', 'stMonth', 'stYear')\n",
    "standings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BTEY-r8caswW"
   },
   "source": [
    "We finally check that there are no missin values in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 33
    },
    "colab_type": "code",
    "id": "o0VICGhUaxpV",
    "outputId": "c9cf755b-8b98-474c-a7ea-eddb95a30161"
   },
   "outputs": [],
   "source": [
    "standings.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ATKZRtcRS84A"
   },
   "source": [
    "### 2012-18_playerBoxScore\n",
    "\n",
    "This dataset contains the performance of each player at each NBA game for games between seasons 2012-13 and 2017-18, as well as some information regarding the game. This information will be helpful in assesing the performance of different players for their NBA games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 292
    },
    "colab_type": "code",
    "id": "qU5vQDmnzLvr",
    "outputId": "73d47624-c584-4901-aa08-1a3b3c30f4cf"
   },
   "outputs": [],
   "source": [
    "player_box_score = pd.read_csv('2012-18_playerBoxScore.csv')\n",
    "player_box_score.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "scFY187FTqhu"
   },
   "source": [
    "Data seems to be relatively clean. We check that each column of the data set containd the data type that should be expected based on the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 862
    },
    "colab_type": "code",
    "id": "OXN0i5rPTqAG",
    "outputId": "909d8d54-471b-492d-ed77-92acfa19c84b"
   },
   "outputs": [],
   "source": [
    "player_box_score.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8WZEcXktUjdW"
   },
   "source": [
    "There are some categorical variables that should be better encoded in the dataset. However, as they will not be used in further analysis, there is no need to encode them at this point. Also, sometimes it is easier to keep them this way, as they can make visualizations easier. \n",
    "\n",
    "One column that we are going to clean is 'gmDate', as it contains more than one piece of information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 292
    },
    "colab_type": "code",
    "id": "t-WwmA2PUGT9",
    "outputId": "14be3c44-74be-439f-a503-0d7c289d2501"
   },
   "outputs": [],
   "source": [
    "# Create new columns for day, month and year\n",
    "cleanDate(player_box_score, 'gmDate', 'gmDay', 'gmMonth', 'gmYear')\n",
    "player_box_score.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TDP5dVuhWbq2"
   },
   "source": [
    "Finally, we check for possible missing values in the columns and we determine the reason behind them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 911
    },
    "colab_type": "code",
    "id": "O4Fl_MLKWYku",
    "outputId": "a5f15b43-ae5f-4f53-f62c-9f58c4950868"
   },
   "outputs": [],
   "source": [
    "player_box_score.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1SHmDBDDW0sW"
   },
   "source": [
    "We observe that there are 41 missing values on both columns 'offFNm3' and 'offLNm3'. This columns correspond to the first and last name of the third officer that was present in the game. We check those games to see if they share some peculiarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 308
    },
    "colab_type": "code",
    "id": "ccLD_nQ5Wz70",
    "outputId": "4e40a6b7-a0ba-4d2a-ab38-6e319e2ba6a4"
   },
   "outputs": [],
   "source": [
    "player_box_score[player_box_score['offFNm3'].isna()].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YOiEqgr8XiYF"
   },
   "source": [
    "Those games do not seem to have anything different from the rest of them. In fact, doing a quick search, it is possible to find that basketball games have between two and three officials. Games whose third official is missing are represented with a missing value on the corresponding cells, such as the one described [here](https://www.basketball-reference.com/boxscores/201503080DET.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Oen1Bw7Ea0-t"
   },
   "source": [
    "### college\n",
    "\n",
    "This dataset contains general statistics of player's performance both in the NBA and in the NCAA, the main US college league. Each row contains the information of one player."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 455
    },
    "colab_type": "code",
    "id": "AkUgnUHLXcHk",
    "outputId": "902a4a8d-c485-4622-bca1-02ff8ecb8434"
   },
   "outputs": [],
   "source": [
    "college = pd.read_csv('college.csv')\n",
    "college.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ym_EUL3Vb9yl"
   },
   "source": [
    "The first issue with this dataset is that metadata has not been provided and some columns are difficult to interpret. Consequently, we have drafted a description of each column comparing the metrics shown in the dataset to the main metrics for basketball players in the [NBA](https://www.basketball-reference.com) and in the [NCAA](https://www.sports-reference.com/cbb/).\n",
    "  * Unnamed: 0 - index column.\n",
    "  * active_from: year a player started  playing in NBA.\n",
    "  * active_to: year a player finished playing in NBA.\n",
    "  * birth_date: date of birth of a player.\n",
    "  * college: college or colleges they attended.\n",
    "  * height: height of a player in inches-feet.\n",
    "  * name: first and last name of players.\n",
    "  * position: position or positions a player uses. The first one if the preferred one if multiple.\n",
    "  * url: string to add to https://www.basketball-reference.com/ to find data for the NBA stats of the player.\n",
    "  * weight: weight of the player in pounds.\n",
    "  \n",
    "The following columns contain data from the NBA performance of players.\n",
    "  * NBA__3ptapg: 3-Point Field Goal Attempts Per Game (3PA)\n",
    "  * NBA__3ptpct: 3-Point Field Goal Percentage (3P%)\n",
    "  * NBA__3ptpg: 3-Point Field Goal Per Game (3P)\n",
    "  * NBA__efgpct: Effective Field Goal Percentage (eFG%)\n",
    "  * NBA_fg%: Field Goal Percentage (FG%)\n",
    "  * NBA_fg_per_game: Field Goals Per Game (FG)\n",
    "  * NBA_fga_per_game: Field Goal Attempts Per Game (FGA)\n",
    "  * NBA_ft%: Free Throw Percentage (FT%)\n",
    "  * NBA_ft_per_g: Free Throws Per Game (FT)\n",
    "  * NBA_fta_p_g: Free Throw Attepmts Per Game (FTA)\n",
    "  * NBA_g_played: Games Played (G)\n",
    "  * NBA_ppg: Points Per Game (PTS)\n",
    "\n",
    "The following columns contain data from the NCAA performance of players.\n",
    "  * NCAA__3ptapg: 3-Point Field Goal Attempts Per Game (3PA)\n",
    "  * NCAA__3ptpct: 3-Point Field Goal Percentage (3P%)\n",
    "  * NCAA__3ptpg: 3-Point Field Goal Per Game (3P)\n",
    "  * NCAA_efgpct: Effective Field Goal Percentage (eFG%)\n",
    "  *\tNCAA_fgapg: Field Goal Attempts Per Game (FGA)\n",
    "  *\tNCAA_fgpct: Field Goal Percentage (FG%)\n",
    "  *\tNCAA_fgpg: Field Goals Per Game (FG)\n",
    "  *\tNCAA_ft: Free Throw Percentage (FT%)\n",
    "  *\tNCAA_ftapg: Free Throw Attepmts Per Game (FTA)\n",
    "  *\tNCAA_ftpg: Free Throws Per Game (FT)\n",
    "  *\tNCAA_games: Games Played (G)\n",
    "  *\tNCAA_ppg: Points Per Game (PTS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Jz5aHGDqeBfw"
   },
   "source": [
    "Data from this dataset is very messy. In order to clean it, severall steps are needed. \n",
    "\n",
    "1. Drop meaningless columns.\n",
    "\n",
    "2. Use a better format for the birth date.\n",
    "\n",
    "3. Encode the universities players attended.\n",
    "\n",
    "4. Encode height with only one numerical value.\n",
    "\n",
    "5. Divide first and last name on the name column to ease the comparison of this data with the previous one.\n",
    "\n",
    "6. Encode the positions for players\n",
    "\n",
    "7. Address some of the missing values.\n",
    "\n",
    "8. Ensure that data is in the correct data type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0qAru3ZvfUy3"
   },
   "source": [
    "#### Drop meaningless columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s2cmTleFfkbd"
   },
   "source": [
    "Columns 'Unnamed: 0' and 'url' are not useful to perform any analysis. Therefore, they should be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hFwr81S2b8FO"
   },
   "outputs": [],
   "source": [
    "college = college.drop(columns = ['Unnamed: 0', 'url'], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BjFsMycefvMs"
   },
   "source": [
    "#### Use a better format for the birth date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "44G-Uklkfzzm"
   },
   "source": [
    "The function defined previously is used to encode 'birth_date' in three different columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mjmayD-ufufL"
   },
   "outputs": [],
   "source": [
    "cleanDate(college, 'birth_date', 'birth_day', 'birth_month', 'birth_year')\n",
    "college = college.drop(columns = 'birth_date')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kv5uQSDuf9cM"
   },
   "source": [
    "#### Encode universities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rk8R1f85gAok"
   },
   "source": [
    "Columns college contains the name of all universities that a player has attended, and a missing value if he has not attended a US college. If he has attended more than one of them , then they are separated by commas. However, this presents an issue when trying to find all different universities that a player has attended, as some universities have a comma in their name, such as \"University of California, Los Angeles\".\n",
    "\n",
    "We observe that this peculiarity occurs when there exists an addition to the university name after it. Therefore, a function is created such that, when one of the porions divided by commas does not contain the words University, College or Institute, it is considered to be part of the previous name. This function is applied to create a boolean dataframe that tells if a player has gone to any of the possible universities. If there is a missing value, we consider that he has not attended a US institution, because he has attended a foreign one or because he has not pursued higher eduacation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "CAcirXiWf8ow",
    "outputId": "94bc5f84-49d7-4195-b10c-013dd289e439"
   },
   "outputs": [],
   "source": [
    "def cleanUniversityLists(college_list):\n",
    "  '''\n",
    "  Args:\n",
    "    college_list (character): character with a list of colleges separated by comma\n",
    "  Returns:\n",
    "    list of all colleges\n",
    "  '''\n",
    "  college_list = college_list.split(', ')\n",
    "  trues = np.array([True if re.search('University|College|Institute', x) else False for x in college_list])\n",
    "  trues = np.append(trues, True)\n",
    "  return list(np.array([college_list[i] if trues[i+1] else college_list[i]+ ', ' + college_list[i + 1] for i in range(len(trues) - 1)])[trues[:len(trues) - 1]])\n",
    "\n",
    "colleges = np.unique(college['college'][~college['college'].isna()].apply(cleanUniversityLists).sum())\n",
    "no_us_college = [False] * len(colleges) + [True]\n",
    "college_attended = pd.DataFrame([list(pd.Series(colleges).isin(cleanUniversityLists(col))) + [False] if col==col else no_us_college for col in college['college']], \n",
    "             columns = np.append(colleges,'No US College'),\n",
    "             index = college.index)\n",
    "college_attended.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1tF3bGL0jLfk"
   },
   "source": [
    "As this dataframe is huge, it might be unnecesary to append it to the original one. To account for the information provided by this dataframe, we will create a feature that reflects if a player has attended a university considered to be one of the best in basketball, as defined in [this site](https://www.ncsasports.org/best-colleges/best-basketball-colleges). We acknowledge that best colleges can change overtime and we might be introducing a source of bias considering those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8-V9C9whjCkr"
   },
   "outputs": [],
   "source": [
    "top_colleges = ['University of North Carolina', \n",
    "                'University of California, Los Angeles', \n",
    "                'Stanford University', \n",
    "                'University of Michigan',\n",
    "                'University of Florida',\n",
    "                'University of Virginia',\n",
    "                'Princeton University',\n",
    "                'Duke University',\n",
    "                'University of California',\n",
    "                'Harvard University']\n",
    "college['Top University'] = (college_attended[top_colleges].sum(axis = 1) > 0).astype('int')\n",
    "college = college.drop(columns = 'college')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qhO2_2YZkX-E"
   },
   "source": [
    "#### Encode height"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U9HXd7t0kbaM"
   },
   "source": [
    "We find the height in inches of each player"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hqv1dEQEkLrE"
   },
   "outputs": [],
   "source": [
    "height = college['height'].str.split('-')\n",
    "college['height'] = height.str[0].astype('float') + height.str[1].astype('float') * 0.0833333"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YAm4LNQdkkoN"
   },
   "source": [
    "#### Find first and last name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tjg15zYTkqQj"
   },
   "source": [
    "In order to be consisten with the rest of datasets availables, we split the name into first and last name. If name contains only two words, then this is easy to achieve. If not, it is difficult to know which part of the name is the first or last name. We observe that the number of names with more or less than 2 words is very low"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "wQnUiVfEkjIh",
    "outputId": "de4b0132-98fa-4cdc-dc76-660387691c9c"
   },
   "outputs": [],
   "source": [
    "(college['name'].str.split(' ').str.len() != 2).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6kHhIhmtoMWb"
   },
   "source": [
    "Therefore, each of them can be addressed individually. We create two vectors that tracks this singular cases. The first one contains the indices of names with two fisrt names and the second one the ones with one name and multiple last names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "9PikKtjFoLTC",
    "outputId": "c093e96e-9ee2-4d28-8211-92c53caa3194"
   },
   "outputs": [],
   "source": [
    "college['name'][college['name'].str.split(' ').str.len() != 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2LQa10oJomqw"
   },
   "outputs": [],
   "source": [
    "# First names with two words\n",
    "name_2 = [232, 638, 1168, 1317, 1876, 2117, 2610, 2939, 3323, 3398, 4332,\n",
    "         4371, 4405]\n",
    "\n",
    "# First names with one word\n",
    "name_1 = [372, 953, 965, 2429, 2608, 4139, 4140, 4141, 4142, 4143, 4144, \n",
    "          4145, 4146, 4148, 4179, 4513]\n",
    "\n",
    "# All of the exceptions to the general rule of naming\n",
    "names_peculiar = name_2 + name_1\n",
    "\n",
    "# Vectors that store the first and last names\n",
    "first_names = college['name'].copy()\n",
    "last_names = college['name'].copy()\n",
    "\n",
    "# Assign the first and last name to cases that follow the general norm\n",
    "mask = first_names.index.isin(names_peculiar)\n",
    "first_names[~mask] = first_names[~mask].str.split(' ').str[0]\n",
    "last_names[~mask] = last_names[~mask].str.split(' ').str[1]\n",
    "\n",
    "# Assign first and last name to peculiar cases\n",
    "first_names[name_2] = first_names[name_2].str.split(' ').str[0:2].str.join(' ')\n",
    "last_names[name_2] = last_names[name_2].str.split(' ').str[2:].str.join(' ')\n",
    "first_names[name_1] = first_names[name_1].str.split(' ').str[0]\n",
    "last_names[name_1] = last_names[name_1].str.split(' ').str[1:].str.join(' ')\n",
    "\n",
    "# Incorporate first and last names into the dataframe\n",
    "college['first_name'] = first_names\n",
    "college['last_name'] = last_names\n",
    "college = college.drop(columns = 'name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gv5q5qhGpU1T"
   },
   "source": [
    "#### Encode positions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PPDKg7hspX4F"
   },
   "source": [
    "We encode positions in two different ways. The first one creates three columns that show if a player plays a specific position. The second one determines which is the main position of a player. In basketball, there are 5 positions (https://en.wikipedia.org/wiki/Basketball_positions), but in this dataset they are sumarized in three (F - forward, C - center, G - guard). But, before that, missing values might complicate this procedure. We check for them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "uWhlhgAcrAm5",
    "outputId": "73f4a0c1-7ae4-4e83-c3b0-321eb3085454"
   },
   "outputs": [],
   "source": [
    "college[college['position'].isna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rMOJnOujrI3R"
   },
   "source": [
    "We observe that there is only one row that does not have a position. As there is only one, and data from almost all row is missing, we drop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HWk8q2bQrWI6"
   },
   "outputs": [],
   "source": [
    "college = college.drop(2152)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vxeqcLY7rZ_y"
   },
   "source": [
    "Now we proceed with the encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7-FnkbHApQPx"
   },
   "outputs": [],
   "source": [
    "# Encode all possible positions\n",
    "positions = np.unique(college['position'].str.split('-').sum())\n",
    "position_array = 1 * np.array([college['position'].str.contains(position) for position in positions]).T\n",
    "college[positions] = pd.DataFrame(position_array, columns = positions, index = college.index)\n",
    "\n",
    "# Encode main position\n",
    "college['main_position'] = college['position'].str.split('-').str[0]\n",
    "\n",
    "# Drop unnecesary columns\n",
    "college = college.drop(columns = ['position'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ohzZCCeUrkDM"
   },
   "source": [
    "#### Address missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9mHvdjLivlGX"
   },
   "source": [
    "There are plenty of missing values on this dataset. We will try to address some of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "pu9x2vvAqpJw",
    "outputId": "9653397b-989c-4190-8f06-6f82686ac075"
   },
   "outputs": [],
   "source": [
    "college.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4nDViaDrEbp7"
   },
   "source": [
    "All values from 'NCAA_efgpct' are missing. However, this value can be found using the variables available as described [here](https://en.wikipedia.org/wiki/Effective_field_goal_percentage)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nm-lyLBPEbp7"
   },
   "outputs": [],
   "source": [
    "college['NCAA_efgpct'] = (college['NCAA_fgpg'] + 0.5 * college['NCAA__3ptpg']) / college['NCAA_fgapg']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pLh43d1wEbp-"
   },
   "source": [
    "Then, three point percentage values are missing if there are no attempts of 3 points. This is reasonable, as the three point percentage is defined as the number of attempts divided by the number of three points.\n",
    "\n",
    "AWe will assign the missing values due to this reason to 0, as we assume that if a player does not even attempt 3 points, he should not be very good at it. In case this introduces some undesired assumption, we also add a column that indicates that this value was missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RBuZNkQZEbp-"
   },
   "outputs": [],
   "source": [
    "# For NBA data\n",
    "NBA_missing = (college['NBA__3ptapg'] == 0) & (college['NBA__3ptpct'].isna())\n",
    "college.loc[NBA_missing, 'NBA__3ptpct'] = 0\n",
    "college['NBA__3ptpct_missing'] = 0\n",
    "college.loc[NBA_missing, 'NBA__3ptpct_missing'] = 1\n",
    "\n",
    "# For NCAA values\n",
    "NCAA_missing = (college['NCAA__3ptapg'] == 0) & (college['NCAA__3ptpct'].isna())\n",
    "college.loc[NCAA_missing, 'NCAA__3ptpct'] = 0\n",
    "college['NCAA__3ptpct_missing'] = 0\n",
    "college.loc[NCAA_missing, 'NCAA__3ptpct_missing'] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QUhnBX-9vugN"
   },
   "source": [
    "There are plenty of columns that have the same ammount of missing values. We check if they are the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "AzMviYP6vstw",
    "outputId": "115806d7-4e86-4828-8794-5187d4407c8b"
   },
   "outputs": [],
   "source": [
    "college[college['NCAA_ppg'].isna()].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yz6NwQ2SBAhq"
   },
   "source": [
    "It looks like all columns that have missing values for 'NCAA_ppg' also have missing values for the rest of the NCAA columns. We check it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "isrZ46GHA9DV",
    "outputId": "ff917beb-07da-4bda-fd8e-94871887b12c"
   },
   "outputs": [],
   "source": [
    "college[college['NCAA_ppg'].isna()].iloc[:, 16:28].isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mbi4lAe4BvPe"
   },
   "source": [
    "All values except one are missing for the rows that 'NCAA_ppg' are missing. This leads to think that the row that has one value that is not missing is a mistake. As we are interested in predicting the performance of a player based on its university results. If there are no results to base our prediction, it does not make sense to try to do so. Therefore, we create a dataframe with observations that do not have all NCAA data missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "UzDxsxl2BPnb",
    "outputId": "fbf42e53-38ff-4067-bd72-8f957b661c5b"
   },
   "outputs": [],
   "source": [
    "college_with_data = college[~college['NCAA_ppg'].isna()]\n",
    "college_with_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "19lilEQCFCcd"
   },
   "source": [
    "Having made this midifications, we check the number of missing values remaining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "GZAWbG4CE3AJ",
    "outputId": "a9129858-36ce-4c35-b0a4-18c131a45e27"
   },
   "outputs": [],
   "source": [
    "college_with_data.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ep6mUYLbFKFN"
   },
   "source": [
    "There is a remarkable high number of 3 point stats that are missing. Doing a little of research, we find that 3 points were not recorded for old games. This might be the cause why there are many of them missing. As a consequence, we leave them there and address them when we are using different modelling techniques.\n",
    "\n",
    "The rest of missing values are also left, as they should be modelled with the aim for the data in mind."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z9JoqJ2XFuSl"
   },
   "source": [
    "#### Data Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7BgtHr2HF2dd"
   },
   "source": [
    "We check that each column has the correct data type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "61MC0SNrFHz6",
    "outputId": "290acada-4ecc-44d4-adac-92c4f726cb71"
   },
   "outputs": [],
   "source": [
    "college_with_data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EmXJaGTbGfKr"
   },
   "source": [
    "Some of the columns should be integers but are encoded as floats. We code them into integers if possible. However, missing values in some columns make this task impossible, as NaN values cannot be integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9_oRgs1yF6my"
   },
   "outputs": [],
   "source": [
    "college_with_data = college_with_data.astype({'birth_day': 'int',\n",
    "                                              'birth_month': 'int',\n",
    "                                              'birth_year': 'int',\n",
    "                                              'weight': 'int',\n",
    "                                              'NCAA_games': 'int'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S33viwMCTEpY"
   },
   "source": [
    "## 2. Predicting Player Efficiency Rating (PER)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IIzdwPFcTIL1"
   },
   "source": [
    "In this section, we will be predicting how good a player will be given their college statistics. Specifically, we will be using the Player Efficiency Rating (PER) to gauge a player's performance. \n",
    "\n",
    "Since our datasets do not have the PER of each player, we will have to manually calculate that ourselves using the formula found on this [site](https://www.sportsbettingdime.com/guides/how-to/calculate-per/). Using the player box score dataset, we will first find the cumulative statistics for each player based on all the games they played. Then, we will calculate the PER and normalize it by minutes played."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "velOV3UBV0BK"
   },
   "outputs": [],
   "source": [
    "# Add up all statistics for each player\n",
    "player = player_box_score.groupby(['playLNm', 'playFNm']).sum().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 211
    },
    "colab_type": "code",
    "id": "GEgrIh8BWXg_",
    "outputId": "e97d90f6-5e97-4d78-e8d9-eedd75b6cd8a"
   },
   "outputs": [],
   "source": [
    "# Calculate the player efficiency rate for all players\n",
    "player['PER'] = (player['playFGM'] * 125.10 + player['playSTL'] * 53.897 + player['play3PM'] * 51.757 + player['playFTM'] * 66.936 \n",
    "                 + player['playBLK'] * 39.190 + player['playORB'] * 39.190 + player['playAST'] * 34.677 + player['playDRB'] * 14.707\n",
    "                 - player['playPF'] * 17.174 - player['playFTA'] * 20.091 - player['playFGA'] * 39.190 - player['playTO'] * 53.897) / player['playMin']\n",
    "player.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TG2IIPFZMCFP"
   },
   "source": [
    "We notice that some players have extremely high PER since they only have very few minutes played. For instance, this player has a PER of 137.667 having played only played one minute between 2012 and 2018."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "colab_type": "code",
    "id": "g9-QqUKmWX0l",
    "outputId": "10f74839-b54c-4d2e-825a-aa7a262494f4"
   },
   "outputs": [],
   "source": [
    "player[player['PER'] == player['PER'].max()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a_zvR75JMaOB"
   },
   "source": [
    "We will filter out players that played less than 60 minutes in total between 2012 and 2018, since their PER does not likely reflect their true abilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AR0-KAVnFYDp"
   },
   "outputs": [],
   "source": [
    "player = player[player['playMin'] >= 60]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oQFWgMfaMvG4"
   },
   "source": [
    "Doing a quick sanity check on the distribution of Player Efficiency Rating, we see that it mostly conforms with regards to NBA Standards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "5rjvxRXbL85w",
    "outputId": "38900152-f801-4685-dda3-0815214d8375"
   },
   "outputs": [],
   "source": [
    "sns.distplot(player[\"PER\"])\n",
    "plt.xlabel('Player Efficiency Rating')\n",
    "plt.ylabel('Percent')\n",
    "plt.title('Distribution of PER')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9__QCt-sNDV2"
   },
   "source": [
    "The only information we need from this dataset is the PER of each player, so we will drop all other columns except their first and last names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mwJ7LK2Mk7sQ"
   },
   "outputs": [],
   "source": [
    "player = player[['playLNm', 'playFNm', 'PER']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gkxXJGC9N1n6"
   },
   "source": [
    "We will merge the player's PER score with the college dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 227
    },
    "colab_type": "code",
    "id": "LjqfBl8rJbsw",
    "outputId": "750fc945-0682-4950-9f72-3241e2b5e35a"
   },
   "outputs": [],
   "source": [
    "college_nba = player.merge(college_with_data, left_on = ['playLNm', 'playFNm'], right_on = ['last_name', 'first_name'])\n",
    "college_nba.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JGOResPWOH1n"
   },
   "source": [
    "To enable us to build a stronger model and get more accurate predictions, we will only consider players that are active for at least one year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 227
    },
    "colab_type": "code",
    "id": "gRZxDuqmpOOW",
    "outputId": "21313699-3dc9-4911-a702-44411229f0e4"
   },
   "outputs": [],
   "source": [
    "college_nba = college_nba[college_nba['active_from'] != college_nba['active_to']]\n",
    "college_nba.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qXDCb3hVOdV-"
   },
   "source": [
    "Since the PER directly correlates with a player's three point shot performance, we will drop those that have null values in 3 point shot in their college data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 227
    },
    "colab_type": "code",
    "id": "lHotH6c3JCvW",
    "outputId": "af8594c1-b9f7-40a6-c798-83eea5900961"
   },
   "outputs": [],
   "source": [
    "college_nba = college_nba[~college_nba['NCAA__3ptpct'].isna()]\n",
    "college_nba.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IFT_TJh8PIkL"
   },
   "source": [
    "Before performing Exploratory Data Analysis, we will separate our dataset into training and testing data by a 80-20 split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NpbZ-zr6QK_2"
   },
   "outputs": [],
   "source": [
    "train, test = train_test_split(college_nba, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e4gs75c5QTBk"
   },
   "source": [
    "We are interested in the correlation between a player's performance in NCAA and in NBA. For instance, we look at the field goal percentage of a player in NCAA and NBA and find that they are positively correlated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "KLPz_3zeQQyG",
    "outputId": "0e319173-e311-44c6-df3b-004c49973a8d"
   },
   "outputs": [],
   "source": [
    "sns.regplot(train[\"NCAA_fgpct\"], train[\"NBA_fg%\"])\n",
    "plt.xlabel('NCAA Field Goal Percentage')\n",
    "plt.ylabel('NBA Field Goal Percentage')\n",
    "plt.title('NCAA vs NBA Field Goal Percentage')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j23Z5aSUQohT"
   },
   "source": [
    "Now, we want to see if the field goal percentage in NCAA has a correlation with the player efficiency rating in NBA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "e3wKZ3S_RGSN",
    "outputId": "78f09295-eaa0-45ec-f82e-8eca7751bc86"
   },
   "outputs": [],
   "source": [
    "sns.regplot(train[\"NCAA_fgpct\"], train[\"PER\"])\n",
    "plt.xlabel('NCAA Field Goal Percentage')\n",
    "plt.ylabel('NBA Player Efficiency Rating')\n",
    "plt.title('NCAA Field Goal Percentage vs NBA Player Efficiency Rating')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Kc4HbmE9CiI"
   },
   "source": [
    "We want to find out whether positions affect the PER. It turns out that players in the center position have a higher PER."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "C50PwoHahsDL",
    "outputId": "cc9380ca-d8b4-4b17-c5cb-1ecedb0a0326"
   },
   "outputs": [],
   "source": [
    "sns.distplot(train[train['main_position'] == 'F'][\"PER\"], label = 'Forward')\n",
    "sns.distplot(train[train['main_position'] == 'G'][\"PER\"], label = 'Guard')\n",
    "sns.distplot(train[train['main_position'] == 'C'][\"PER\"], label = 'Center')\n",
    "plt.xlabel('Player Efficiency Rating')\n",
    "plt.xlabel('Percent')\n",
    "plt.title('Distribution of PER Based on Position')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IJ47rW8CRQJe"
   },
   "source": [
    "We want to see which features/columns of our dataset correlate the most with the player efficiency rating. Specifically, we are looking at columns that correspond to a player's biological trait, background, and NCAA performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 562
    },
    "colab_type": "code",
    "id": "jqOom0jobrUF",
    "outputId": "89f4935e-ca60-44d6-8ab6-52803c8d08c6"
   },
   "outputs": [],
   "source": [
    "columns = ['height', 'weight', 'active_from', 'active_to', 'birth_year', 'NCAA_efgpct', 'NCAA_fgapg', 'NCAA__3ptapg', 'NCAA__3ptpct', 'NCAA__3ptpg', \n",
    "           'NCAA_fgpct', 'NCAA_fgpg', 'NCAA_ft', 'NCAA_ftapg', 'NCAA_ftpg', 'NCAA_ppg', 'C', 'F', 'G', 'Top University']\n",
    "correlations = []\n",
    "for column in columns:\n",
    "    correlations.append(train[column].corr(train['PER']))\n",
    "plt.figure(figsize=(12, 8))\n",
    "plot = sns.barplot(columns, correlations)\n",
    "plot.set_xticklabels(plot.get_xticklabels(), rotation=45)\n",
    "plt.xlabel('Features')\n",
    "plt.ylabel('Correlation with PER')\n",
    "plt.title('Features and their correlations with PER')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oAapN57NR43C"
   },
   "source": [
    "### Ridge Regression\n",
    "\n",
    "We will now build a ridge regression model that aims to predict PER based on the given features. Since top university and NCAA_fpapg have virtually no correlation with the response variable, they will not be included in the features. We will use the SKlearn pipeline to build our mdoel, and standardize all features using StandardScaler() except the positions (c, f, g) because they were one-hot encoded from the categorical variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "51pbKXchbWmK"
   },
   "outputs": [],
   "source": [
    "ridge_model = Pipeline([\n",
    "    (\"SelectColumns\", ColumnTransformer([\n",
    "        (\"Scaler\", StandardScaler(), ['height', 'weight', 'active_to', 'active_from', 'birth_year', 'NCAA__3ptapg', 'NCAA__3ptpct', 'NCAA__3ptpg', \n",
    "                                      'NCAA_efgpct', 'NCAA_fgpct', 'NCAA_fgpg', 'NCAA_ft', 'NCAA_ftapg', 'NCAA_ftpg', 'NCAA_ppg']),\n",
    "        (\"Keep\", \"passthrough\", ['C', 'F', 'G'])\n",
    "    ])),\n",
    "    (\"LinearModel\", Ridge(alpha = 0.5))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kAeTz93ieYCD"
   },
   "outputs": [],
   "source": [
    "def rmse_score(model, X, y):\n",
    "    return np.sqrt(np.mean((y - model.predict(X))**2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mDIGED4USejr"
   },
   "source": [
    "In order to find the best hyperparameter alpha, we iterate through 30 equally spaced datapoints between 0 and 15, and find the alpha with the lowest cross validation error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FyQThii9ZkeD"
   },
   "outputs": [],
   "source": [
    "alphas = np.linspace(0, 15, 30)\n",
    "cv_values = []\n",
    "train_values = []\n",
    "for alpha in alphas:\n",
    "    ridge_model.set_params(LinearModel__alpha=alpha)\n",
    "    ridge_model.fit(train, train['PER'])\n",
    "    cv_values.append(np.mean(cross_val_score(ridge_model, train, train['PER'], scoring=rmse_score, cv=5)))\n",
    "    train_values.append(rmse_score(ridge_model, train, train['PER']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "LscHrcEKaNZ1",
    "outputId": "1330c1f0-3a1c-4807-ac8f-3cfe1d68c170"
   },
   "outputs": [],
   "source": [
    "sns.lineplot(alphas, cv_values)\n",
    "plt.xlabel('Alpha Values')\n",
    "plt.ylabel('Cross Validation Error')\n",
    "plt.title('Cross Validation Error vs Alpha Values')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "fWyla-yKgIo7",
    "outputId": "7dc36dd3-b80a-41d6-c5cb-ee55e5b751fd"
   },
   "outputs": [],
   "source": [
    "sns.lineplot(alphas, train_values)\n",
    "plt.xlabel('Alpha Values')\n",
    "plt.ylabel('Training Error')\n",
    "plt.title('Training Error vs Alpha Values')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g1RqDwOWUUP1"
   },
   "source": [
    "After finding the best alpha value, we find the performance of our final ridge regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "NePnYTRRgQHE",
    "outputId": "55c40aa7-01c4-4590-c00f-979781537735"
   },
   "outputs": [],
   "source": [
    "best_alpha = alphas[np.argmin(cv_values)]\n",
    "print(\"Best Hyperparameter: \" + str(best_alpha))\n",
    "ridge_model.set_params(LinearModel__alpha=best_alpha)\n",
    "ridge_model.fit(train, train['PER'])\n",
    "print(\"Training Score: \" + str(rmse_score(ridge_model, train, train['PER'])))\n",
    "print(\"Validation Score: \" + str(np.mean(cross_val_score(ridge_model, train, train['PER'], scoring = rmse_score, cv=5))))\n",
    "print(\"Test Score: \" + str(rmse_score(ridge_model, test, test['PER'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oXc8zBayT3KH"
   },
   "source": [
    "We compare the performance of our model with the baseline model that always predicts the mean of PER. We find that our model performs much better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "rVj5u_B0vH8d",
    "outputId": "b52030bd-6d35-4160-81e0-2b22b2495303"
   },
   "outputs": [],
   "source": [
    "baseline_model = DummyRegressor(\"mean\")\n",
    "baseline_model.fit(train, train['PER'])\n",
    "print(\"Baseline Score: \" + str(rmse_score(baseline_model, train, train['PER'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nEQBWO2DUDdT"
   },
   "source": [
    "Finally, we graph the residual plot based on our final model. It is a good plot that shows no pattern and a similar vertical spread throughout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "b1iTiddEZMQK",
    "outputId": "5ad5a8e1-fed8-4207-ec99-34717d1a448d"
   },
   "outputs": [],
   "source": [
    "PER_fitted = ridge_model.predict(train)\n",
    "residuals = train['PER'] - PER_fitted\n",
    "residuals.sum() # Sanity check: residuals sum to 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "e5MEjXwLsL6Y",
    "outputId": "10fd8eaf-99a0-472e-9787-7d0a9ee0e67d"
   },
   "outputs": [],
   "source": [
    "plot = sns.scatterplot(PER_fitted, residuals)\n",
    "plt.xlim((5, 22.5))\n",
    "plt.xlabel('Fitted PER')\n",
    "plt.ylabel('Residuals')\n",
    "plt.title('Residual Plot of Final Model')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OBSGXN9sHm0j"
   },
   "source": [
    "## 3. Predicting Position"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2jGUSd4AHuh6"
   },
   "source": [
    "The aim of this part is to predict which position is a player more likely to fullfill in his profesional life given their college performance.\n",
    "\n",
    "To do so, we first divide data into training and test before performing exploratory data analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bCF0A9DXIw0S"
   },
   "outputs": [],
   "source": [
    "train, test = train_test_split(college_with_data, test_size=0.2, random_state= 55)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7_mgBDgaIw8C"
   },
   "source": [
    "We assume that NCAA performance is a good indicator to the position that a player is going to fullfill. We wonder if biological characteristics are also important for this categorization. We start by seeing if height and weight change between categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "1B6U48FwHpO5",
    "outputId": "3402ecf6-8bff-4b07-9cd5-239c4581e12d"
   },
   "outputs": [],
   "source": [
    "sns.scatterplot(x = 'height', y = 'weight', data = train[train['main_position'] == 'C'], color = 'red', label = 'C')\n",
    "sns.scatterplot(x = 'height', y = 'weight', data = train[train['main_position'] == 'F'], color = 'green', label = 'F')\n",
    "sns.scatterplot(x = 'height', y = 'weight', data = train[train['main_position'] == 'G'], color = 'blue', label = 'G')\n",
    "plt.title('Height and weight by category')\n",
    "plt.xlabel('height (in)')\n",
    "plt.ylabel('weight (lb)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aLMN-bQUI_Os"
   },
   "source": [
    "From the graph it seems that indeed height and weight help determine which position will be assigned to a player.\n",
    "\n",
    "Other biological features are the day, month and year of birth. We will use the year of birth as a feature to account for modifications in the importance of features as years change. We wonder if the month of birth is important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "TDwbN7inIi6J",
    "outputId": "b7d5ec4f-60ae-4d8d-906a-1031905e5b12"
   },
   "outputs": [],
   "source": [
    "bins = np.arange(.5, 12.6, 1)\n",
    "sns.distplot(train.loc[train['C'] == 1, 'birth_month'], kde = False, label = 'C', bins = bins, norm_hist = True)\n",
    "sns.distplot(train.loc[train['F'] == 1, 'birth_month'], kde = False, label = 'F', bins = bins, norm_hist = True)\n",
    "sns.distplot(train.loc[train['G'] == 1, 'birth_month'], kde = False, label = 'G', bins = bins, norm_hist = True)\n",
    "plt.legend()\n",
    "plt.xlabel('Month of brith')\n",
    "plt.ylabel('Percent per month')\n",
    "plt.title('Distribution of the month of birth between classes')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d7IZfoXKJpxN"
   },
   "source": [
    "We observe that month does not seem to be a good predictor for the position. However, we observe that the number of basketball players born in April is surprisingly small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "VY5_efdOJl3x",
    "outputId": "e522a0c6-e2dd-4f99-cd0e-95b003ae5031"
   },
   "outputs": [],
   "source": [
    "sns.distplot(college_with_data['birth_month'], kde = False, label = 'C', bins = bins, norm_hist = True)\n",
    "plt.xlabel('Month of brith')\n",
    "plt.ylabel('Percent per month')\n",
    "plt.title('Distribution of the month of birth')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-iO-CLP9J8k0"
   },
   "source": [
    "It is known that the month of birth of children is a factor that can influence how they perform during early years of school. As kids born in January and kids born in December are relatively different in age but are put in the same class, their development differs. A similar phenomena could happen here. We use hypothesis testing to determine if this difference is due to chance.\n",
    "\n",
    "We define\n",
    "\n",
    "* H0: a basketball player is equally likely to be born at any month and any observed difference is due to chance.\n",
    "* H1: a basketball player is less likely to be born in April than in any other month.\n",
    "\n",
    "We use a p-value cutoff of 5%.\n",
    "\n",
    "We use the number of players that are born in April as the test statistic. Small values of the test statistic point towards the alternative hypothesis. The observed value of the test statistic is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "_KDr86F_J7Sh",
    "outputId": "47247949-8fd5-480c-d2d0-f9053c391754"
   },
   "outputs": [],
   "source": [
    "empirical_test_statistic = len(college_with_data[college_with_data['birth_month'] == 4])\n",
    "empirical_test_statistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X3I9zHyoK8sj"
   },
   "source": [
    "To find the probability of finding a value as extreme as the test statistic, we use a probability distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "yshXZwH5Kwdx",
    "outputId": "caf8c303-73c4-49c8-fa99-b6b8142d1868"
   },
   "outputs": [],
   "source": [
    "distribution = binom(len(college_with_data), 1/12)\n",
    "sum([distribution.pmf(k) for k in range(empirical_test_statistic + 1)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "itTfNI81LDWj"
   },
   "source": [
    "As 0.015 < 0.05 reject the null hypothesis in favor of the alternative. However, the alternative hypothesis, that there are less players born in April than any other month, is a side conclusion that sould be explored more deeply. As it is a contraintuitive result, further research should be done. For example, it could be possible that there are less children born in April, which causes that there are less players born that month."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ypE99svRLZpr"
   },
   "source": [
    "### Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d4FJa9PJLdcK"
   },
   "source": [
    "Here, we will predict the position of a player using data from their biological characteristics, as well as the NCAA data. To do so, we first need to addapt the college_with_data dataframe in order to be of use in our techniques."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "colab_type": "code",
    "id": "lpANCTjffx8v",
    "outputId": "f2b80531-8037-4169-fca8-597b1494b603"
   },
   "outputs": [],
   "source": [
    "college_with_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YyKIk6x-LCWQ"
   },
   "outputs": [],
   "source": [
    "# Columns that contain missing values and will be used for training\n",
    "# NBA features are removed\n",
    "missing_columns = train.columns[train.isna().sum() > 0][6:]\n",
    "# Quantitative columns that will be usedfor training\n",
    "quantitative_columns = list(train.columns[2:4]) + list(train.columns[16:28]) + ['birth_year']\n",
    "# Categorical columns that will be used for training\n",
    "categorical_columns = ['main_position'] + list(train.columns[38:])\n",
    "# All output columns\n",
    "#They include columns that will be added posteriorly with information about missing values.\n",
    "output_columns = quantitative_columns + categorical_columns + [col + '_missing' for col in missing_columns]\n",
    "\n",
    "# Define and fit scalers and imputers\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(train.loc[:, quantitative_columns])\n",
    "\n",
    "quantitative_imputer = SimpleImputer()\n",
    "quantitative_imputer.fit(train.loc[:, quantitative_columns])\n",
    "categorical_imputer = SimpleImputer(strategy = 'most_frequent')\n",
    "categorical_imputer.fit(train.loc[:, categorical_columns])\n",
    "\n",
    "def cleanDf(data):\n",
    "  '''\n",
    "  Args:\n",
    "    data (dataframe): data that needs to be transformed\n",
    "  Returns:\n",
    "    dataframe with the data with missing values removed, columns explaining \n",
    "    where missing values were, quantitative variables scaled and useful \n",
    "    columns selected.\n",
    "  '''\n",
    "  t = data.copy()\n",
    "\n",
    "  # Add a binary column that explains where missing values were\n",
    "  for col in missing_columns:\n",
    "    t[col + '_missing'] = 0\n",
    "    t.loc[t[col].isna(), col + '_missing'] = 1\n",
    "\n",
    "  # Scale and imput columns\n",
    "  t.loc[:, quantitative_columns] = quantitative_imputer.transform(t.loc[:, quantitative_columns])\n",
    "  t.loc[:, categorical_columns] = categorical_imputer.transform(t.loc[:, categorical_columns])\n",
    "  t.loc[:, quantitative_columns] = scaler.transform(t.loc[:, quantitative_columns])\n",
    "  \n",
    "  # Only return output_columns\n",
    "  return t.loc[:, output_columns]\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FqKNt0lCeG9A"
   },
   "source": [
    "We find the training and test cleaned data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JIItYww2eGXN"
   },
   "outputs": [],
   "source": [
    "clean_train = cleanDf(train)\n",
    "clean_test = cleanDf(test)\n",
    "X_train = clean_train.drop(columns = 'main_position')\n",
    "Y_train = clean_train['main_position']\n",
    "X_test = clean_test.drop(columns = 'main_position')\n",
    "Y_test = clean_test['main_position']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-s0KpnIIeOY3"
   },
   "source": [
    "#### Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KeM78kyYfLy7"
   },
   "source": [
    "Now we fit a multi-class logistic regression to try to predict the main position of players. We will use cross validation to determine which norm penalty or parameters are the best for this model. We also add a line such that the warnings are ignored.\n",
    "\n",
    "The reason behind this is that, the only solver that allows l1 penalties and multiple classes in logistic regression does not converge. However, we have tested the 'newton-cg' solver when the penalty is l2 and the errors in both models are equivalent. Therefore, we consider that using 'saga' does not hurt our predictions, but we will use 'newton-cg' when possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "5xOf1y_3eNHA",
    "outputId": "8a6e01a8-950f-4380-d6cb-21934726137b"
   },
   "outputs": [],
   "source": [
    "def accuracy(model, X, y):\n",
    "  '''\n",
    "  Args:\n",
    "    model: an sklearn model that accepts a predict function\n",
    "    X (dataframe): features to predict from\n",
    "    y (list-like): correct predictions for data in X\n",
    "  Returns:\n",
    "    accuracy of the model\n",
    "  '''\n",
    "  return np.mean(model.predict(X) == y)\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "errors = {}\n",
    "for norm, c in [(norm, c) for norm in ['l1', 'l2'] for c in np.arange(1, 50, 5)]:\n",
    "  logistic_model = LogisticRegression(penalty = norm, C = c, solver = 'saga', random_state = 55)\n",
    "  errors[norm + ' c=' + str(c)] = np.mean(cross_val_score(logistic_model, X_train, Y_train, scoring = accuracy, cv = 5))\n",
    "\n",
    "max(errors.keys(), key = lambda x: errors[x]), max(errors.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KlBSaQV5hdNR"
   },
   "source": [
    "We will try different models before commiting to the best one. But first, we wish to explore which features are better to predict each of the outputs. \n",
    "\n",
    "To do so, we use the preoperties of the l1 norm as feature selection. We start with a very high penalty and then gradually reduce it. We will see which coefficients stop being zero first, as they will be the more important when predicting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "mkV_v2hNhb3K",
    "outputId": "882dee87-d314-4d64-a1a5-a3c66ee8d45a"
   },
   "outputs": [],
   "source": [
    "# Find the coefficients for the logistic regression models\n",
    "coeffs_C = list()\n",
    "coeffs_F = list()\n",
    "coeffs_G = list()\n",
    "for c in 10.0 ** np.arange(-3, 5, .1):\n",
    "  logistic_model = LogisticRegression(penalty = 'l1', C = c, solver = 'saga', random_state = 42)\n",
    "  logistic_model.fit(X_train, Y_train)\n",
    "  coeffs_C.append(logistic_model.coef_[0])\n",
    "  coeffs_F.append(logistic_model.coef_[1])\n",
    "  coeffs_G.append(logistic_model.coef_[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298
    },
    "colab_type": "code",
    "id": "osxnraKwiumC",
    "outputId": "17be97af-9eec-4852-915a-9021f814162d"
   },
   "outputs": [],
   "source": [
    "# The best coefficients to predict centers are\n",
    "coeffs_C = pd.DataFrame(coeffs_C, columns = X_train.columns)\n",
    "coeffs_C['C'] = 10.0 ** np.arange(-3, 5, .1)\n",
    "coeffs_C_plot = coeffs_C.loc[:, coeffs_C.ne(0).idxmax().sort_values()[:10].index]\n",
    "coeffs_C_plot.plot(x = 'C', logx = True).legend(loc='center left',bbox_to_anchor=(1.0, 0.5))\n",
    "plt.ylabel('Coefficients')\n",
    "plt.title('Change in coefficients for LASSO regression and class C')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298
    },
    "colab_type": "code",
    "id": "32-LPdmhjD6P",
    "outputId": "b7f495cd-897b-493e-dbde-21d44bcacb15"
   },
   "outputs": [],
   "source": [
    "# The best coefficients to predict fronts are\n",
    "coeffs_F = pd.DataFrame(coeffs_F, columns = X_train.columns)\n",
    "coeffs_F['C'] = 10.0 ** np.arange(-3, 5, .1)\n",
    "coeffs_F_plot = coeffs_F.loc[:, coeffs_F.ne(0).idxmax().sort_values()[:10].index]\n",
    "coeffs_F_plot.plot(x = 'C', logx = True).legend(loc='center left',bbox_to_anchor=(1.0, 0.5))\n",
    "plt.ylabel('Coefficients')\n",
    "plt.title('Change in coefficients for LASSO regression and class F')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 298
    },
    "colab_type": "code",
    "id": "QpNS0ykTjZAi",
    "outputId": "3828e205-5a60-4f2e-e032-5eeb13aa6d4f"
   },
   "outputs": [],
   "source": [
    "# The best coefficients to predict guards are\n",
    "coeffs_G = pd.DataFrame(coeffs_G, columns = X_train.columns)\n",
    "coeffs_G['C'] = 10.0 ** np.arange(-3, 5, .1)\n",
    "coeffs_G_plot = coeffs_G.loc[:, coeffs_G.ne(0).idxmax().sort_values()[:10].index]\n",
    "coeffs_G_plot.plot(x = 'C', logx = True).legend(loc='center left',bbox_to_anchor=(1.0, 0.5))\n",
    "plt.title('Change in coefficients for LASSO regression and class G')\n",
    "plt.ylabel('Coefficients')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cEr9vN7djnap"
   },
   "source": [
    "As it was expected, height and weight have a great impact over the prediction. Also, some measure of time, such as the year of birth or the active_to columns appear on all three predictions. This leads to think that what is important for one position has changed overtime. \n",
    "\n",
    "Finally, when we are predicting fronts, we see that some lines are overlapped. This is a consequence that some columns have missing values at the exact same position in the training data. However, as we don't know how the test set is going to behave, we leave all columns to make the model more robust."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7iiTmOxsmUwy"
   },
   "source": [
    "#### PCA Regression\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1CPnWy-FqB6v"
   },
   "source": [
    "As we have seen that some features are represented multiple times on the dataset, we can try PCA. If we find a representation of data in a lower dimension, we won't have repeated features. We define a PCA function and use as predictors the PCA components. We use cross validation to determine the best number of predictors. We have restricted the values that we try for the sake of time execution, but more values have been tested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 223
    },
    "colab_type": "code",
    "id": "pjFEIUW8jkZQ",
    "outputId": "e0590c11-5e2d-4a9d-825c-09b66c84701b"
   },
   "outputs": [],
   "source": [
    "# Parameters for PCA\n",
    "u, s, vt = np.linalg.svd(X_train, full_matrices = False)\n",
    "X_mean = np.mean(X_train)\n",
    "X_sd = np.std(X_train)\n",
    "\n",
    "def PCA(t, k):\n",
    "  '''\n",
    "  Args:\n",
    "    t (dataframe): data to perform PCA to\n",
    "    k: number of principal components\n",
    "  Returns:\n",
    "    dataframe of k principal components of t\n",
    "  '''\n",
    "  D = (t - X_mean) / X_sd\n",
    "  Z = D @ vt[:k, :].T\n",
    "  Z.columns = ['PC' + str(k) for k in range(1, k+1)]\n",
    "  return Z\n",
    "\n",
    "errors_PCA = {}\n",
    "for norm, c, k in [(norm, c, k) for norm in ['l1', 'l2'] \n",
    "                   for c in np.arange(35, 46, 5) \n",
    "                   for k in np.arange(15, len(X_train.columns) + 1)]:\n",
    "  PCA_logistic_model = LogisticRegression(penalty = norm, C = c, solver = 'saga', random_state = 55)\n",
    "  errors_PCA[norm + ' c=' + str(c) + ' k=' + str(k)] = np.mean(\n",
    "      cross_val_score(PCA_logistic_model, PCA(X_train, k), \n",
    "                      Y_train, scoring = accuracy, cv = 5))\n",
    "  \n",
    "max(errors_PCA.keys(), key = lambda x: errors_PCA[x]), max(errors_PCA.values())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pQPtlT0I3FDL"
   },
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C5U7m8as2AoR"
   },
   "source": [
    "We observe that there has not been an improvement over linear regression. We explore why this might be. We plot data using the first two principal components and divide it by category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "OQ4GKyCv1kMD",
    "outputId": "a5faa3eb-57fe-46b0-e338-2e6c6c23cc86"
   },
   "outputs": [],
   "source": [
    "sns.scatterplot(data = PCA(X_train, 2), x = 'PC1', y = 'PC2', hue = Y_train)\n",
    "plt.legend()\n",
    "plt.title('First principal components of observations by position')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8B7wtgxl7ULP"
   },
   "source": [
    "From the PCA plot, it seems that the positions might not have linear boundaries between them. As logistic regression imposes linear boundaries, it might be useful to try a method that does not, such as random forests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "W6dq8s6_iasI",
    "outputId": "47b84dff-1b41-4e14-f339-3231d44bf7ff"
   },
   "outputs": [],
   "source": [
    "forest_model = RandomForestClassifier(n_estimators = 50, random_state = 55)\n",
    "np.mean(\n",
    "      cross_val_score(forest_model, X_train, \n",
    "                      Y_train, scoring = accuracy, cv = 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lWChQbhD3J_6"
   },
   "source": [
    "#### Majority vote model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFCSF7mliC6i"
   },
   "source": [
    "As the errors produed by all three models are very similar, we try a majority vote, to see if the limitations of one can be solved by the other two. We will use the best parameters for each model. We implement cross validation to check is this majority vote model performs better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "mVLICWRmiBXT",
    "outputId": "2a0951e6-9537-4fb3-9f6f-242fc8117387"
   },
   "outputs": [],
   "source": [
    "# Define all models with best parameters\n",
    "logistic_model = LogisticRegression(penalty = 'l1', C = 6, solver = 'saga', random_state = 55)\n",
    "PCA_logistic_model = LogisticRegression(penalty = 'l1', C = 40, solver = 'saga', random_state = 55)\n",
    "forest_model = RandomForestClassifier(n_estimators = 50, random_state = 55)\n",
    "\n",
    "# Perform cross-validation\n",
    "folds = KFold(n_splits = 5)\n",
    "error = []\n",
    "for tr, val in folds.split(X_train):\n",
    "  X_train_cv = X_train.iloc[tr, :]\n",
    "  Y_train_cv = Y_train.iloc[tr]\n",
    "  X_val = X_train.iloc[val, :]\n",
    "  Y_val = Y_train.iloc[val]\n",
    "  logistic_model.fit(X_train_cv, Y_train_cv)\n",
    "  PCA_logistic_model.fit(PCA(X_train_cv, 17), Y_train_cv)\n",
    "  forest_model.fit(X_train_cv, Y_train_cv)\n",
    "  logistic_y = logistic_model.predict(X_val)\n",
    "  PCA_y = PCA_logistic_model.predict(PCA(X_val, 17))\n",
    "  forest_y = forest_model.predict(X_val)\n",
    "  y_predicted = [mode([logistic_y[i], PCA_y[i], forest_y[i]]) for i in range(len(logistic_y))]\n",
    "  error.append(np.mean(y_predicted == Y_val))\n",
    "np.mean(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wt7uCF4Bro29"
   },
   "source": [
    "We observe that this cross-validation error is sligtly smaller than the one obtained from logistic regression and logistic regression with PCA. However, the differece is very small and it could be easily due to chance. As we believe our last model to be more robust, we will use it for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "frCGAMUB75Ug",
    "outputId": "2f73ca30-d152-4607-d5a8-2a2c166b3320"
   },
   "outputs": [],
   "source": [
    "def predictPosition(X):\n",
    "  '''\n",
    "  Args:\n",
    "    X (dataframe): features for prediction\n",
    "  Returns:\n",
    "    list with predictions\n",
    "  '''\n",
    "  logistic_y = logistic_model.predict(X)\n",
    "  PCA_y = PCA_logistic_model.predict(PCA(X, 17))\n",
    "  forest_y = forest_model.predict(X)\n",
    "  y_predicted = [mode([logistic_y[i], PCA_y[i], forest_y[i]]) for i in range(len(logistic_y))]\n",
    "  return y_predicted\n",
    "\n",
    "np.mean(predictPosition(X_test) == Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R330m6pXsuZp"
   },
   "source": [
    "We have found a classifier that performs with an accuracy of 89% approximately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8AJhgjfu2-s8"
   },
   "source": [
    "### Performance by class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KPdRiP0C5fyt"
   },
   "source": [
    "We study the distribution of classes over the test set to see if there is a major imbalance, i.e., some class is being badly predicted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "colab_type": "code",
    "id": "Is0Uh27o4JlR",
    "outputId": "a35d56f0-aa83-4bba-d485-4c4ee4a0da2c"
   },
   "outputs": [],
   "source": [
    "# Predictions\n",
    "Y_hat = np.array(predictPosition(X_test))\n",
    "# Real classes\n",
    "Y_test = np.array(Y_test)\n",
    "\n",
    "# Classification matrix\n",
    "class_matrix = pd.DataFrame([[sum((Y_hat == i) & (Y_test == j)) for i in ['C', 'F', 'G']] \n",
    "                             for j in ['C', 'F', 'G']], \n",
    "             columns = ['Predicted C', 'Predicted F', 'Predicted G'], \n",
    "             index = ['Actual C', 'Actual F', 'Actual G'])\n",
    "class_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rf7xSgZA5uEG"
   },
   "source": [
    "We observe that all classes are represented more or less correctly in this table. Maybe a great proportion of centers are misclassified as forwards. As there are less centers in the dataset, the penalty for misclassifying them is smaller than for other classes."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "ATKZRtcRS84A",
    "Oen1Bw7Ea0-t",
    "0qAru3ZvfUy3",
    "BjFsMycefvMs",
    "kv5uQSDuf9cM",
    "qhO2_2YZkX-E",
    "YAm4LNQdkkoN",
    "gv5q5qhGpU1T",
    "ohzZCCeUrkDM",
    "Z9JoqJ2XFuSl",
    "oAapN57NR43C",
    "ypE99svRLZpr",
    "-s0KpnIIeOY3",
    "7iiTmOxsmUwy",
    "pQPtlT0I3FDL",
    "lWChQbhD3J_6",
    "8AJhgjfu2-s8"
   ],
   "name": "Data 100 Final Project.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}