week_3_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.2** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-machine-learning/resources/bANLa) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3 - Evaluation\n",
    "\n",
    "In this assignment you will train several models and evaluate how effectively they predict instances of fraud using data based on [this dataset from Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud).\n",
    " \n",
    "Each row in `fraud_data.csv` corresponds to a credit card transaction. Features include confidential variables `V1` through `V28` as well as `Amount` which is the amount of the transaction. \n",
    " \n",
    "The target is stored in the `class` column, where a value of 1 corresponds to an instance of fraud and 0 corresponds to an instance of not fraud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "Import the data from `fraud_data.csv`. What percentage of the observations in the dataset are instances of fraud?\n",
    "\n",
    "*This function should return a float between 0 and 1.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.016684632328818484"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    # Your code here\n",
    "    df = pd.read_csv('fraud_data.csv')\n",
    "    ans = (len(df[df['Class'] == 1]) / len(df[df['Class'] == 0]))\n",
    "    return ans\n",
    "\n",
    "answer_one()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use X_train, X_test, y_train, y_test for all of the following questions\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "df = pd.read_csv('fraud_data.csv')\n",
    "\n",
    "X = df.iloc[:,:-1]\n",
    "y = df.iloc[:,-1]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Using `X_train`, `X_test`, `y_train`, and `y_test` (as defined above), train a dummy classifier that classifies everything as the majority class of the training data. What is the accuracy of this classifier? What is the recall?\n",
    "\n",
    "*This function should a return a tuple with two floats, i.e. `(accuracy score, recall score)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.98525073746312686, 0.0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    from sklearn.dummy import DummyClassifier\n",
    "    from sklearn.metrics import recall_score, accuracy_score\n",
    "    \n",
    "    # Your code here\n",
    "    \n",
    "    dummy_majority = DummyClassifier(strategy = 'most_frequent').fit(X_train, y_train)\n",
    "    # Therefore the dummy 'most_frequent' classifier always predicts class 0\n",
    "    y_dummy_predictions = dummy_majority.predict(X_test)\n",
    "\n",
    "    ans = (accuracy_score(y_test, y_dummy_predictions), recall_score(y_test, y_dummy_predictions))\n",
    "    \n",
    "    return ans\n",
    "answer_two()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "Using X_train, X_test, y_train, y_test (as defined above), train a SVC classifer using the default parameters. What is the accuracy, recall, and precision of this classifier?\n",
    "\n",
    "*This function should a return a tuple with three floats, i.e. `(accuracy score, recall score, precision score)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.99078171091445433, 0.375, 1.0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    from sklearn.metrics import recall_score, precision_score, accuracy_score\n",
    "    from sklearn.svm import SVC\n",
    "\n",
    "    # Your code here\n",
    "    svm = SVC().fit(X_train, y_train)\n",
    "    y_predictions = svm.predict(X_test)\n",
    "    \n",
    "    ans = (accuracy_score(y_test, y_predictions), recall_score(y_test, y_predictions), precision_score(y_test, y_predictions))\n",
    "    \n",
    "    return ans\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "Using the SVC classifier with parameters `{'C': 1e9, 'gamma': 1e-07}`, what is the confusion matrix when using a threshold of -220 on the decision function. Use X_test and y_test.\n",
    "\n",
    "*This function should return a confusion matrix, a 2x2 numpy array with 4 integers.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5320,   24],\n",
       "       [  14,   66]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    from sklearn.metrics import confusion_matrix\n",
    "    from sklearn.svm import SVC\n",
    "\n",
    "    # Your code here\n",
    "    svm = SVC(C = 1e9, gamma = 1e-07).fit(X_train, y_train)\n",
    "    svm_predicted = svm.decision_function(X_test) > -220\n",
    "    \n",
    "#     print(svm_predicted)\n",
    "    \n",
    "    confusion = confusion_matrix(y_test, svm_predicted)\n",
    "    \n",
    "    ans = confusion\n",
    "    \n",
    "    return ans\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Train a logisitic regression classifier with default parameters using X_train and y_train.\n",
    "\n",
    "For the logisitic regression classifier, create a precision recall curve and a roc curve using y_test and the probability estimates for X_test (probability it is fraud).\n",
    "\n",
    "Looking at the precision recall curve, what is the recall when the precision is `0.75`?\n",
    "\n",
    "Looking at the roc curve, what is the true positive rate when the false positive rate is `0.16`?\n",
    "\n",
    "*This function should return a tuple with two floats, i.e. `(recall, true positive rate)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.82499999999999996, 0.98750000000000004)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "        \n",
    "    # Your code here\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    from sklearn.metrics import precision_recall_curve\n",
    "    from sklearn.metrics import roc_curve\n",
    "\n",
    "    lr = LogisticRegression().fit(X_train, y_train)\n",
    "    \n",
    "    y_scores_lr = lr.fit(X_train, y_train).decision_function(X_test)\n",
    "    \n",
    "#     lr_predicted = lr.predict(X_test)\n",
    "    \n",
    "    precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr)\n",
    "    closest_zero_p = np.argmin(np.abs(precision-0.75))\n",
    "#     closest_zero_p = precision[closest_zero]\n",
    "    closest_zero_r = recall[closest_zero_p]\n",
    "    \n",
    "#     print(closest_zero_r)\n",
    "    \n",
    "    \n",
    "    fpr_lr, tpr_lr, _ = roc_curve(y_test, y_scores_lr)\n",
    "#     roc_auc_lr = auc(fpr_lr, tpr_lr)\n",
    "    \n",
    "    closest_zero_fpr_lr = np.argmin(np.abs(fpr_lr - 0.16))\n",
    "#     closest_zero_p = precision[closest_zero]\n",
    "    closest_zero_tpr_lr = recall[closest_zero_fpr_lr]\n",
    "    \n",
    "#     print(closest_zero_tpr_lr)\n",
    "\n",
    "    \n",
    "#     y_proba_lr = lr.fit(X_train, y_train).predict_proba(X_test)\n",
    "    \n",
    "#     confusion = confusion_matrix(y_test, lr_predicted)\n",
    "\n",
    "    ans = (closest_zero_r, closest_zero_tpr_lr)\n",
    "    \n",
    "    return ans\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "Perform a grid search over the parameters listed below for a Logisitic Regression classifier, using recall for scoring and the default 3-fold cross validation.\n",
    "\n",
    "`'penalty': ['l1', 'l2']`\n",
    "\n",
    "`'C':[0.01, 0.1, 1, 10, 100]`\n",
    "\n",
    "From `.cv_results_`, create an array of the mean test scores of each parameter combination. i.e.\n",
    "\n",
    "|      \t| `l1` \t| `l2` \t|\n",
    "|:----:\t|----\t|----\t|\n",
    "| **`0.01`** \t|    ?\t|   ? \t|\n",
    "| **`0.1`**  \t|    ?\t|   ? \t|\n",
    "| **`1`**    \t|    ?\t|   ? \t|\n",
    "| **`10`**   \t|    ?\t|   ? \t|\n",
    "| **`100`**   \t|    ?\t|   ? \t|\n",
    "\n",
    "<br>\n",
    "\n",
    "*This function should return a 5 by 2 numpy array with 10 floats.* \n",
    "\n",
    "*Note: do not return a DataFrame, just the values denoted by '?' above in a numpy array. You might need to reshape your raw result to meet the format we are looking for.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.66666667,  0.76086957],\n",
       "       [ 0.80072464,  0.80434783],\n",
       "       [ 0.8115942 ,  0.8115942 ],\n",
       "       [ 0.80797101,  0.8115942 ],\n",
       "       [ 0.80797101,  0.80797101]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():    \n",
    "    from sklearn.model_selection import GridSearchCV\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "    # Your code here\n",
    "    lr = LogisticRegression()\n",
    "\n",
    "    grid_values = {'C': [0.01, 0.1, 1, 10, 100], 'penalty': ['l1', 'l2']}\n",
    "\n",
    "    # default metric to optimize over grid parameters\n",
    "    grid_lr = GridSearchCV(lr, param_grid = grid_values, scoring = 'recall')\n",
    "    grid_lr.fit(X_train, y_train)\n",
    "    \n",
    "#     print(grid_lr.cv_results_['mean_test_score'].reshape(5,2))\n",
    "    ans = np.array(grid_lr.cv_results_['mean_test_score'].reshape(5,2))\n",
    " \n",
    "    return ans\n",
    "\n",
    "answer_six()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use the following function to help visualize results from the grid search\n",
    "def GridSearch_Heatmap(scores):\n",
    "    #%matplotlib notebook\n",
    "    import seaborn as sns\n",
    "    #import matplotlib.pyplot as plt\n",
    "    plt.figure()\n",
    "    sns.heatmap(scores.reshape(5,2), xticklabels=['l1','l2'], yticklabels=[0.01, 0.1, 1, 10, 100])\n",
    "    plt.yticks(rotation=0);\n",
    "\n",
    "#GridSearch_Heatmap(answer_six())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-machine-learning",
   "graded_item_id": "5yX9Z",
   "launcher_item_id": "eqnV3",
   "part_id": "Msnj0"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}