sklearn_example  4  from linear regressionn to logistic regression.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "X = [[2,0,-1.4],[2.2,0.2,-1.5],[2.4,0.1,-1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 从线性回归到逻辑回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在第2章，线性回归里面，我们介绍了一元线性回归，多元线性回归和多项式回归。这些模型都是广义线性回归模型的具体形式，广义线性回归是一种灵活的框架，比普通线性回归要求更少的假设。这一章，我们讨论广义线性回归模型的具体形式的另一种形式，逻辑回归（logistic regression）。\n",
    "\n",
    "和前面讨论的模型不同，逻辑回归是用来做分类任务的。分类任务的目标是找一个函数，把观测值匹配到相关的类和标签上。学习算法必须用成对的特征向量和对应的标签来估计匹配函数的参数，从而实现更好的分类效果。在二元分类（binary classification）中，分类算法必须把一个实例配置两个类别。二元分类案例包括，预测患者是否患有某种疾病，音频中是否含有人声，杜克大学男子篮球队在NCAA比赛中第一场的输赢。多元分类中，分类算法需要为每个实例都分类一组标签。本章，我们会用逻辑回归来介绍一些分类算法问题，研究分类任务的效果评价，也会用到上一章学的特征抽取方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  逻辑回归处理二分类\n",
    "普通的线性回归假设响应变量呈正态分布，也称为高斯分布（Gaussian distribution ）或钟形曲线（bell curve）。正态分布数据是对称的，且均值，中位数和众数（mode）是一样的。很多自然现象都服从正态分布。比如，人类的身高就服从正态分布，姚明那样的高度极少，在99%之外了。\n",
    "\n",
    "在某些问题里，响应变量不是正态分布的。比如，掷一个硬币获取正反两面的概率分布是伯努力分布（Bernoulli distribution），又称两点分布或者0-1分布。表示一个事件发生的概率是$P$，不发生的概率是$1-P$，概率在{0,1}之间。线性回归假设解释变量值的变化会引起响应变量值的变化，如果响应变量的值是概率的，这条假设就不满足了。广义线性回归去掉了这条假设，用一个联连函数(link function)来描述解释变量与响应变量的关系。实际上，在第2章，线性回归里面，我们已经用了联连函数。普通线性回归作为广义线性回归的特例使用的是恒等联连函数(identity link function)，将解释变量的通过线性组合的方式来联接服从正态分布的响应变量。如果响应变量不服从正态分布，就要用另外一种联连函数了。\n",
    "\n",
    "在逻辑回归里，响应变量描述了类似于掷一个硬币结果为正面的概率。如果响应变量等于或超过了指定的临界值，预测结果就是正面，否则预测结果就是反面。响应变量是一个像线性回归中的解释变量构成的函数表示，称为逻辑函数（logistic function）。一个值在{0,1}之间的逻辑函数如下所示：\n",
    "$$F(t)=\\frac 1 {1+e^{-t}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 函数的图形如下所示\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.font_manager import FontProperties\n",
    "font = FontProperties(fname=r\"/Library/Fonts/Microsoft/Fangsong.ttf\", size=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x10fb69710>]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHopJREFUeJzt3X2clXP+x/HXp9SkW21NliYrVJbk\nUZbIY9fIzRa23O1PWblZK/xEK6RIa8va5CakKJQfm2zRUslPdjV2UYR+7kpJURmr+1Lpdj6/P77T\nNqZp5tScM9c513k/H4/rMefMuebM+2umj+98r+v6XObuiIhIvFSLOoCIiCSfiruISAypuIuIxJCK\nu4hIDKm4i4jEkIq7iEgMVVjczWyMmS03s0/28LqZ2cNmttDMPjKzdsmPKSIieyORmftTQKdyXu8M\ntCjeegKPVj6WiIhURoXF3d3/CawuZ5euwNMezAIOMLODkhVQRET23n5JeI+mwNISz5cVf+6b0jua\nWU/C7J5atWodd8ghhyTh26enoqIiqlWL7yGNOI8vzmMDjS9139eKt/DYffeP7uH1nY9/+Nquz5V8\nDLDffs7BB38PwIIFC1a6e25FeZJR3K2Mz5XZ08DdRwOjAVq1auXz589PwrdPTwUFBeTn50cdI2Xi\nPL44jw00vrK4w8aNsHLlrm3VqrCtWRO2tWvDtm5d2L77DtavD9vmzXufs0YNqFULcnJ232rW/OHW\nvDmMHBm+zsy+SuT9k1HclwHNSjzPAwqT8L4iIpW2YQMsWQLLlu3avvlm1/btt7B8OXz//Z7fo149\nOOAAaNgQGjSAgw8OH+vXD6/Vqwd16+7a6tQJW+3au7b99w/FfOfH6tVTO+5kFPfJQC8zew5oD6xz\n992WZEREUqGoKBTszz8PW0HBYQwfDosWwVdfhVl3aY0bw0EHwY9/DC1bwoEHQm5u2Bo3DlujRmE7\n4ADYLxmVsopVGNnMxgP5QGMzWwb8AagB4O6PAdOAs4CFwCbgilSFFZHs5R5m4B9/HLZPPoF582D+\nfNi0add+NWrkcfjhYSmjQwc45JCwNWsGeXmhqOfkRDeOqlJhcXf37hW87sB1SUskIlnPHZYuhXff\nDdv778OcOT+chTdrBkcdBaecAkceGWbgRxwBn3/+Tzp2zI8oefrIwD82RCRuiorCbPyNN+DNN+Gt\nt6Cw+MhdzZpwzDFw4YXQti0ceywcfXRY8y7LF19UXe50puIuIpFYsgRefRWmT4cZM8KZKRCWUE45\nJSyptG8PbdpkxzJKsqm4i0iVKCoKSywvvQSTJ8PcueHzeXlwzjnQsSPk54fiLpWn4i4iKVNUBDNn\nwoQJMHFiOPWwenX4xS/gyiuhUyf46U/ByrpaRipFxV1Ekm7BAnjmmbB99VU4r/uss+CCC6Bz53C+\nuKSWiruIJMWWLfDCC/DYY/Cvf0G1anDGGXDXXdC1a7jQR6qOiruIVEphYbg0ftSocNn+4YfDkCHQ\no0e4klOioeIuIvtk7ly45x4YPx62b4cuXaBXr3BgNMZ9yTKGiruI7JU5c8JSy6RJoWfKtdfCDTeE\nGbukDxV3EUnIZ5/BwIHhrJcGDWDAAOjdO/RhkfSj4i4i5Vq+HO64A554IszU77gDbrppz1eISnpQ\ncReRMm3ZAg8/DIMHh3a4vXqF2XpuhbeJkHSg4i4iu3njDbjmmrAUc845cN990KpV1Klkb+iYtoj8\nx+rVcMUVoQ3A5s3w8sswZYoKeyZScRcRIBTy1q3hL3+Bfv3g00/DVaWSmbQsI5LlNmwIZ72MGRNa\n606dCu3aRZ1KKkszd5EsNmcOHHccjB0L/fvD7Nkq7HGh4i6Shdxh+HA48UTYuBFefx3uvlt90+NE\nyzIiWWbz5mpccgk8+2w4E+app8KNoCVeVNxFssiiRXDdde1YvDi0EOjfX31g4krFXSRLvPUWnHsu\nbNmSwyuvwC9/GXUiSSX9P1skCzz7bOjW2LAhjBz5gQp7FlBxF4kx99CW9ze/gZNOglmzIC/v+6hj\nSRVQcReJqaIiuPnmcEFS9+4wfTr86EdRp5KqojV3kRjavj3cgPrpp+H66+HBB3XgNNvoxy0SM9u2\nwcUXh8I+aBA89JAKezbSzF0kRrZuDUswkyaFTo433RR1IomKirtITGzbBhddBC++GJZheveOOpFE\nScVdJAZ27IDLLguF/eGHwzq7ZDetxIlkOPdwY43x42HIEBV2CVTcRTJc377h/qa33w633hp1GkkX\nKu4iGezBB8OB0+uuC/c6FdlJxV0kQ02cCH36wPnnh9MdzaJOJOlExV0kA735JlxyCXToEG6LV716\n1Ikk3ai4i2SYxYvhvPPg0ENh8mTYf/+oE0k6Sqi4m1knM5tvZgvNrF8Zrx9iZjPMbI6ZfWRmuq2u\nSAqsXw+/+lU49XHqVPWKkT2rsLibWXVgBNAZOArobmZHldptADDB3dsC3YCRyQ4qku127AhXn86f\nD88/Dy1aRJ1I0lkiM/cTgIXuvsjdtwLPAV1L7eNA/eLHDYDC5EUUEYCBA2HatHDv044do04j6c7c\nvfwdzC4EOrn774qf9wDau3uvEvscBEwHGgJ1gNPd/f0y3qsn0BMgNzf3uAkTJiRrHGlnw4YN1K1b\nN+oYKRPn8aXj2N58sxF33HEMZ59dyM03L6jUe6Xj+JIp7uM79dRT33f3n1W4o7uXuwG/Bp4o8bwH\nMLzUPn2Am4ofnwTMBaqV974tW7b0OJsxY0bUEVIqzuNLt7HNn+9ev7778ce7b95c+fdLt/ElW9zH\nB7znFdRtd09oWWYZ0KzE8zx2X3a5EphQ/D+LmUAtoHEC7y0i5di4MZzHXrMmvPAC5OREnUgyRSLF\nfTbQwsyam1lNwgHTyaX2WQKcBmBmPyUU9xXJDCqSjW64AebODX1jmjWreH+RnSos7u6+HegFvArM\nI5wV86mZDTKzLsW73QRcZWYfAuOBy4v/fBCRffSXv8CYMaFnzOmnR51GMk1CLX/dfRowrdTnBpZ4\nPBc4ObnRRLLXggWh0+PPfw5/+EPUaSQT6QpVkTSzdSt06wa1asGzz8J+uuuC7AP92oikmYEDYc6c\ncOONvLyo00im0sxdJI288QYMHQpXXQVdS18qKLIXVNxF0sTatXDppXD44fDAA1GnkUynZRmRNNG7\nN3z9Nbz1FsT4AkupIpq5i6SBKVPg6afhttugffuo00gcqLiLRGz1aujZE9q0gQEDok4jcaFlGZGI\n9e4NK1eGjo81a0adRuJCM3eRCE2ZEq5Eve02aNs26jQSJyruIhFZvx6uvRZatw4tBkSSScsyIhG5\n7TYoLAzdHrUcI8mmmbtIBN5+G0aOhOuv19kxkhoq7iJVbOvWcAVqXh7cdVfUaSSutCwjUsXuvz/0\naJ8yBerVizqNxJVm7iJVaPFiGDw43F3pnHOiTiNxpuIuUoV694Zq1eDBB6NOInGnZRmRKvLSS2Ep\n5t57dcs8ST3N3EWqwKZN4X6orVuH2btIqmnmLlIFhgyBJUvgn/+EGjWiTiPZQDN3kRRbtCjcgOPi\ni8M9UUWqgoq7SIr16RPugzp0aNRJJJtoWUYkhV59NRxI/fOfoWnTqNNINtHMXSRFtm2D3/8ejjgC\nbrwx6jSSbTRzF0mRxx6Dzz6DyZMhJyfqNJJtNHMXSYHVq+HOO+H003UlqkRDxV0kBQYNgrVr4YEH\nwCzqNJKNVNxFkmz+fBgxInR+POaYqNNItlJxF0myvn2hdu0wexeJig6oiiRRQUE4gDpkCDRpEnUa\nyWaauYskSVER3HxzaAp2ww1Rp5Fsp5m7SJKMHw/vvw/PPAP77x91Gsl2mrmLJMHmzeGG1+3ahR4y\nIlHTzF0kCYYPD10fx44NN+MQiZp+DUUqac0auPtu6NwZOnaMOo1IkFBxN7NOZjbfzBaaWb897PNf\nZjbXzD41s2eTG1MkfQ0ZAuvWhY8i6aLCZRkzqw6MAM4AlgGzzWyyu88tsU8LoD9wsruvMTOdBCZZ\nYelSeOgh6NED2rSJOo3ILonM3E8AFrr7InffCjwHdC21z1XACHdfA+Duy5MbUyQ93XknuOuCJUk/\niRxQbQosLfF8GdC+1D4tAczsLaA6cKe7/2/pNzKznkBPgNzcXAoKCvYhcmbYsGGDxpehEh3bl1/W\n5qmnjuf885exePEXLF6c+mzJEOefHcR/fIlKpLiX1fbIy3ifFkA+kAf8y8xau/vaH3yR+2hgNECr\nVq08Pz9/b/NmjIKCAjS+zJTo2M47D+rUgUcfbUbjxs1SHyxJ4vyzg/iPL1GJLMssA0r+5uYBhWXs\n85K7b3P3xcB8QrEXiaV33oEXX4RbboHGjaNOI7K7RIr7bKCFmTU3s5pAN2ByqX1eBE4FMLPGhGWa\nRckMKpIu3KF/f8jNDXdaEklHFS7LuPt2M+sFvEpYTx/j7p+a2SDgPXefXPzamWY2F9gB3OLuq1IZ\nXCQqf/87zJgRzpKpVy/qNCJlS+gKVXefBkwr9bmBJR470Kd4E4mtnbP2n/wErr466jQie6b2AyJ7\n4W9/C83Bxo7VfVElvan9gEiCduyAAQPgyCPDRUsi6Uwzd5EEPfsszJsHEyZA9epRpxEpn2buIgnY\nujVcjdq2LVxwQdRpRCqmmbtIAsaMgUWL4OWX1dJXMoN+TUUq8P33MHgwdOgQ2vqKZALN3EUqMGoU\nFBbCuHFgZTXjEElDmrmLlGPDhnAjjtNOA7UrkUyi4i5SjuHDYcWKsCwjkklU3EX2YO1aGDoUzj4b\nTjop6jQie0fFXWQPhg0LBV434pBMpOIuUoZVq0Jxv+ACaNcu6jQie0/FXaQM994bDqb+8Y9RJxHZ\nNyruIqWsXl2D4cOhe3c4+uio04jsGxV3kVLGjz+ELVvgD3+IOonIvlNxFylh2TJ46aWmXHoptGwZ\ndRqRfafiLlLC3XeHG3IMHFjxviLpTMVdpNiXX8ITT8BZZ33DoYdGnUakclTcRYoNHhw6Pl5yyVdR\nRxGpNBV3EeDzz+F//geuvRZyc7dGHUek0lTcRQjns+fkQL9+UScRSQ4Vd8l6c+eGW+j16gUHHhh1\nGpHkUHGXrDdwINStC337Rp1EJHlU3CWrffABvPAC9OkDjRpFnUYkeVTcJasNHAgNG8KNN0adRCS5\nVNwla82cGW543bcvNGgQdRqR5FJxl6w1YAA0aQLXXx91EpHk0w2yJSu9/nrYhg2DOnWiTiOSfJq5\nS9Zxh9tvh7w8uOaaqNOIpIZm7pJ1pk6FWbNg9GioVSvqNCKpoZm7ZJWiorDWfsQRcPnlUacRSR3N\n3CWrTJgAH30E48ZBjRpRpxFJHc3cJWts2wZ33AGtW0O3blGnEUktzdwla4wdCwsXwuTJobWvSJzp\nV1yywvffh86PHTrAOedEnUYk9RIq7mbWyczmm9lCM9tjU1Qzu9DM3Mx+lryIIpX3yCNQWAh//jOY\nRZ1GJPUqLO5mVh0YAXQGjgK6m9lRZexXD7gBeCfZIUUqY+3aUNQ7d4Zf/CLqNCJVI5GZ+wnAQndf\n5O5bgeeArmXsNxgYCmxOYj6RSrv3XlizBv70p6iTiFSdRA6oNgWWlni+DGhfcgczaws0c/epZnbz\nnt7IzHoCPQFyc3MpKCjY68CZYsOGDRpfGli5sib339+e005bybp180gkcqaMbV9pfNkhkeJe1gql\n/+dFs2rAMODyit7I3UcDowFatWrl+fn5CYXMRAUFBWh80bv66nDh0ujRB3LYYYndZilTxravNL7s\nkMiyzDKgWYnneUBhief1gNZAgZl9CZwITNZBVYnaZ5/Bk0+Gm14fdljUaUSqViLFfTbQwsyam1lN\noBsweeeL7r7O3Ru7+6HufigwC+ji7u+lJLFIgm6/HWrXDu0GRLJNhcXd3bcDvYBXgXnABHf/1MwG\nmVmXVAcU2RczZ8KkSXDLLZCbG3UakaqX0BWq7j4NmFbqcwP3sG9+5WOJ7Dt3uOkmOOgg3T5Pspfa\nD0jsTJoUZu6PPw5160adRiQaaj8gsbJ1K9x6a2gOdsUVUacRiY5m7hIrjz4KX3wB06ZB9epRpxGJ\njmbuEhtr1sCgQXD66dCpU9RpRKKl4i6xMXhwKPD33afmYCIq7hILCxbA8OHwu9/BscdGnUYkeiru\nEgs33wz77x9m7yKiA6oSA6+9BlOmwD33wIGJtY8RiT3N3CWjbd8OffqE3jG9e0edRiR9aOYuGW3k\nSPjkk3DhUk5O1GlE0odm7pKxVqyAgQPhjDPg3HOjTiOSXlTcJWPddhts3AgPP6xTH0VKU3GXjPTe\ne6FXe+/ecOSRUacRST8q7pJxduyA666DJk3CsoyI7E4HVCXjPP44vPsujBsH9etHnUYkPWnmLhnl\n22+hf3/o2BG6d486jUj6UnGXjNK3bziIOmKEDqKKlEfFXTLGG2/A00+HAq+DqCLlU3GXjLB5M1x1\nFTRvHk6BFJHy6YCqZITBg+Hzz2H6dKhdO+o0IulPM3dJex99BEOHwmWXhatRRaRiKu6S1nbsCD3a\nGzaE+++POo1I5tCyjKS1YcNg9uxwTnujRlGnEckcmrlL2vrsMxgwALp21TntIntLxV3S0o4d8Nvf\nQp068NhjOqddZG9pWUbS0rBhMHNmWI758Y+jTiOSeTRzl7Tz6adhOebcc7UcI7KvVNwlrWzZAr/5\nDTRoAKNGaTlGZF9pWUbSyh13wIcfwtSpoaWviOwbzdwlbcyYAffdB9dcA2efHXUakcym4i5pYeVK\n6NEDWrQIBV5EKkfLMhK5oiK4/PJww+spU8LpjyJSOSruErlhw+Dll+GRR6Bt26jTiMSDlmUkUu++\nC/36wXnnwX//d9RpROIjoeJuZp3MbL6ZLTSzfmW83sfM5prZR2b2DzP7SfKjStysWAEXXghNm8KT\nT+q0R5FkqrC4m1l1YATQGTgK6G5mR5XabQ7wM3dvAzwPDE12UImX7dvDBUrLl8OkSaHro4gkTyIz\n9xOAhe6+yN23As8BXUvu4O4z3H1T8dNZQF5yY0rcDBgA//gHPPootGsXdRqR+EnkgGpTYGmJ58uA\n9uXsfyXwSlkvmFlPoCdAbm4uBQUFiaXMQBs2bND49qCgIJd77jmaX/2qkObNF5Bu/5n0s8tscR9f\nohIp7mWthHqZO5pdAvwMOKWs1919NDAaoFWrVp6fn59YygxUUFCAxre7998Pd1Xq0AEmTjyYnJyD\nkx+ukvSzy2xxH1+iEinuy4BmJZ7nAYWldzKz04HbgVPcfUty4kmcFBZCly6Qmwt/+xvk5ESdSCS+\nEinus4EWZtYc+BroBlxccgczawuMAjq5+/Kkp5SMt2lTuOnGunXw9tvqGyOSahUWd3ffbma9gFeB\n6sAYd//UzAYB77n7ZOBeoC4w0cL5bEvcvUsKc0sG2b4dunWDDz4IM/Y2baJOJBJ/CV2h6u7TgGml\nPjewxOPTk5xLYsIdrrsutBUYMSIsy4hI6ukKVUmpu+6C0aOhf39dgSpSlVTcJWUeeQQGDgzdHv/0\np6jTiGQXFXdJiaeeguuvDwdR1VpApOqpuEvSTZwIV14JZ5wBf/0r1KgRdSKR7KPiLkn117+GnjEd\nOuhcdpEoqbhL0owbBxdfDCefDK+8optuiERJxV2SYuzYcOD0lFNg2jSoWzfqRCLZTcVdKm3oUPjt\nb8Ma+9SpmrGLpAMVd9lnRUVwyy1w661w0UXhQqXataNOJSKge6jKPtqypRoXXxwOoPbqBQ89BNU0\nVRBJGyruste+/Rb69DmWuXNhyBDo21fnsYukGxV32Stz5oSbWf/733V54QU4//yoE4lIWfSHtCTs\nmWfC+es7dsBDD/2fCrtIGlNxlwpt3hw6O156KZx4YribUqtW30UdS0TKoeIu5Zo3D9q3h5Ej4aab\n4LXXdKMNkUygNXcpkzs8/jjceGM4vXHqVDj77KhTiUiiNHOX3SxdCp06wdVXw0knwYcfqrCLZBoV\nd/mPoiIYNQpat4Y33wx3Tpo+HQ4+OOpkIrK3tCwjAHz0EVxzDcycCfn5oQf7YYdFnUpE9pVm7llu\n1Sq44QZo1w4+/xyefhpef12FXSTTqbhnqS1bQsuAFi3C8kvPnjB/fujsqKtNRTKfinuW2b49tOdt\n2RJ+/3s47rhwwHTkSPjRj6JOJyLJouKeJbZtC0X96KNDe94mTcLB0unTwwFUEYkXHVCNue++gzFj\n4IEHYMkSaNsWJk2Cc8/V8otInKm4x9QXX4SllieegPXrw63vHn0UOndWURfJBiruMbJ1a7hhxqhR\noU3AfvvBr38d1tZPOCHqdCJSlVTcM5w7vPtu6Ng4fjysXg3NmsGgQXDllboASSRbqbhnoKIimD0b\nnn8eJk6Er76CWrVCn/UePeDMM6F69ahTikiUVNwzxHffwYwZ8PLLYenlm2/CssuZZ8If/xgOkDZo\nEHVKEUkXKu5patu2MDufMQP+/nd4663wubp1Q1Ovrl1DM6+GDaNOKiLpSMU9TaxZE4r522+Hpl2z\nZsHGjeG1Y48NrXd/+ctw1ktOTrRZRST9qbhHYMWK0Khrzhz44AN4773Q1wWgWjVo0wYuvxxOPRVO\nOQUaN440rohkIBX3FCkqgq+/DkV73rxd28cfw7ff7tqvWbPQAuCKK+D448Mpi/XrR5dbROJBxX0f\nucPKleHGFkuWhDNWvvwSFi2CxYthwYKfs2XLrv3r1oUjjwwXER1zTNjattWsXERSQ8W9hB07YO3a\n0AZ3xYpQvJcvDzPtb7+Ff/8bCgvDmSqFhfygeEO4Hd1hh0Hz5tCqVSEdOzajRYtQ1Js21ZWhIlJ1\nEiruZtYJeAioDjzh7kNKvZ4DPA0cB6wCLnL3L5MbtWzuochu2hS2jRt3bRs2hFMId35cvx7Wrdu1\nrV0bDmSW3NzL/j4NGsBBB4XtpJMgL2/Xdsgh8JOfQKNGuwp4QcEX5Oc3q4r/BCIiu6mwuJtZdWAE\ncAawDJhtZpPdfW6J3a4E1rj7EWbWDbgHuKi89121qia33x5O79u2LVw6v3PbsmXXx53b5s27b5s2\nwfff77kgl2X//eGAA0KxbtAgLIu0aBFOKWzUaNeWmxtea9IkbDpDRUQySSIz9xOAhe6+CMDMngO6\nAiWLe1fgzuLHzwOPmJm577nsrlqVwz33QI0aULNm2GrUCEV05/OcnF1bvXrhKsxatUKB3vmxdu1d\nH+vUCR/r1g2P69QJX1evXvhc/frhwh8RkbhLpNQ1BZaWeL4MaL+nfdx9u5mtAxoBK0vuZGY9gZ7F\nT7fs2GGf7NgRZuEx1JhS44+ZOI8vzmMDjS/TtUpkp0SKe1mHAUvPyBPZB3cfDYwGMLP33P1nCXz/\njKTxZa44jw00vkxnZu8lsl8id2JaBpQ8MpgHFO5pHzPbD2gArE4kgIiIJF8ixX020MLMmptZTaAb\nMLnUPpOBy4ofXwi8Xt56u4iIpFaFyzLFa+i9gFcJp0KOcfdPzWwQ8J67TwaeBJ4xs4WEGXu3BL73\n6ErkzgQaX+aK89hA48t0CY3PNMEWEYmfRJZlREQkw6i4i4jEUOTF3cyuN7P5ZvapmQ2NOk8qmNnN\nZuZmFps2YWZ2r5l9ZmYfmdnfzOyAqDMlg5l1Kv59XGhm/aLOk0xm1szMZpjZvOJ/b72jzpRsZlbd\nzOaY2dSosySbmR1gZs8X/7ubZ2Ynlbd/pMXdzE4lXN3axt2PBu6LMk8qmFkzQuuGJVFnSbLXgNbu\n3gZYAPSPOE+llWi10Rk4CuhuZkdFmyqptgM3uftPgROB62I2PoDewLyoQ6TIQ8D/uvuRwLFUMM6o\nZ+7XAkPcfQuAuy+POE8qDAP6UsZFXZnM3ae7+/bip7MI1z9kuv+02nD3rcDOVhux4O7fuPsHxY+/\nIxSHptGmSh4zywPOBp6IOkuymVl94BeEMxNx963uvra8r4m6uLcEfm5m75jZG2Z2fMR5ksrMugBf\nu/uHUWdJsd8Cr0QdIgnKarURm+JXkpkdCrQF3ok2SVI9SJhIFUUdJAUOA1YAY4uXnZ4wszrlfUHK\n22iZ2d+BH5fx0u3F378h4U/E44EJZnZYJl0AVcH4bgPOrNpEyVPe2Nz9peJ9bif8uT+uKrOlSEJt\nNDKdmdUFXgB+7+7ro86TDGZ2DrDc3d83s/yo86TAfkA74Hp3f8fMHgL6AXeU9wUp5e6n7+k1M7sW\nmFRczN81syJC058Vqc6VLHsan5kdAzQHPrTQ5D0P+MDMTnD3f1dhxH1W3s8OwMwuA84BTsuk/yGX\nI5FWGxnNzGoQCvs4d58UdZ4kOhnoYmZnAbWA+mb2F3e/JOJcybIMWObuO//Sep5Q3Pco6mWZF4GO\nAGbWEqhJTLq5ufvH7t7E3Q9190MJP5x2mVLYK1J8A5dbgS7uvinqPEmSSKuNjGVhlvEkMM/dH4g6\nTzK5e393zyv+t9aN0AIlLoWd4rqx1Mx2doQ8jR+2Xd9N1N3NxwBjzOwTYCtwWUxmgNngESAHeK34\nL5NZ7n5NtJEqZ0+tNiKOlUwnAz2Aj83s/4o/d5u7T4swkyTuemBc8cRjEXBFeTur/YCISAxFvSwj\nIiIpoOIuIhJDKu4iIjGk4i4iEkMq7iIiMaTiLiISQyruIiIx9P+REsI7RW3lhgAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10fb690b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "plt.figure()\n",
    "plt.axis([-6, 6, 0, 1])\n",
    "plt.grid(True)\n",
    "X = np.arange(-6,6,0.1)\n",
    "y = 1 / (1 + np.e ** (-X))\n",
    "plt.plot(X, y, 'b-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在逻辑回归中，$$t$$是解释变量的线性组合，公式如下：\n",
    "$$F(t)=\\frac 1 {1+e^{-(\\beta_0+\\beta_x)}}$$\n",
    "对数函数（logit function）是逻辑函数的逆运算：\n",
    "\n",
    "$$g(x)=ln {\\frac {F(x)} {1-F(x)}} = \\beta_0+\\beta_x$$\n",
    "定义了逻辑回归的模型之后，我们用它来完成一个分类任务。\n",
    "### 垃圾邮件分类¶\n",
    "经典的二元分类问题就是垃圾邮件分类（spam classification）。这里，我们分类垃圾短信。我们用第三章介绍的TF-IDF算法来抽取短信的特征向量，然后用逻辑回归分类。\n",
    "\n",
    "我们可以用UCI Machine Learning Repository的短信垃圾分类数据集（SMS Spam Classification Data Set）。首先，我们还是用Pandas做一些描述性统计"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      0                                                  1\n",
      "0   ham  Go until jurong point, crazy.. Available only ...\n",
      "1   ham                      Ok lar... Joking wif u oni...\n",
      "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
      "3   ham  U dun say so early hor... U c already then say...\n",
      "4   ham  Nah I don't think he goes to usf, he lives aro...\n",
      "含spam短信数量： 747\n",
      "含ham短信数量： 4825\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('SMSSpamCollection.txt', delimiter='\\t', header=None)\n",
    "print(df.head())\n",
    "print('含spam短信数量：', df[df[0] == 'spam'][0].count())\n",
    "print('含ham短信数量：', df[df[0] == 'ham'][0].count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每条信息的前面已经被打上了标签。共5574条短信里面，4827条是ham，747条是spam。ham短信用0标记，spam短信用1标记。观察数据会看到更多建模时需要的信息。下面的几条信息体现两种类型的特征：\n",
    "\n",
    "Spam: Free entry in 2 a wkly comp to win FA Cup final tkts 21st May\n",
    "Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's\n",
    "Spam: WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.\n",
    "Ham: Sorry my roommates took forever, it ok if I come by now?\n",
    "Ham: Finished class where are you.\n",
    "让我们用LogisticRegression类来预测："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/zp/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn.cross_validation import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，用pandas加载数据.csv文件，然后用train_test_split分成训练集（75%）和测试集（25%）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('SMSSpamCollection.txt', delimiter='\\t', header=None)\n",
    "X_train_raw, X_test_raw, y_train, y_test = train_test_split(df[1],\n",
    "df[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#然后，我们建一个TfidfVectorizer实例来计算TF-IDF权重：\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(X_train_raw)\n",
    "X_test = vectorizer.transform(X_test_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，我们建一个LogisticRegression实例来训练模型。和LinearRegression类似，LogisticRegression同样实现了fit()和predict()方法。最后把结果打印出来看看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "classifier = LogisticRegression()\n",
    "classifier.fit(X_train, y_train)\n",
    "predictions = classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "预测类型：ham. 信息：God bless.get good sleep my dear...i will pray!\n",
      "预测类型：ham. 信息：Living is very simple.. Loving is also simple.. Laughing is too simple.. Winning is tooo simple.. But, being 'SIMPLE' is very difficult.. Gud nte.:-\n",
      "预测类型：ham. 信息：Once a fishrman woke early in d mrng. It was very dark. He waited a while &amp; found a sack ful of stones. He strtd throwin thm in2 d sea 2 pass time. Atlast he had jus 1stone, sun rose up &amp; he found out tht those r nt stones, those were diamonds. Moral:\"Dont wake up early in d mrng'' GOOD night\n",
      "预测类型：ham. 信息：jay says he'll put in  &lt;#&gt;\n",
      "预测类型：ham. 信息：Aldrine, rakhesh ex RTM here.pls call.urgent.\n"
     ]
    }
   ],
   "source": [
    "for i, prediction in enumerate(predictions[-5:]):\n",
    "    print('预测类型：%s. 信息：%s' % (prediction, X_test_raw.iloc[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分类模型的运行效果如何？有线性回归的度量方法在这里不太适用了。我们感兴趣的是分类是否正确（如第一章介绍的肿瘤预测问题），并不在乎它的决策范围。下面，我们来介绍二元分类的效果评估方法。\n",
    "\n",
    "### 二元分类效果评估方法\n",
    "二元分类的效果评估方法有很多，常见的包括第一章里介绍的肿瘤预测使用的准确率（accuracy），精确率（precision）和召回率（recall）三项指标，以及综合评价指标（F1 measure）， ROC AUC值（Receiver Operating Characteristic ROC，Area Under Curve，AUC）。这些指标评价的样本分类是真阳性（true positives），真阴性（true negatives），假阳性（false positives），假阴性（false negatives）。阳性和阴性指分类，真和假指预测的正确与否。\n",
    "\n",
    "在我们的垃圾短信分类里，真阳性是指分类器将一个垃圾短信分辨为spam类。真阴性是指分类器将一个正常短信分辨为ham类。假阳性是指分类器将一个正常短信分辨为spam类。假阴性是指分类器将一个垃圾短信分辨为ham类。混淆矩阵（Confusion matrix），也称列联表分析（Contingency table）可以用来描述真假与阴阳的关系。矩阵的行表示实际类型，列表示预测类型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4 1]\n",
      " [2 3]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQkAAAD3CAYAAAAOh6G5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAF99JREFUeJzt3X+0XWV95/H3JyEkAQIBEiQmgYBm\ngKKAmgUofwiCFZDK2JEx/rZDpXXqVNfYzqr+gaNr1up0uqbtcnCqcfhZUUEYaWRCKUUoYptoAiES\nghL5ISloSMBAIL/uvZ/5Y+8Lp5d79jn33nOyz7n381prr5yzz3Of/Zybe77n+bWfR7aJiGhmWt0F\niIjeliAREZUSJCKiUoJERFRKkIiISgkSEVEpQSJikpE0XdL9km4d5bWZkm6QtFnSGklLWuWXIBEx\n+Xwa2NTktUuB52y/HvhL4M9aZZYgETGJSFoEvBv4P02SXAxcWz6+CThXkqryTJDoM5JmSJrR+LyN\nn5k+yrlZ47x2/mZ6218B/wUYavL6QuBJANsDwA7gyKoMD+hk6aJ7JM0DTgLWAZ+T9NPypbcB/7FM\nMweYbXurpOOAF4FdwO8Dfz4iyz+W9N+AgwDZ3inpAHj5jwdJJ9neJOlE2w8DbwTmAt/v5nudSt51\nzsHe/uxgW2nXbdizEdjdcGqF7RXDTyRdBGy1vU7S2U2yGa3WUHlvRoJEn7C9TdKRwB7gOeAXwEyK\noDGc5gVJn5D0FduPSfoERRD5n6NkudW2JQ0AfyVpTXn+n4HhAHSRpNOAM8qAchHwM0kfAu6z3azd\nG23a9uwga25f1FbaGQt+vtv2sookZwHvkXQhMAs4VNI3bH+4Ic0WYDGwpfxSOAx4tuq6qTr2Edu3\n2B6k+PZ/BLiTV/8f/iVwsKR3Unxr/DFwoqTXDCeQtBg4RdIXgBOAH9q+BrjV9k8b8noSeIyiCnsK\nRRX2CdvXJ0B0ihn0UFtHy5zsz9leZHsJsBz4/ogAAbAS+Fj5+H1lmsqaRIJEH5B0QFlDWCDpHIro\n/9vAlym/9SXNkrScIoCcDcwHTgfOBWZQ1Armllm+mSKY/KPtDcCZkj5GUaOY03hp4HHgtcB24O+A\nkyUd3qqzK9pjYAi3dYyXpC9Jek/59ErgSEmbgf8M/Emrn09zow/YHpD0DeAC4CjgT20/K+l3gUck\nnWP7Lkl3AucAqyiGwe4HDgGOBb5oe7Ds6Dwc+Ciwt+zUXG37G5Km236h4dKi6IM4HphN0cdxCkXA\nuJui2RMTYMw+t9cnMaZ87bsp/o+wfXnD+d3AJWPJK0GiT9jeJcnAL4H3SnqSolf6HcANZZpnyjRn\nAvOAX1F8Wf2ybKZge5+km4H3An9re4ekr0s6mKKGMtJhFMHh28DRwFKKGui/lfRd27/u3rueGiZS\nS9gf0twYJ0nnS/ppOXOtZZVtgtcartoPf4i/Y/vvgX22rweOlTSn/KCfTNGp+QAwHRgADm3Mr6wt\nDAAfkbSMotnxNdv/Y+Slba+haNJcQtF/sRO4w/bV3QwQkq6StFXSg926Ri8wMIjbOuqSmsQ4lFX0\nrwDvpOgt/rGklbYf6tIlZ5WdjPdRBPZTJe3ller+TIoAchlwm+1HJb2NYtLMwRSjEo3lPwG4vxze\nPImio/N1wGvKY5XtPcCBAOX7eqj82dfafr5L77PRNcAVwHX74Vq1Sk1icjod2Gz7Udt7KariF3fr\nYrZ3AVcD91I0MU4uv+F/S9LfUUytPRD4ErC2/LEd5XyHGRT9GABIOgKY0zA6sa382eOA54EfAHvL\n1/aNUpxXTczqBtv30GJobjIwMGi3ddQlNYnxeXnWWmkLcEY3Lzg8NCnpqob+hd+WNMP2qz7Mtr9X\n/ruNYiRj+PyzNHz4bD8D/K8m1/zGKKevHeVcTEDrwc16JUiMz5hnrXXKcIBoeD7at303r9/bdeM+\n45r7G9qRIDE+w7PWhi0CnqqpLNHHbNjX2zEiQWKcfgwsLe+P+BeK2W0frLdI0Z/E4KgV096Rjstx\nKDsEPwXcTnHf/o22N9ZbqslF0rco7iM5QdIWSZfWXaZuMDDk9o66pCYxTrZXUcxsjC6w/YG6y7C/\n9HpNIkEiokbFZKoEiYioMOQEiYhoIjWJiKhkxD7vl0ms45bRjQmQdFndZZjsJvvveLgm0c5RlwSJ\niZnUf8A9YpL/jsWgp7V11CXNjYgaFStT9fZ3dU8FiXlHTPeSxS1XiO8Zxyw8gGWnzurxSbX/2s82\nHFR3EcZkFgdxqI7oq9/xbl5kr/e03T5Ix+UYLFk8gx/dvrh1whi3d732tLqLMOmt8Z1tp7VVa1Oi\nHT0VJCKmoqHUJCKiGSP2urc/hr1duohJLh2XEdHSYKZlR0QzRgymJhERVYYyuhERzRTTshMkIqKJ\nfrjBK0EiokY2PT+ZqrdLFzHpiaE2j5Y5FTvL/0jSA5I2SvriKGk+LukZSevL43db5ZuaRESNih28\nOvZdvQd4h+2d5e7x90q6zfbqEelusP2pdjNNkIioWac6LsuNk3aWT2eUx4RvjktzI6JGRgy5vaMd\nkqZLWg9spdj9fc0oyf6dpA2SbpLU8o7KBImImg0yra0DmCdpbcPxqgV5bA/aPo1iV7nTJb1hRJLv\nAUtsnwL8A23s7ZrmRkSNxjgEus32srbytX8t6W7gfODBhvPbG5J9nWJH+kqpSUTUqNjBa1pbRyuS\n5kuaWz6eDZwHPDwizYKGp++h2IGuUmoSETXr4MpUC4BrJU2nqADcaPtWSV8C1tpeCfyhpPcAA8Cz\nwMdbZZogEVEjWx27d8P2BuBNo5y/vOHx54DPjSXfBImImvX6jMsEiYgaFYvOZD2JiGgqC+FGRAVD\n7gKNiOaGZ1z2sgSJiJplIdyIaKpYTyI1iYiokOZGRDRV9EmkuRERFbJhcEQ0ZcTAUIZAI6JCZlxG\nRFMZ3YiIltJxGRFNZcZlRLSUPomIaKpYvi5BIiKacYZAI6JCFp2JiJbS3IiIpvqhT6KrA7SSzpf0\nU0mbJf1JN68V0a86uc1fN3StJlGu/f8V4J3AFuDHklbafqhb14zoN1N9nsTpwGbbjwJI+jZwMZAg\nETHMMDCFZ1wuBJ5seL4FOKOL14voO/3QJ9HNIDHaO/erEhU7I18GcMzC9KPG1NPrQaKb9ZwtwOKG\n54uAp0Ymsr3C9jLby+Yf2duTSiI6bbhPopc7LrsZJH4MLJV0nKQDgeXAyi5eL6Iv2WrrqEvX6ve2\nByR9CrgdmA5cZXtjt64X0a+m9IxL26uAVd28RkQ/szvXJyFpFnAPMJPis32T7S+MSDMTuA54C7Ad\neL/tx6vyTU9hRK3E4FDHWv17gHfY3ilpBnCvpNtsr25IcynwnO3XS1oO/Bnw/qpMe3uANmIK6FSf\nhAs7y6czymPkiOLFwLXl45uAcyVVZp4gEVGj4XkSbY5uzJO0tuG4bGR+kqZLWg9sBe6wvWZEkpfn\nL9keAHYAR1aVMc2NiDq56Jdo0zbbyyqzsweB0yTNBb4r6Q22H2xI0tb8pUapSUTUbAi1dYyF7V8D\ndwPnj3jp5flLkg4ADgOercorQSKiRqZzfRKS5pc1CCTNBs4DHh6RbCXwsfLx+4Dv29V1mTQ3ImrV\n0dmUC4BryzuwpwE32r5V0peAtbZXAlcCfyNpM0UNYnmrTBMkImo2NNSZIGF7A/CmUc5f3vB4N3DJ\nWPJNkIiokU2tU67bkSARUbNevws0QSKiZmMYAq1FgkREzdLciIimTL23gbcjQSKiZj3e2kiQiKiV\nwR0aAu2WBImImqW5ERGVMroREU0N37vRyxIkIupkIEEiIqqkuRER1RIkIqI5ZQg0Iir0wV2glStT\nlTtvNXvtfEmHdL5IEVOM2zxq0rQmUS5/9WFJLzacPo5iP881FAFmKXB/V0sYMen1aU3C9i7gRuAQ\nXnkXNwB/Q7H6za+AU7pdwIhJr19rEqUTba+QdI7tu8raxbuANwCDwOu6XsKIya7PRzeOkvQfgLdI\nWgAcA3wTWARsAN7d5fJFTG6T4AavhylW4H0G+KHtbwJI2le+dkx3ixcxBfR5TeIpYIiio/IcSYPA\nzcBdtocyuhHRAT0+BFoZJGy/KOk7wKEU+wduBC6wvapMcnuXyxcx6anPaxLY3kGxqeiT5alVDa99\nvkvlipgaah65aMeEZlxKmmP7hU4VJmLqUc83N1ruBSpppqQ3lo/nSnp7w8vv7VrJIqaKHp8n0TRI\nSJoh6TxgL3CqpLOAN1OOaJSdlgkSERM11OZRk6oZl/uArcCZFIHiCNvfByzpJOAi4IP7pZQRk9Xw\nojPtHDWpqkkIOBB4AZgHPC1pFjALeNH2t8up2xExAXJ7R8t8pMWS7pK0SdJGSZ8eJc3ZknZIWl8e\nl4+WV6OqjssjKO7ZmA28nWKb8qUUNYt1wC9aFzsiWupcf8MA8Fnb90maA6yTdIfth0ak+4Hti9rN\ntGmQsL1d0vPApRQTqOYCV5UFmS3pEtvfGfPbqPDQU/N5y3/9ZCezjBFmrHqm7iJMeoN/eG8t17X9\nNPB0+fgFSZuAhcDIIDEmrUY3lgBXU9QmrgbOBV6iuG9j7YiRjogYhzE0N+ZJWttwXNY0T2kJxd3a\na0Z5+a2SHpB0m6STW5Wv1YzLR8oL3mN7n6TNwHxgpu3HJB0l6UDbe1tdKCKaaL9TcpvtZa0SlSOP\nNwOfsf38iJfvA461vVPShcAtFN0ITbWcJwFg+7Hy35/bXm17e/l8DVkCL2L8TEeHQCXNoAgQ19v+\nv6+6nP287Z3l41XADEnzqvJsK0hUsf3SRPOImMo6OLoh4Epgk+2/aJLm6DIdkk6niAHbq/KtrAUM\nNyUkLbf97fLcO4GTbH+5dbEjoqXOjW6cBXwE+Imk9eW5z1NOgLT9VeB9wCclDQC7gOV29c4frZoK\nH6LosBwqqzELgKds3zHutxER/1qHgoTte2mxYKbtK4ArxpJvqyCxvewEeQT4I8q5EZJOpXhre0Zr\n90REe9ptStSpVZB4hGIY9FHgAdurJMm2Jb0OWNztAkZMej1+F2irILEZGCiDwsyyo+M8SdcAz9n+\neddLGDHZ9XhNotXoxn8Cfqt8/BLwRuBB4N8AR3axXBFThobaO+rSzkK4j0r6TWA1MAc4y/ZKSRMe\nPo2Y8iZBn8Qc4ByK5eveTVkxkrQcmCZpdzouIyaon4OE7Rv2V0EipqweDxKtNgxeJOnShueLyxWq\nkDRNUvbdiJigTs247JbKIGF7C/B7w4EB2A0cXj7+CMVOXhExibVzc9ZOijvHAJ4DDitXqHrJ9j91\nrWQRU0WPNzda3bvxmxRv4QJJR5enDwGO7vSCMxFTkusd3mxH0yAh6TSKFW32Amsp1pF4gmLXrnuA\nx/dD+SImvx6vSVStlr2+7JPYARxH0dR4KzBoe3VWpYqYONHnHZeSDqdY01LAE7a/V56fBayXdGL3\nixgxyfXr5jwAtp+z/WFgOq80Tf4JOKDcI/TALpcvYnJrsxZRZ02iraXnbN/Z8PgzDS9NaBXeiKDn\n+yQmtD6l7YFOFSRiqurb0Y2I2E8mc00iIiao5k7JdiRIRNSs328Vj4huS5CIiCqpSUREtQSJiGim\n7olS7UiQiKhbgkREVElNIiKqJUhERKUeDxLZOyOiTh28C7RcqPouSZskbZT06VHSSNKXJW2WtEHS\nm1vlm5pERN06V5MYAD5r+z5Jc4B1ku6w3Xi39gXA0vI4A/jr8t+mUpOIqFmntvmz/bTt+8rHLwCb\ngIUjkl0MXOfCamCupAVV+aYmEVGzMYxuzJO0tuH5CtsrRs1TWgK8CVgz4qWFwJMNz7eU555udtEE\niYg6je0u0G22l7VKJOkQ4GbgM7afH/lyk1I0lSARUbcOjm5ImkERIK5vsk/vFmBxw/NFwFNVeaZP\nIqJGnVwtW5KAK4FNtv+iSbKVwEfLUY4zgR22mzY1oIs1CUlXARcBW22/oVvXieh7natJnEWx/eZP\nJK0vz30eOAbA9leBVcCFwGbgJeB3WmXazebGNcAVwHVdvEZE35M7EyVs38vofQ6NaQz8wVjy7VqQ\nsH1P2cMaEc308zZ/EbGf9Pi07NqDhKTLgMsAZhxyeM2lidj/ev0u0NpHN2yvsL3M9rIDZh1cd3Ei\n9r8e3+av9ppExJTWBytTda0mIelbwD8DJ0jaIunSbl0roq9N1ZqE7Q90K++IyWJ4MlUvS3MjomYa\n6u0okSARUads8xcRrWQyVURUS00iIqqk4zIimjPQoRu8uiVBIqJm6ZOIiKYyTyIiqtlpbkREtdQk\nIqJagkREVElNIiKaM5B7NyKiSoZAI6JaRjciokr6JCKiudwqHhFVihmXvR0lEiQi6paOy4iokppE\nRDRn9/w8ido354mY6uT2jrbykq6StFXSg01eP1vSDknry+PyVnmmJhFRt842N64BrgCuq0jzA9sX\ntZthgkREnTq8q7jteyQt6VyOaW5E1G94TYlWR+e8VdIDkm6TdHKrxKlJRNSt/c//PElrG56vsL1i\njFe7DzjW9k5JFwK3AEurfiBBIqJmYxgC3WZ72USuZfv5hserJP1vSfNsb2v2MwkSEXUyMLj/hkAl\nHQ38yrYlnU7R5bC96mcSJCJqJNzRyVSSvgWcTdE02QJ8AZgBYPurwPuAT0oaAHYBy+3qAiRIRNSt\ng0HC9gdavH4FxRBp2xIkIuqWadkR0ZTJDV4RUS03eEVEtQSJiGjKhqHebm8kSETUrbdjRIJERN3S\nJxER1RIkIqKp7OA1Nru2bdm2/muffaLucozBPKDpjTE96Wt1F2DM+u93DMe2n7Tjt4F3XE8FCdvz\n6y7DWEhaO9G78qLalPgdJ0hERFMGBnt7eCNBIqJWBvd2kMjydRMz1lWBaiNp1C8ESfMkTSsfH9hm\nXgd1smwt9M3veNz2//J1Y5KaxASMY+mwcZN0BrAYeD1wK7AEmE0R6A8BrgeOKJMfZHuzpAXAoO2t\nwB8B/70hvxOAx4DjgeMl7QFeA/z9iOseArwWOBF4BtgE/HtJ19ne3Z13+4r9+TuuRUY3ooN+RvGh\nHigfPw580PYKSdNsDwFPSfoEMCTpTIqAcqOkE4EtI/J7ArgUWEex+MihwJ2SZg1/+CXNBS6hCESD\ntldK+j3gxf0RIKaMHu+4THOjT9h+juID+3OKb/bfBw6T9CHgU5JUpvs6MB14GvgR8BJFbeFlko4H\nlgPPUtQw3g4sAj4InNCQ9FDgX8rrvUXSucCLXXqLU1ePNzcSJPqApNmSPg4cB3yEolnxS+BG29cD\nvyqS6Q8kLQWOBg7jlf/fgcb8bD8KfBM4nGK15B8Bu4HHbT/QkPQwiibINcAPbd/JWNZ2jtZsGBxs\n76hJgkQfsL3L9jXA/cCf276vfGl5WYMYLJsbNwC/ATxK0V+xc7T8JL0eeAdwZXnqQdt/Cxwo6fCG\npBuBIduPAcdIOpgioBzX0Tc41fV4TSJ9En3G9jPlaIQomgtHUTQpsL2tHHmYB+yjmKloYCawpyGb\n+RSLo74XOA84RNIuii+NIeCuMt0FwM3l4+uBc4Dbgae69f6mpB7vk0iQ6C8q/51O8UFdSDHi8ZCk\nQ4GDKYLB9cD7KZokv6TYF/L9DflspqglvBH4CsWS6gfb3th4Mdv/T9LxkpZT/K0cR9FEWdOVdzcl\nZVfx6BBJb+OVIc4BipGLG4BB4DSKpsUXgbspdmT6R4qh0beVaWcO52X7GYraxiLb62w/Dpwqac7I\n65b9F1cDPwDWAw8Dx5a7P8VEGeyhto66pCbRP1YDvyHpKOAU4Drbe4D7JS22PSTpkxQBYbftn8DL\nk6heQ9FXQXnuTGC27Rsa8r8F+KakW4Dv2t4h6U+Bd1GMaPyCYgh2FrABeKTL73fq6PGahFrsyxF9\nRpKqNlspOx8HM8+hNxx2wHy/dc7FbaW9/ddXrqvjZrfUJCaZVrsx2c48h14yPATawxIkImrmLIQb\nEc1l0ZmIqNIHN3hlCDSibh5q72iDpKskbZX0YJPXJenLkjZL2iDpza3yTJCIqJEBD7mto03XAOdX\nvH4BxTyapcBlwF+3yjBBIqJOdkdrErbvoZiu38zFFHNsbHs1MLdcd6Sp9ElE1Mz7dwh0IfBkw/Mt\n5bmnm/1AgkREjV7gudv/wTfNazP5LElrG56vGMfKXRrlXGVbJkEioka2q/oPumELxU2BwxbR4q7e\n9ElETC0rgY+WoxxnAjtsN21qQGoSEZOKpG8BZwPzJG0BvkCxdgi2vwqsAi6kWC7gJeB3WuaZG7wi\nokqaGxFRKUEiIiolSEREpQSJiKiUIBERlRIkIqJSgkREVEqQiIhK/x/VUek/GISOoAAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1befeeb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "y_test = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]\n",
    "y_pred = [0, 1, 0, 0, 0, 0, 0, 1, 1, 1]\n",
    "confusion_matrix = confusion_matrix(y_test, y_pred)\n",
    "print(confusion_matrix)\n",
    "plt.matshow(confusion_matrix)\n",
    "plt.title('混淆矩阵',fontproperties=font)\n",
    "plt.colorbar()\n",
    "plt.ylabel('实际类型',fontproperties=font)\n",
    "plt.xlabel('预测类型',fontproperties=font)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 准确率\n",
    "准确率是分类器预测正确性的评估指标。scikit-learn提供了accuracy_score来计算："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "y_pred, y_true = [0, 1, 1, 0], [1, 1, 1, 1]\n",
    "print(accuracy_score(y_true, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LogisticRegression.score()用来计算模型预测的准确率：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn.cross_validation import train_test_split, cross_val_score\n",
    "\n",
    "df = pd.read_csv('SMS')\n",
    "X_train_raw, X_test_raw, y_train, y_test = train_test_split(df['message'], df['label'])\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(X_train_raw)\n",
    "X_test = vectorizer.transform(X_test_raw)\n",
    "classifier = LogisticRegression()\n",
    "classifier.fit(X_train, y_train)\n",
    "scores = cross_val_score(classifier, X_train, y_train, cv=5)\n",
    "print('准确率：',np.mean(scores), scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你的结果可能和这些数字不完全相同，毕竟交叉检验的训练集和测试集都是随机抽取的。准确率是分类器预测正确性的比例，但是并不能分辨出假阳性错误和假阴性错误。在有些问题里面，比如第一章的肿瘤预测问题中，假阴性与假阳性要严重得多，其他的问题里可能相反。另外，有时准确率并非一个有效的衡量指标，如果分类的比例在样本中严重失调。比如，分类器预测信用卡交易是否为虚假交易时，假阴性比假阳性更敏感。为了提高客户满意度，信用卡部门更倾向于对合法的交易进行风险检查，往往会忽略虚假交易。因为绝大部分交易都是合法的，这里准确率不是一个有效的衡量指标。经常预测出虚假交易的分类器可能有很高的准确率，但是实际情况可能并非如此。因此，分类器的预测效果还需要另外两个指标：精确率和召回率。\n",
    "\n",
    "精确率和召回率\n",
    "第一章的肿瘤预测问题中精确率和召回率的定义我们已经介绍过。在本章的垃圾短信分类器中，精确率是指分类器预测出的垃圾短信中真的是垃圾短信的比例：\n",
    "\n",
    "$$P = \\frac {TP} {TP+FP}$$\n",
    "召回率在医学领域也叫做灵敏度（sensitivity），在本例中是指所有真的垃圾短信被分类器正确找出来的比例。\n",
    "\n",
    "$$R = \\frac {TP} {TP+FN}$$\n",
    "精确率和召回率各自含有的信息都很少，它们对分类器效果的观察角度不同。精确率和召回率都不能从表现差的一种分类器中区分出好的分类器。例如，假设一个测试集包括10个阳性和0个阴性结果。分类器即使将每一个样本都预测为阳性，其召回率都是1：\n",
    "\n",
    "$$R = \\frac {10} {10+0}=1$$\n",
    "分类器如果将每一个样本都预测为阴性，或者只是预测出假阳性和真阴性，其召回率都是0。类似的，一个分类器如果只预测一个样本，结果为阳性，而且这个样本确实为阳性，那么这个分类器就是100%精确的了。\n",
    "\n",
    "scikit-learn结合真实类型数据，提供了一个函数来计算一组预测值的精确率和召回率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn.cross_validation import train_test_split, cross_val_score\n",
    "\n",
    "df = pd.read_csv('mlslpic/sms.csv')\n",
    "X_train_raw, X_test_raw, y_train, y_test = train_test_split(df['message'], df['label'])\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(X_train_raw)\n",
    "X_test = vectorizer.transform(X_test_raw)\n",
    "classifier = LogisticRegression()\n",
    "classifier.fit(X_train, y_train)\n",
    "precisions = cross_val_score(classifier, X_train, y_train, cv=5, scoring='precision')\n",
    "print('精确率：', np.mean(precisions), precisions)\n",
    "recalls = cross_val_score(classifier, X_train, y_train, cv=5, scoring='recall')\n",
    "print('召回率：', np.mean(recalls), recalls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们的分类器精确率99.2%，分类器预测出的垃圾短信中99.2%都是真的垃圾短信。召回率比较低67.2%，就是说真实的垃圾短信中，32.8%被当作正常短信了，没有被识别出来。这些数据会不断变化，因为训练集和测试集是随机抽取的。\n",
    "\n",
    "计算综合评价指标\n",
    "综合评价指标（F1 measure）是精确率和召回率的调和均值（harmonic mean），或加权平均值，也称为F-measure或fF-score。\n",
    "\n",
    "$$\\frac 1 {F1} + \\frac 1 {F1} = \\frac 1 P + \\frac 1 R $$\n",
    "即\n",
    "\n",
    "$$F1 = 2{\\frac {PR} {P+R}}$$\n",
    "综合评价指标平衡了精确率和召回率。一个二元分类模型，精确率和召回率为1，那么综合评价指标为1。如果精确率或召回率为0，那么综合评价指标为0。scikit-learn也提供了计算综合评价指标的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f1s = cross_val_score(classifier, X_train, y_train, cv=5, scoring='f1')\n",
    "print('综合评价指标：', np.mean(f1s), f1s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本例的综合评价指标是80%。由于精确率和召回率的差异比较小，所以综合评价指标的罚值也比较小。有时也会用F0.5和F2，表示精确率权重大于召回率，或召回率权重大于精确率。\n",
    "\n",
    "ROC AUC\n",
    "ROC曲线（Receiver Operating Characteristic，ROC curve）可以用来可视化分类器的效果。和准确率不同，ROC曲线对分类比例不平衡的数据集不敏感，ROC曲线显示的是对超过限定阈值的所有预测结果的分类器效果。ROC曲线画的是分类器的召回率与误警率（fall-out）的曲线。误警率也称假阳性率，是所有阴性样本中分类器识别为阳性的样本所占比例：\n",
    "\n",
    "$$F = \\frac {FP} {TN+FP}$$\n",
    "AUC是ROC曲线下方的面积，它把ROC曲线变成一个值，表示分类器随机预测的效果。scikit-learn提供了计算ROC和AUC指标的函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn.cross_validation import train_test_split, cross_val_score\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "df = pd.read_csv('mlslpic/sms.csv')\n",
    "X_train_raw, X_test_raw, y_train, y_test = train_test_split(df['message'], df['label'])\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(X_train_raw)\n",
    "X_test = vectorizer.transform(X_test_raw)\n",
    "classifier = LogisticRegression()\n",
    "classifier.fit(X_train, y_train)\n",
    "predictions = classifier.predict_proba(X_test)\n",
    "false_positive_rate, recall, thresholds = roc_curve(y_test, predictions[:, 1])\n",
    "roc_auc = auc(false_positive_rate, recall)\n",
    "plt.title('Receiver Operating Characteristic')\n",
    "plt.plot(false_positive_rate, recall, 'b', label='AUC = %0.2f' % roc_auc)\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot([0, 1], [0, 1], 'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.0])\n",
    "plt.ylabel('Recall')\n",
    "plt.xlabel('Fall-out')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 网格搜索\n",
    "在第二章我们曾经提到过超参数，是需要手动调节的参数，模型无法学习。比如，在我们的垃圾短信分类模型中，超参数出现在TF-IDF中，用来移除太频繁和太稀缺单词的频率阈值，目前函数正则化的权重值。在scikit-learn里面，超参数是在模型建立时设置的。在前面的例子中，我们没有为LogisticRegression()设置参数，是因为用的都是默认值。但是有时候默认值不一定是最优的。网格搜索（Grid search）就是用来确定最优超参数的方法。其原理就是选取可能的参数不断运行模型获取最佳效果。网格搜索用的是穷举法，其缺点在于即使每个超参数的取值范围都很小，计算量也是巨大的。不过这是一个并行问题，参数与参数彼此独立，计算过程不需要同步，所有很多方法都可以解决这个问题。scikit-learn有GridSearchCV()函数解决这个问题："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', TfidfVectorizer(stop_words='english')),\n",
    "    ('clf', LogisticRegression())\n",
    "])\n",
    "parameters = {\n",
    "    'vect__max_df': (0.25, 0.5, 0.75),\n",
    "    'vect__stop_words': ('english', None),\n",
    "    'vect__max_features': (2500, 5000, 10000, None),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),\n",
    "    'vect__use_idf': (True, False),\n",
    "    'vect__norm': ('l1', 'l2'),\n",
    "    'clf__penalty': ('l1', 'l2'),\n",
    "    'clf__C': (0.01, 0.1, 1, 10),\n",
    "}\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)\n",
    "\n",
    "df = pd.read_csv('mlslpic/sms.csv')\n",
    "X, y, = df['message'], df['label']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "grid_search.fit(X_train, y_train)\n",
    "print('最佳效果：%0.3f' % grid_search.best_score_)\n",
    "print('最优参数组合：')\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print('\\t%s: %r' % (param_name, best_parameters[param_name]))\n",
    "predictions = grid_search.predict(X_test)\n",
    "print('准确率：', accuracy_score(y_test, predictions))\n",
    "print('精确率：', precision_score(y_test, predictions))\n",
    "print('召回率：', recall_score(y_test, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GridSearchCV()函数的参数有待评估模型pipeline，超参数词典parameters和效果评价指标scoring。n_jobs是指并发进程最大数量，设置为-1表示使用所有CPU核心进程。在Python3.4中，可以写一个Python的脚本，让fit()函数可以在main()函数里调用，也可以在Python自带命令行,IPython命令行和IPython Notebook运行。经过网格计算后的超参数在训练集中取得了很好的效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多类分类\n",
    "现实中有很多问题不只是分成两类，许多问题都需要分成多个类，成为多类分类问题（Multi-class classification）。比如听到一首歌的样曲之后，可以将其归入某一种音乐风格。这类风格就有许多种。scikit-learn用one-vs.-all或one-vs.-the-rest方法实现多类分类，就是把多类中的每个类都作为二元分类处理。分类器预测样本不同类型，将具有最大置信水平的类型作为样本类型。LogisticRegression()通过one-vs.-all策略支持多类分类。下面，我们用它来分析一个多类分类问题。\n",
    "\n",
    "假设你想看电影，而你又非常讨厌看太次的电影。所以有品位的你可以在看每部电影之前都会看一堆影评，不过你更讨厌看影评。那么下面我们就用好影评来给电影分类。\n",
    "\n",
    "本例中，我们利用烂番茄（Rotten Tomatoes）网站影评短语数据对电影进行评价。每个影评可以归入下面5个类项：不给力（negative），不太给力（somewhat negative），中等（neutral），有点给力（somewhat positive）, 给力（positive）。解释变量不会总是直白的语言，因为影评内容千差万别，有讽刺的，否定的，以及其他语义的表述，语义并不直白，这些都会让分类充满挑战。数据集可以从kaggle上下载。首先，我们还是用Pandas简单探索一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import zipfile\n",
    "# 压缩节省空间\n",
    "z = zipfile.ZipFile('mlslpic/train.zip')\n",
    "df = pd.read_csv(z.open(z.namelist()[0]), header=0, delimiter='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "In [37]:\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model.logistic import LogisticRegression\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.metrics import classification_report, accuracy_score, confusion_matrix\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "import zipfile\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('vect', TfidfVectorizer(stop_words='english')),\n",
    "    ('clf', LogisticRegression())\n",
    "])\n",
    "parameters = {\n",
    "    'vect__max_df': (0.25, 0.5),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),\n",
    "    'vect__use_idf': (True, False),\n",
    "    'clf__C': (0.1, 1, 10),\n",
    "}\n",
    "\n",
    "z = zipfile.ZipFile('mlslpic/train.zip')\n",
    "df = pd.read_csv(z.open(z.namelist()[0]), header=0, delimiter='\\t')\n",
    "X, y = df['Phrase'], df['Sentiment'].as_matrix()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5)\n",
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=3, verbose=1, scoring='accuracy')\n",
    "grid_search.fit(X_train, y_train)\n",
    "print('最佳效果：%0.3f' % grid_search.best_score_)\n",
    "print('最优参数组合：')\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print('\\t%s: %r' % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions = grid_search.predict(X_test)\n",
    "print('准确率：', accuracy_score(y_test, predictions))\n",
    "print('混淆矩阵：', confusion_matrix(y_test, predictions))\n",
    "print('分类报告：', classification_report(y_test, predictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.04   0.01   0.04 ]\n",
      " [ 0.01   0.01  -0.005]\n",
      " [ 0.04  -0.005  0.07 ]]\n"
     ]
    }
   ],
   "source": [
    "# np.cov() 求解一个矩阵的协方差矩阵\n",
    "print(np.cov(np.array(X).T))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "特征值：[-0.99999998 -1.00000002],\n",
      "特征向量：[[ 0.70710678  0.70710678]\n",
      " [ 0.70710678  0.70710678]]\n"
     ]
    }
   ],
   "source": [
    "## 求解一个矩阵的特征向量 特征值\n",
    "import numpy as np\n",
    "w,v = np.linalg.eig(np.array([[1,-2],[2,-3]]))\n",
    "print(\"特征值：{},\\n特征向量：{}\".format(w,v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  感知器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.85      0.92      0.89       396\n",
      "          1       0.85      0.81      0.83       397\n",
      "          2       0.89      0.86      0.87       399\n",
      "\n",
      "avg / total       0.86      0.86      0.86      1192\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/zp/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/stochastic_gradient.py:117: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.metrics import f1_score, classification_report\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import Perceptron\n",
    "categories = ['rec.sport.hockey', 'rec.sport.baseball', 'rec.autos'] \n",
    "newsgroups_train = fetch_20newsgroups(subset='train', \n",
    "                                      categories=categories,\n",
    "                                      remove=('headers', 'footers', 'quotes'))\n",
    "newsgroups_test = fetch_20newsgroups(subset='test', \n",
    "                                     categories=categories , \n",
    "                                     remove=('headers', 'footers', 'quotes'))\n",
    "vectorizer = TfidfVectorizer()\n",
    "X_train = vectorizer.fit_transform(newsgroups_train.data) \n",
    "X_test = vectorizer.transform(newsgroups_test.data)\n",
    "classifier = Perceptron(n_iter=100, eta0=0.1) \n",
    "classifier.fit(X_train, newsgroups_train.target) \n",
    "predictions = classifier.predict(X_test) \n",
    "print(classification_report(newsgroups_test.target, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##   SVM\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import fetch_mldata \n",
    "import matplotlib.cm as cm\n",
    "digits = fetch_mldata('MNIST original', data_home='data/mnist').data\n",
    "counter = 1\n",
    "for i in range(1, 4):\n",
    "    for j in range(1, 6):\n",
    "        plt.subplot(3, 5, counter)\n",
    "        plt.imshow(digits[(i - 1) * 8000 + j].reshape((28, 28)), cmap=cm.Greys_r) \n",
    "        plt.axis('off')\n",
    "        counter += 1\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}