Chapter 17 - Support Vector Machines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 17\n",
    "---\n",
    "# Support Vector Machines\n",
    "Formally, a hyperplane is an n-1 subspace in an n-dimensional space.\n",
    "\n",
    "Support vector machine classify data by finding the hyperplane that maximizes the margin between the classes in the training data. \n",
    "\n",
    "## 17.1 Training a Linear Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "features = iris.data[:100, :2]\n",
    "target = iris.target[:100]\n",
    "\n",
    "scaler = StandardScaler()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "svc = LinearSVC(C=1.0)\n",
    "\n",
    "model = svc.fit(features_standardized, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "scikit-learn's `LinearSVC` implements a simple SVC. To get an intuition behind what an SVC is doing, let us plot out the data and hyperplane. While SVCs work welll in high dimensions, in our solution we only loaded two features and took a subset of observations so that the data contains only two classes. This will let us visualize the model. Recall that SVC attempts to find the hyperplane--a line when we only have two dimensions--with the maximum margin between the classes. In the following code, we plot the two classes on a two-dimensional space, then draw the hyperplane."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD9CAYAAACoXlzKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztnXeYJFXVh9+7mSUsrMACghRI3rvkIEFEBAkFkkHEgOQcBKGQJCJQKBIUEBFBVIIImCglfAgoCCIi4YIEgRIkS5Sd2Xy/P06109NTPTvd093V3XPe59lnZ7prbp3u6fnVrXPP/R3jvUdRFEXpHkYVHYCiKIrSWFTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5jTNEBKIqijASCKPkQcCpwehqH7zXzXCrsiqIoTSSIEgPsDlwMTAbuBH7bzHNqKkZRFKVJBFGyDHAzcAPwErBeGodNFXXQGbuiKErDyWbp+wHfAcYDJwAXpHE4pxXnV2FXFEVpIEGUrAhcDnwK+CNwQBqHz7YyBhV2RVGUBhBEyWjgSOAsYC5wKHB5GofzWh2LCruiKMowCaJkKvAjYCMgAQ5N4/ClouJRYVcURamTIErGASciZYzvA/sA16Vx6IuMS4VdURSlDoIoWR+4EpgGXA8clcbhm8VGJaiwK4qi1EAQJROBrwPHAa8BO6Vx+JtCg6pAhV1RFGWIBFHyCeAKYCWk8uWEZu8irQcVdkVRlPkQRMkiwLnAIcBzwJZpHN5VbFTV0Z2niqIogxBESQg8ARyEbDhas51FHXTGriiKkksQJUsAFwKfQ4R9tzQOHyw2qqFhvC+0KkdRFKWtyOwA9gK+B0xCNhydk8bhrEIDqwGdsSuKomQEUbIscCmwI/AgsH8ah67YqGpHhV1RlBFPECWjgAOAbwNjga8A303jcG6hgdWJCruiKCOaIEpWAn4IbAHcBRyYxuFzhQY1TFTYFUUZkWSmXccAZwKzgQOBHxVtB9AIVNgVRRlxBFFiETuADZBuRoemcfhysVE1DhV2RVFGDJlp19eyf+8CnwVu6IZZejkq7IqijAiCKNkQmaVPBX4GHJvG4X+Kjao5qLAritLVZKZdZyL59FeAHdI4TIqNqrmopYCiNBFjzMeMMT82xvzWGPNlY8z4omMaSQRR8kngcaR88QfA1G4XddCdp4rSNIwxRyDGUROQSdR0ZGv65t77mUXG1u0EUTIJqUk/EPgn0nf0nmKjah0q7IrSBIwxiyK3/QtUPDUdONJ7f1XroxoZBFGyI3AZsBRwHvD1NA57i42qtWiOXVGaw6bALAYK+4LA7oAKe4PJTLu+i1S6PI40wHio2KiKQYVdUZrD++SvYXng7RbH0tVkpl17I6K+MNJ/9FudZNrVaFTYFaU53IeI+0KAKXu8F/h+IRF1IUGULIe8nyHwAGLa9WSxURWPCruiNAHv/TxjzDbA7cgs0gPjgJO9938uNLguIDPtOgj4FlCyBri4U027Go0unipKEzHGjEby7ZOAe7337xQcUscTRMnKSN/RzYE7gYPSOHy+2KjaCxV2RVE6giBKxiD16GcAM7Ovr+o2O4BGoKkYRVHaniBK1gJ+BKwH/Ao4PI3DV4qNqn1RYVcUpW0JomQ8cAoQIdVEewI36ix9cFTYFUVpS4Io2RiZpa8O/AT4ShqHbxUbVWegwq4oFRhjpgC7AROBxHv/j4JDGlEEUbIQ8E3gKOAlYLs0Dm8tNqrOQhdPFaUMY8wuwDVIeeIYYC5wsff+hEIDGyEEUbI1cDkQAJcAJ6Vx+N9Cg+pAVNgVJcMYMwnxd5lY8dR0YFvv/b2tj2pkEETJYoivy37AM4hp15+KjapzUdteReljG2SGXskCwD4tjmXEEETJLsCTwJeAc4C1VNSHh+bYFaUPM8jj1Z5T6iSIkinA94A9gEeAMI3Dh4uNqjtQYVeUPm4j/2+iB7iuxbF0LZlp1xeAC5G019eA89I4nF1oYF2ECrvS1hhjDLA+sAbwNPAX36SFIe/9u8aY/ZESu1HI38dM4MfAH5txzpFGECXLI17p2wJ/Rky7nio2qu5DF0+VtsUYsxBwK7A2UqVikA5EW3vv32/ieZdDNsKUyh01PTBMMtOuQ4EY+T2eBFySxuG8QgPrUlTYlbbFGHMpUiVR3id0JnCN937/YqJSaiWIklUR067NELfLg9M4TAsNqstRYVfaFmPMfxE/80pmABOblZJRGkMQJWOB44HTkXWKY4GfqB1A89Ecu9LOjKvy+NiWRqHUTBAl6yBrFesANwFHpHH4WrFRjRxU2JV25v+QRbby/RbzgLt0tt6eBFEyATgNOAH4D7B7Goc3FRvVyENTMUrbYoxZEXgQ2SA0EWkrNwP4mPf+mSJjazTGmMmIJe3rwOOdeOEKomRTZJa+KtKs+/g0DrW/awGosCttjTFmMWBfYF3gUeBK731XiYUx5jSkSmQmchf9T2A77/2rhQY2RIIoWRg4GzgceBHpaHR7sVGNbFTYFaVAjDGfAa4FFix7eA7wsPd+o2KiGjpBlGwD/AD4CLKL9OQ0Dj8oNipFc+yKUizH0l/UQf4upxljVvDev1BATPMliJLJwPmIv8tTwGZpHGqT7jZBTcAUpVg+VOXx2cBirQxkqARRshti2rUPcBawjop6e6EzdkUplt8Aq9B/E1YJ1+JYBiWIkqWBi4FdgYeBbdM4fKTYqJQ8NMeujCiy7kibAo94759vg3gmIyK5JFL9Mw+p/DnIe39NkbGVyEy7vgRcgMR4OvCdNA7nFBqYUhUVdmVEkJmJ3QV8ouzhfwJre++nFxOVkDX4OATYDvg3cJH3/q9FxlQiiJIA6Wi0NfAn4MA0Dp8uNChlvqiwKyMCY8yPkVlnJY9579dqcThtTxAlo4HDkMYXHjgRuExNuzoDFXZlRGCMmUV1K4KJ3vveVsbTzgRRsjpi2rUJ4q55cBqHLxYblVILuniqjBQG85dZEvhXqwJpVzLTrq8iOfQPgC8CP1PTrs5DZ+zKiMAY8yaweM5Tc4GxnbiFv5EEUbIeYgewFvAL4Mg0Dl8vNiqlXnTGrhSKMWZVYK73/p9NPtUhwI05j5/dbFE3xkwEJgDvtNsFJIiSUpXL8cAbwC5pHP6q2KiU4aIblJRCMMZ8yRgzF9m1+KwxZrYxZvdmnc97fxOwPfACsvnnDeBg7/1pzTqnMWZRY8wvgHeAV4GnjDGbNet8tRJEyeaI/86JSPu/NVTUuwNNxSgtxxizPJBWeXpx7/1bLQynaRhj7kP6tZb7yk8H1vLeP1dMVBBEySJIi7pDkQvdgWkc3llUPErj0Rm7UgQ/HeS5q1sWRRMxxkxDerVWNgsZBxzR+oiEIEq2R3a0HgJcCExTUe8+NMeuFMFHBnnuoy2LormsgKR8KhkLrN7iWAiiZHFk5+jnEZ+XTdI4fKDVcSitQYVdKYK/ActXea5bzKQeJd//ZQZwX6uCyOwA9kA8XhYDvgGcncbhzFbFoLQezbErLSerEvkAMBVPzQPGe++7woPEGHM1sDvS/QmktPJtYA3v/X+aff4gSpYBLgV2Ah4C9k/j8LFmn1cpHs2xKy3He98DbIF0DCrRC2xUj6gbYyYbY35qjOk1xsw0xvzSGPPhnON2MMY8bYyZa4x52RhzcOYh0yz2Q0oJ/4UI+nXA+s0W9SBKTBAl+yMpl22QTUcbq6iPHHTGrrQcY8x44BlgGfrSgbORSpk1ahF3Y8wo4HFgJfoWKucg5YwrlawCjDHbADfTN3sG6AFO9t5fWPeLaTOCKFkR+CGwJXAPcEAah83eI6C0GTpjV4pgVyTfW77GMxZYCqk1r4WtgeXoX30yBlgE2LPssbPoL+pk359mjBld4znbjiBKRgdRcixS8bIBcDCwpYr6yEQXT5UiWB1YOOfxBbLnflPDWGuQv0i5EDCt7PuVq/z8gshF4J0aztlWBFEyFbED2AhIgEPSOPx3sVEpRaLCrhTBU8ji6UIVj/cC/6hxrH8gufrKevEP6N+B6FlgvZyf7wHer/GcbUEQJeOACDgFeQ37ANepaZeiOXal5RhjJiA59qXpn2P/F7B6jTn20UiO/aP0iftc4HVg5WyhFmPMtsBN9E/HTAdO9d5fUP+rKYYgSjZAZunTkEXZo9M4fLPYqJR2QXPsHYoRvmiMedIY85Yx5jfGmKlFxwVgjNneGPNQFtc9xphNy5/33s9A0ga/QQR9FiK6m9RaFeO9nwt8HHEknIksnN6CVNj0lB13K/BZZObuEe+WryK7LzuGIEomBlHybeABYDLwmTQOP6eirpSjM/YOxRhzCnIbvmD2kEdmoOt5758pMK69gCvpPzPuBbb13v8x53gD0CjXQ2OMmd9YQzmmHQmiZAuk4mUlpF3dCWkcvldoUEpbosLegRhjFkTK+SqrPOYA13rv81rANZ1MpF8CBtSQAw967zdqcUhdQRAlk4BzkUqX55ESxruKjUppZzQV05l8FBHxSsYAG7c4lnImAlOqPGdbGUi3EETJDsATwIHAdxDTLhV1ZVC0KqYzeZn8Ej8PFFm33ItUmSyS89wrLY6lowmiZAngImBvpLpn1zQOHyw2KqVT0Bl7B5L5ld+ICGk5vcDZrY9I8N7PA76NiHs5PcDXWx5QB5LZAeyN2AHsjlgSrKeirtSCCnvnsj/iaz4DqQZ5Bfi89/7eQqOSC8sdyN0DiLHXT7z315QfZIxZ2Bjz68zfpccY8wtjzAKVgxlj1jTG/DzzeLnJGLNOM4M3xowyxuxrjHnQGOOMMacZY/I2UzWcIEqWRSqFrkVy6eukcfiNNA5nteL8Svegi6cdTlYTvjDwn3ao9DDGfA3Zvl/J3t7767NjRgNvAZMqjnkTmFJ6HcaYjYH/Q/qFjkIuEjOA7fIqbBoU/5WIFUGp2mgG8BxSbdQUq9sgSkYBByB3O2OBk4HvpnE4txnnU7ofFXaloRhjZiHiVMk73vvJ2TERcE6VIQ713l+WHfcg4ntSyaPe+7UbEW85xpiVgceQC0k5HwCHe+9/0uhzBlGyElLCuAXwB6RN3fONPo8ystBUjNIwjDEfIl/UQUy/Smw7yDA7ln1dLe2yZpPsdjcmv9poIWCrRp4oiJIxQZQch1xI1kWqXrZSUVcagVbFKI1kMM+V8rTCYBUyL5V9/S6weM4x7zUp7fQafWsD5cxC7A4aQhAl0xA7gA2QnPphaRy+3KjxFUVn7ErD8N7PRtre5XFT2ddfqzYEcGrZ998hv8LmoroCnD93Au8hufxy5iBCPCyCKBkfRMkZwMNAgFgc7KyirjQazbF3MFmFyNFIc+jbgcu89+/WOdaOyO7GpZF+nYd475+qY5wFgKcRj/QSjwNrlc+yjTFfRnLLJS/0OcDnvPe/KDtmVHbMF8vGuh7YN/OIqTW2aUirOAv8GzjOe397xTErAr9EbH7nIjYNX/De31Hr+coJomQj5OIwFfgZcGwahw3vpOScWxDp3LQzsjv5Ymtty3qsKu2BCnuHYozZDfgJslFpNFLD/hawTq2t14wxJwJxxcMe2NB7/1CNY+0PXJEz1rYlEc3y479Acu2l6pPpwI3e+33LxpqGNH4ejzg3zkZKOzf33v+9xrg+AdzFwD6rR3jvL8k5fgVkJ+0/svr8ugiiZEGkgfQxSArqkDQOk3rHG4xM1B9E7gYmIu97L/BVa+2lzTin0p6osHcgxpgxiC3t5IqnZgIXee9PrGEsgwhmXhehZ7z3q9YYWy8Dq0oAXvPeL50dsxlwK32iXqIH2Kwk2saYu4BPMFCM/+y935QaMMa8BCyb89Qs733eLt5hE0TJlsgdx4rAZcCJaRw2zfvdOXc0so+g0kOoF5hirf1vs86ttBe6eNqZrMrAxhIgM9udgCELO7A2+aIO4iI4ZIwxi5Ev6iBt70psxUDxAfk8bgWUZuObMVDUAT5Wh0NjnjEZwDhjzHLe+5eqPF8zQZQsitSkH4BYPGyRxuE9jRp/EHYh/32dBWyIrCEoIwBdPO1M3qX6RfmtGsd6Y5DnavJGZ+BCZznl6Yx3kI0/lcymf4u6D6qdp46qmMHSKXWtS+QRRMlnENOu/YBvAWu2SNRBNnjlvS+j6eDWf0rtqLB3IN77l4G/IkJYznRqrBjJxnqtytM31jjWTKQzUh7ls8XryRcgX3HOH5Dvh1NPhcrvqzz+gvd+2CmKIEqWDKLkeuDXyMV1ozQOT0zjsDL+ZnIxAy+u85CmIjWtSSidjebYOxRjzBRErFZBZtYTgPOBk2udzRpjAqQSZhFEXA3iKLh2rdUnxpglkD6kHyp7OAVWK9+Sb4zZBhHxUkppBvAZ7/09ZceMA64BdkDWD8Yj1T97ZV2YaolrLDKTXpm+1/gOMNV7/2otY5UTRIlBeo1ehFg7nAmcW5S/S5ZnPwdJv4xGLtrbWGubtvHJObcU4l20CnAvcK21dnqzzqfMHxX2DscYsyZSoviw935Y7dGMMSGSc7/Fe/9onWNMRSpZJiBCPANJdayf3R2Ujvse8GX6qjdmAN/13p+UM2YArIYs5g5LoIwx6wOfBh6qLHWslSBKPoIsim6HtKrbP43DJ4czZiNwzk1CWg++DfzNWtu0P3Ln3AbI3dhY5Hc+HfgPsIG1Vtv1FYQKu9JQjDH3I6JSvug5Byll3Ds7Zl3gT+RXb6xbT/18K8lMuw5GcuijgJOAS0aiaZdz7glgjYqHZwFXWGsPLyAkBa2KURpI5jS5AQMrWcYAYdn3O5DfKGQUsD3QtsIeRMkqSJ3+xxHnyYPSOHyh2KiKwTm3BNLNq5JxwK6ACntBqLArjWQusliXVz5ZnnPuRWbxlcfNZeBiaVsQRMkY4DjgDCTG/YGr0jgcybe8s8gvR4X8qielRWhVjNIwMq+YX9NfxEH+yK8s+/7n5JcfGvp7yrQFQZSsBfwF2Z37O2CNNA6vHOGijrX2PeCPDCyL7UUqmpSC0Bl7i8msbT+LVI3cBdxbr1OhMWYSsBeyeHof8Ie87e/GmC8AXwL+C3y93oXRIXIIsoFqZWTi4JGFxdNLB3jvXzTGHIjsyiyJwmjEk2WwuvqWEkTJBOAUZMPXW8DuaRwWduFxzk1GPjtLIJ+dPzVzYXSIfBG4G/kMGuR3fidi4NYP59w0xJZ5FnCDtfbF1oXZGpxzH6avUUtirS2kzFQXT1tI5ldyCyJi45Ga4z8Au9ZRVrge8gc0BlmEnI44K25TKivM7AKeQsrQyjnLe3/KMF7KYHEtD9wPLAoskMX1b2AT7/3bFccuhvjFzANu9d6/14yY6iGIkk2QXPrqwNXAV9I4fHvwn2oezrmPI3cLo5D3tQcR1J2ttbVuJGsozrlRiPVDADxkrX0855gYOBLJv5dSdodaa69uYahNxTm3B/JZMUiV0EzgKuDIVl+AVdhbRNYO7jUG+otPR7rzDPkDngn2c8AKFU/1AKd678/PjjsNyQnnseRwyyOrxHYH8En6589nAVd77w9q9PkaTRAlCyGt/Y5EvOEPTuPw1iJjcs6NRgzElqx4ajpwlLX2yoE/1T445zZE7jAqq6BmAB/phrLIrMT0VeSiW850YEdr7V2tjEdz7K1jffL9XRZE6rlrYWVgSs7jE4F9y77fr8rPe+CIGs85X4wx45EWb5WLouOQ29O2JoiSrRGL4aOASwBbtKhnrMtAwYD6PjtFsBf5HkJz6F8t1cl8moE7wUH+Jj/X4lg0x95CSrsd86jVFnaw2yxf5etaxqiXVp+vIQRRshiSE/4y4iX/8TQO7y02qn4M9t7VbSncQjryc1EjbfUadcbeOv5GfinfdPpXjAyFfyK3fZUfmJ6KsSp90UsYmtCFyHs/C1kzqFwvmIn4w7QdQZTsAjyJLAKeA6zdZqIO0nEpb4t+PZ+dIrie/PLHMciaUzdwO/kT5R6ksUpL0Rx7CzHGbIr4u4xCFk9nIr7ke9WxeLoWsng2AUl1zEQqY8JMYEu5+MeRrj3lnOa9P7NivNHA1sDywF+99w9XOe+nkfZ1PcBR3vunK55fFlk8nYTchvYALyI+6w1zURwuQZQsBXwP2B14BLEDyH3N7YBzbmPgNvp/dm4H9rDWtv2OV+fcN4DjkTTdXGRysb+19tomntMgVVqbAg8BF9W7iJmNtQHSYP154E5r7byKY3YGrkUmXGOQ1/l94HhdPO1yjDGLImKyOLKg9GA95Y7GmJWRGuJJ9H2Ifg3sU3mRMMbsgaQZ3gW+UbllPxPjPyElmKX8+D3AzqWLRHbcY8C0ilB+6r0vb11XMu/aCVkLeBSpeGkL8clMu74AXIhceM4AzkvjMC8/2lZkC3R7IJ+du4G/tEG545Bxzq2KlDvOBG6y1g7W1Hy451oaeJb+zVxmAKtba9Max5oAJPRZZcxFGt1sbq19teLYKcjfd6nc8Yl6X8NwUGHvUIwxjyMeHeXptOnAsd77H9Y41j3AJvS/lewFzvTen5MdczzSPCKP1Spn7u1IECXLIxtntgH+jMzS29a+QKmfKh42AKm1trKabH5jnYnsOi5fwJ6DzNq3rT/K5qHC3oFk/TifIL9S4mHv/Xo1jDUZydfnVey84L1fMTvuFWQTSh63ee/b8gMO/zPtOhTZOWqACLg0jcNOWHhU6sA5N5iwjarlTsc59zKwTM5Ts4HF2tGiWKtiOpPxVK+GqNaarhpjqb5qP77iuGrktWNrC4IoWRVZRN4MyVEfnMbhv4qNSimYUQxc4B+MvElPibbUUK2K6UyeAfJ2afYiizdDxnv/OrIYVMks4Iay72/IOabEabWcsxUEUTI2iJKTkBz/VKS+fzsV9RFDtfz923UsNv+SgTXqHng888tpO9ryatOpZF2NNkR2mD6UtyiaVap8DFmofMB7/59az+O9n2eM2RvZYj4amaV/gOxGvbCO0L+AlCmOp8/q4GXgG2XHHAF8HumyVM6j3vu7Kwc0xqyBNMN2w22OUStBlKyDtM9bB+nSdGQah68BOOcCYE3geWutG855nHPrIYvETwI/z7u9z7bbb4Isct9nra27MiirzNgI8Yq531pb82enYqx1kRTDQ5WLgO1OtqC5OSKwf7TWzqw4ZBekOqt88uqRhc1aORmpGFscWAj5+5hNG28O0xx7A8jEOkZ2LM5ExPYl4NPe+3+XHbcicAeyNXwuIqLf9N6fVed5l0E+XMshFTY3Zw6LtY6zEjKzLU+pPAusXl7Nkl1MrqZvQjAd2Np7/0DZMQsj1TkbIR/+8cBvkWqdplaeZKZdpwEnIF18Dkvj8GYA59yYLPZdkd/RWKQ+fIdaZ13ZFv/SnUBp41kvsK619qmy49ZA0j+TkNTZeOAEa+33an1tzrkVkM/OFOSzMwE4x1pbzTJisLGWRkolg7Kxvg98pROqbJxz29HnEFra9Le7tfaOsmPGA79CBNkgv6d7ge2ttYM1Xa92zgUQA7aPIZvYrrbW1to4vmWosDeArJzwKvqXVs0B/u693zA7xiAzu1UYWMmym/f+thaFOwBjzGvkWxTc4L3fKzumVLpYuWD7HrC09743O+7HyB9AeX6+FzjXe1+zCA2VIEo2Q2bpqyC/i+PSOHyn9Lxz7kRE9MsvXjOBX1pr967lXM65q+hv3VDibWvth7JjRiH1+8vQf8dxD7CVtfb+Gs5X6kG7Kv3tGqYDe1prf1dj/H9CBKr8jn06cLC19ppaxmo1WTnh8wxc1+lBfGfeyo47F/H7Kf+8zgCustYe1opYi0Rz7I3hKPqLOsgfjc36dQJYZGZd+Z4viHwAC8EYsxz5og6wc9nX+5KfuhtF5vdhjBnDQFEH+eM6dFiBViGIkoWDKLkYqcMfB3w6jcP9ykU94wgGisF4YNdsdlcL1S4Ek51zJSfNTZC0VaWNxARk00wtrIFsHKv04Kn5s5PZyq7PwN/lgsDRNcZVBHtRXbf2KPv6IAZOQiYA+2YXyq5Ghb0xLFbl8TnIbTjZ/9UWbSY3PKKhM9i5y//4J5NfGTOKvtc4lurrNgvVHtrgBFGyDTKTPQz4LjAtjcM7qhxe7fylnZy1MNjaVMmBsZR+yTtfpcPn/GjkZ2dhBjbGKLFojWMVwSTyq1TG0vc5hOqVWuMZAbrX9S+wRdxMvhfGHKTeHMQrJu/97kUW+IriMar/oT9b9vUtyAJtJaOR3p9k6ZgBXtyIwFUT3JoJomRyECU/RuwYeoDN0jg8Oo3DvPhK3Ea+OD5jrX2/xhCerPL4XMTWgez/PAGaTu1doh4m30CuF/hFjWM9S75n0Uyk+qPduZ38+Gcjv+MSdzOwjNcDD3SCBcNwUWFvDBcg5VWlRZm52dcHeO/nwP9E74js8dJMrgd4Abi8pdGWkVXunJjz1Dz6pxx+jyw+lTZj+OzrC7335SWEByEXgJIVwQwkD398I+INomR34B/APohv+jppHP55CD8aIZYKpQvwrCzOA+sIY2/yZ+OnlBYfs+qXE+n/+56exV5THttaOwO5K+mh7+LUA/yLnBZ0zrnxzrn1nHMr5ow1F+nXWj5WL7JF/ls5Y411zq1blmKqG+fcaOfc2s651YeRDnkQuQCVbwqaDlxvrX2k7LGjgfeRCxb0/b67Pr8OunjaMLJqkP2QjkAvApd47x+rOGYKMqsomXLNAY7x3hfeH9IYsxVwHvBhxDDp8MoyxSyHvjviL90DXOG9/7+csZZHcr8WaYv3/axevm6CKFka8UjfBZnB7p/G4SOD/1R/nHNLILn+TZBZ98XW2rpKMbPFuePoa//3ELIo+t+yY5ZD7mY+mh0zGzG+uq7Oc64LHI7sAL4FqcyYXnHM55H3CSQ98TjSZenVsmMWpc/7xCMXnq9ba8+pGGsnZCF6DHJn9jywUz3vmXPuU8B1SJ57FDIR2tlaW+3uZ7CxNkYqrUqpl3eB7ay1D1UctzTyfq2PGL1dYq19qdbzdSIq7C3EGPMgsDb9c9U9SFnkffk/NbLJTLv2Bc5HROF04Pw0DgtrB+ec2xOxyy1fMJ/vP4jbAAAUV0lEQVQJ3GKt3T07xiCz85Xov+jZA2xirW1439kqnYrmAM5au07ZcQnwKfqvLUwHPmet/U12zGpI+rB8rHnIpOWjlc6G84nrw8imuvKxPFKSuqy1trL5+WBjLQakDNxP8R6wXPmFdSSjqZgWYYxZFZmpVy5ALgAc2/qI2p8gSlZAcqpXIjPPtdI4/FaRop5xAgOroMYDO2SzYRCL1w8zsJJlPDKLbAZHM9BSYgywctZIGufckgwUdZDX89Wy7w9m4BrBKGRj3WY1xrUvA98Hk8W6fY1j7UX+4vVo6tt81JXoztPWMYX81lkGWLbFsbQ1QZSMRtYjzkZmiYcBP2gj065q5aFzkCqVd+nbSFTJaKTstRksS/5kbU4Wz+NIRc4s8iuBlir7ejny9cFXHDcUPlzlfKOp/l5WYwr55ncT6oira9EZe+t4hPwqiRnIwqQCBFGyBlKTfiHiCT81jcPvt5GoA9xJvmj3IguaAH8hX8x6aN7v+/fkV4yMR9IqIN238t7L2fSvXLqV/K5N4xDL41q4k/yKKoP8rmvhj1XimpE9p6DC3jK89+8jOx/LP5QzgbeQTj4jmsy06xTg78gOyy8AYRqHLxYbWS5fRyouSndgHhHsI0qldNbaN5BqqfLf9wxk0bBZ7ewuA96grxKE7PzftNa+k8U1C0nZ9NBXDjgLeT3l1hbXIBep8gvFdOAya+2/qY1fI4vV5Vv5pwM35i2eOucmOOeWdc7lTYTuRjxgysfqQSYBtV5wys/5oSxN1RXo4mmLMcZsB3wF2chyC3BBPUZg3UQQJeshYrcm4gFyVBqHbxQb1eBkFS8nAJ9AFvPOtdbeV3HMSki1TKl6Yx5wmrW2Lm+gIcY1GTgGMSf7D3Chtfa3FcdMQjZ2lacAr7TW7l9x3MJISmwvZHHyEuAX9fjJZF4rhyJlqjORi9DPyhdhMxuGb9K3A3YuYkT3nfJzOufGAgcgPkkesZK40lpb89pLZgp3DVI545FF3i80Y3G7laiwK4URRMkCSJXL8chM89A0Dn9dbFSNwzn3AQMXWQG2ttYOKBNtFc65FxADsEq+Yq29oMXh/A/n3KlI7X/5e9YDHGWt/VETzjcW2UeyFH2Lux65e1mhdJfTiWgqRimEIEo+jqw7nIjUSq/RZaK+B/miDrJfoBCyu4igytOntDCUfmTloccz8D2bSPPi2h4pmyyv2DFI5do+TTpnS9CqGKWlBFGyCHAOUunyArBVGod3FhtVU1iLPkvfSj7c4ljKyesDWqKyNryVjKe6n0+zql2WJ7+gYSKyqaxj0Rm70jKCKNkeye0egiwsTutSUQf4DfmiDpJ3L4rBqlCK7C41E+lhkMdjVR4fLg+R75P0AbJA27GosCtNJ4iSxYMo+Smyjf2/wCZpHH4ljcMhNwHOmlu0HZn/yQABt9Y+iDRkqGQeVTYoZb4sDbGUHSSud5DfQx612gk3jGxx9Bj6V7uUqo0a4jOUw/2I90x55c9MZHftr5p0zpagi6dK08jsAPYALkasjc8CzknjsLKNWVWcc19CNiotg7QcPKUZC2m14pzbFrEKXhnZkPQd4OyKKo/RSA/anZG057PA3tbav1eMdQTwbfp2jd4PfMpam1eTPr+41ke6Ia2HCNaVwFczI7Hy4y5EDNvGA68CB1prC99P4ZzbCjgDsWJ4DDjVWvvA4D81rPONR6qb9kNy69cDZ7ZrL9OhosKuNIUgSpYBLkXK7v6KmHblWfpWxTm3D+J8We4x0gMcaa1tVi34fHHObYaYuVXGdYm19oQax9qD/Ebhz1hrV61xrI8iC9Lluepe4HZr7c75P6V0I5qKURpKECUmiJIDkA0p2yD+I5vUKuoZZzGwYcJE4MzhRTlsvk5+XEc456o1eKhGtQqZVeqwyj2G/O5V22T12soIQatilIYRRMmKwA+BLZGdgAekcfjPYQxZzUNnaeecKbDx8mpVHp+LWOo+V8NYg3mlbIRsmBkqlc6hJWYiKaO0hrGUDkaFXRk2mWnXUcgMew7iDHhFA/xdXkByrZW8WKCog+R+K5tUk33/co1jvQwMaIiRUauPyoPAhgws4RsPPFXjWEoHo6kYZVgEUTIVaQN3PvAHZKPR5Q0y7Sp1ICqnBzipAWMPh9MZaLY1HTivcpFyCBxV5fHHrLVpjWNdhMzOyy96PcAvR0qDCUXQxVOlLoIoGYe0mzsF8RE5Crg+jcOGfqCcc7sgG5pWRFIJJ1tra+3zWcv5JiCv6QCkSuW3wInW2lcqjtscuZhNQzxZYqQjU7mnyRik4uJwZEHzduAEa+0LFWOdhHiilO6gnwDWqqc3p3POInsENkdKSy9FqjzyLKOVLkWFXamZIEo2QMroLNLu7Og0Dt8sNqrG4Jy7Dfg4fZ7fc4A3gdVqbXrtnLsO+Ax9C61zkYvg6pn7I865lZFWfwvSl9qpq8JGUUpoKkYZMkGUTAyi5Dykj+liwGfSOPxcF4n62kh3oPJGDmOQrfZfrHGsFZD69fIqmdHZ94eWPRZl5yvP108EjizrxqQoNaHCrgyJIEo+iSwaHodUvkxN4/C3g/9Ux7EW+U0oFkQWJWthGv190UtMADYu+34jBraNI/vZjvYrUYpDq2KUQQmiZBLwLWSX4nPAlmkc3lVsVE2jWpliL9KYutax8koPZyM59BJPAaszcJI1nureKYoyKDpjV6oSRMmOyEajA5CNNGt2saiDVPe8gHQUKuERMa7JxsBa+wTSjq5y1j4TsVgocQ7SWamcXuDmUh5eUWpFF0+VAQRRsgRSOrc34sa4fxqHDzbyHFlHnWOR3PU8xJP9u9baIfvINIOsPdptSFrGID4qe+R0R1oZaXW4GTKzPttae2vFMYsgVSm7I+mWfwAHVXqfOOe2QboTLY9cRK4Ejiv6vRgqzrlPAScjlUv3A2dYa7VuvkBU2JX/kZl27Y2YWy2CtCmL0zicNegP1khmjnUvIp6lhcpeRBS2KnLzkXPuZ8iiZ6nhw0zEznbtkilXJup/QxY5S/nxqh42We/OcdbavIbOpWMMsDDQ20mlic65vYEr6F/50wt8LLtrUQpAhV0BIIiSZRFXwB2AvyCz9Kb8YTrntkd6m1Y2VvgACK21hXSbd86thpQeLlDx1HREtK/KjrsOca2sXPR8B5jSScI8HLIepa8BS1Q8NQ/4rRqPFYcuno5wgigZBRyI2MaOQRptfzeNw5o3x9TARuR3y5kAfAwoRNiRype8170g8EkkXQSSfsmrZBkHLAc835To2o8pyF1GJaOATVoci1KGCvsIJoiSlZDSxS0QO4CD0jisxcCqXl5BZsGV/S17qd1rpZG8Qv/t+CVmIouqJV4m36BsNLILdaTw7iDPvdayKJQBaFXMCCSIkjFBlBwPPA6si8zYt2qRqIM0M5hDfxH1SDXKL1sUQx53AW8xcNY+B8kjlzibgR42M4Aba92d2slkaw4/Id8356zWR6SU0Bz7CCOIkmlI6d4GSF/Ow9I4HNIsOes28yXg84iwXY4YTNX8Icp2ef4cSV0YpO57T2vtk7WO1Uiccx9BFnGXyR7qQapifldx3OFIqWKpq/1NwAH1dD3qZLLPxGXAZ5GKHgOcbq09v9DARjgq7COEIErGA1/L/r0DHAncMFTTrqyS5S5khl9KoUwHrrbW5vbwHMKYBinxm2etfbGeMRqNc+4hpK1cOR8AS1aKdiZqAfBG1kt0xJLZH0wB/lWHw6XSYFTYRwBBlGyEzNKnAj8Djk3jsKZccOay+BMGLnr2Amtaa4fTUKMtyFreVfNAv8hae0wr41GUetHF0y4miJIFkTZyxyALfmEah78b/Keqsg35lSzzkMXXjhd2JMVUjR2Q91FR2h5dPO1Sgij5FLI4eixSnz51GKIO8Dr9t9qXmEv3VIK8Mshz3fIalRGAzti7jCBKFkVq0g8AngU+kcZhI+rCr0IaU5fjkYqR3zdg/Hbg20ij6sqWdyDNNxSlI9AcextijBkH7Inc/r8O/NB77+b3c0GU7ITMzpdETLvOSOOwYVUazrkdgZ8iwjcKWYTd0Vr7aMVxSyNukBbZxfqjTllcdM7tDNxI/w1I51lrKy9qSoZzbjmkz+2qyBrFj0dS2Wc7osLeZhhjJiB/HKsj1SdzkBTIgd77a/N+JoiSKYi/y57Ao4gdwN+aEV/me7IBUrf9d2vtvIrn10J2jo5DdpL2INUz67dL5ctgOOe+iszaJyAXrw+Au4GdKl+rAs65jYE7kLv/8cjv+x1gPWvt60XGNpJRYW8zjDGHIrPtiRVPfQAs6b3/3ww8M+3aB3FiXAg4A/h2GoeFeZU45x4E1qd/OmMucJO1dq9iohoazrmlkB2mEyqe+gDY21p7S+ujal+yctWngFUqnpoNXGmtPaT1USmgi6ftyF4MFHWQ6pP/dfEJouQjQIKkRp4G1k7j8OyCRX0CUudemaMeDWzf+ohq5lOIKFWyELBLi2PpBJZA9iFUMhZxyFQKQhdP249quclRwPTMtOsQ4NzssaOBS5ps2jVU5mb/8gyyOmFH5nTyvWLmAv9tcSydwEzyF5pB3kulIHTG3n58n4F/FB546yPH/+p9JN97CbLtfWoah812YhwymV3tzQwsi+xFzMbandvIF/ZZ9Dk7KhnW2vcQ87jKu5we5HOsFITO2NsM7/3vjTEXIfa5swHPqNE9y+x38U1m9JjHEJH8MnD1UO0AWsyhSBPmNZD00WjgHuAbRQY1FKy1vc65ELiFvknPOKSb0aPVf7J9yPLenwS2RgzNrrXWDlafP7+xNgFC5I7lWmvtvyoO+xJwJ2Kt4BFNuQW4sJ5zKo1BF0/bFGPMMsDHF9lw10mLbvHlg40x6yLOh4encfhqweENSiYI6wMrAY9ba+dbqtlOZB4wWyFrHX+w1r5VcEhDwjk3BvmMbIGsC8xE0ki7VbbtG8JYBrGQ2AV5H2ZnY+1rrb0h59iNkXz7w9bap4f3SpThosLepgRRMgHZFHMiMvM6Io3DG4uNSmlnnHOfR5wWK33u30NMzIbc4jC7c7megTYSPcBS1lpdc2hjNMfehgRRsgnwd6RB8DXAGirqyhD4IgNFHWSBc+Max9qbfG+gOUj1kNLGaI69jQiiZCGkQcGRSOf7bdM4vK3YqJQOYk6Vx80gzw02lie/6qXWsZQWozP2NiGIkq0R064jkaoXq6Ku1MiPyC8znAE8UONYVzOwSxSIZtxZ41hKi9EZe8EEUbIY8B2k0uVpYPM0Du8tNiqlQ7kZ2Qj2WaQaSaqqYGdrba0lsXcj+frDkFn73Oz/3UZal6hORBdPCySIkl2R2fkSwLeAb6RxqN1nlGHhnFsT2BJ4G2ldWPdCp3NuVWBbpNzxl51i5jbSUWEvgCBKlgIuBnYDHgH2S+Pw78VGpShKt6CpmBaSmXZ9EbgAqQ3+GnBekf4uiqJ0HyrsLSKIkuWBHyAt5u4DDkjj8Klio1IUpRtRYW8ymWnXYUCcPXQkcGkah+rtrShKU1BhbyJBlKwGXAFsihhMHZzGYaXXRlfinFsHWBl4zFqrdyaK0kJ08bQJBFEyFukPejpSV3wM8NM2Ne1qKM65SUgP1DWRErmxwP8Bu9eypV1RlPrRGXuDCaJkXWSjyNpI78wj0zh8rdioWsqlSLON8WWPbQWcmv1TFKXJ6Iy9QQRRsgBwGjJTfxNxYby52KhaS+YuOB2xuq3kDWvtlBaHpCgjEp2xN4AgSjZDZumrIA0ZjkvjcCRu5BhDfvckyG/3pyhKE1BhHwZBlCwMnAMcDqTAp9M4vKPQoArEWjvDOfcIA/uezgVq8gNXFKV+1ASsToIo2RZwSCnjRcC0kSzqZRwIfIA0eQDp+PQOkqJSFKUFaI69RoIo+RBwPrKD9B/A/mkc3l9sVO2Fc24ZpEXeNKQ36xWd0oVIUboBFfYhktkB7IaYdk1GUjBnpXE4c9AfVBRFaTGaYx8CQZQsjQj6LsDfkFx6RzQ3VhRl5KHCPgjZLP3LSOplPHACcEEah9pBRlGUtkWFvQpBlKwAXI5srvkjcGAah88UG5WiKMr8UWGvIIiS0cARwNlImd6hwOVq2qUoSqegwl5GECVrIKZdGyN+JwencfhSsVEpiqLUhgo7EETJOCR/firSAuzzwLUjwbRLUZTuY8QLexAl6yN2AGsCPweOSuPwjWKjUhRFqZ8RK+yZadcZwHHA68DOaRz+utioFEVRhs+IFPYgSj6B5NJXyv7/ahqH7xYblaIoSmMYUcIeRMkiwLnAIcDzwFZpHN5ZbFSKoiiNZcSYgAVRsj3wBHAQsuFoTRV1RVG6ka6fsQdRsjhwIbAP8CSwexqHfyk2KkVRlObRtSZgmR3AnsD3gEWRDUfnqGmXoijdTlfO2IMoWQb4PvAZ4K+Ite7jxUalKIrSGrpK2LNZ+v7AeUjfzeOBC9M4nFtoYIqiKC2ka4Q9iJIVgR8CWwJ3I6Zd/yw0KEVRlALoeGHPTLuOAs4CZgMHA1eoaZeiKCOVjhb2IEosYgewIXALcGgah/8uNipFUZRi6Uhhz0y7TgJOBt4DPgdcr6ZdiqIoHSjsQZRsiMzSLXAtcEwah28WG5WiKEr70FHCHkTJKYhx16vAjmkc3lJwSIqiKG1Hp1kKPIdUvkxVUVcURcmna3eeKoqijFQ6bcauKIqizAcVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMlTYFUVRugwVdkVRlC5DhV1RFKXLUGFXFEXpMv4fd1aU9iy4j6kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "((-2.75, 2.75, -3.543971732575235, 3.98520933763276), None)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "color = [\"black\" if c == 0 else \"lightgrey\" for c in target]\n",
    "plt.scatter(features_standardized[:,0], features_standardized[:,1], c=color)\n",
    "\n",
    "w = svc.coef_[0]\n",
    "a = -w[0] / w[1]\n",
    "xx = np.linspace(-2.5, 2.5)\n",
    "yy = a * xx - (svc.intercept_[0]) / w[1]\n",
    "\n",
    "plt.plot(xx, yy)\n",
    "plt.axis(\"off\"), plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 17.2 Handling Linearly Inseparable Classes Using Kernels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(0)\n",
    "\n",
    "features = np.random.randn(200, 2)\n",
    "\n",
    "target_xor = np.logical_xor(features[:, 0] > 0, features[:, 1] > 0)\n",
    "target = np.where(target_xor, 0, 1)\n",
    "\n",
    "svc = SVC(kernel=\"rbf\", random_state=0, gamma=1, C=1)\n",
    "\n",
    "model = svc.fit(features, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "For reasons best learned elsewhere, a support vector classifier can be represented as:\n",
    "$$\n",
    "f(x) = \\beta_0 + \\sum_{i\\in s}^{p}{\\alpha_i K(x_i, x_{i'})}\n",
    "$$\n",
    "where \\beta is the bias, S is the set of all support vector observations, \\alpha are the model parameters to be learned, and $(x_i, x_{i'})$ are pairs of two support vector observations, $x_i$ and $x_{i;}$. Don't worry if you don't understand kernel functions. For our purposed, just realized that K\n",
    "\n",
    "1. Determines the type of hyperplane used to separate our classes and\n",
    "2. we create different hyperplanes by using different kernels.\n",
    "\n",
    "For example if we wanted the basic linear hyperplane used to separate our classes like the one we used in Recipe 17.1, we can use the linear kernel:\n",
    "\n",
    "$$\n",
    "K(x_i, x_{i'}) = \\sum_{j=1}^{p}{x_{ij}x_{i'j}}\n",
    "$$\n",
    "where p is the number of features. However, if we wanted a nonlinear decision boundary, we swap the lienar kernel with a polynomial kernel:\n",
    "$$\n",
    "K(x_i, x_{i'}) = (1 + \\sum_{j=1}^{p}{x_{ij}x_{i'j}})^2\n",
    "$$\n",
    "where d is the degree of the polynomial kernel function. Alternatively, we can use oen of the most common kernels in support vector machines, teh radial basis funciton kernel:\n",
    "\n",
    "$$\n",
    "K(x_i, x_{i'}) = e^{(-\\gamma \\sum_{j=1}^{p}{(x_{ij}x_{i'j})^2})}\n",
    "$$\n",
    "where \\gamma is a hyperparameter and must be greater than zero. The main point of the preceding explanation is that if we have lienarly inseparable data we can swap out a lienar kernel with an alternative kernel to create a nonlinear hyperplane decision boundary.\n",
    "\n",
    "We can understand the intuition behidn kernels by visualizing a simple example. This function, based on one by Sebastian Raschka, plots the observations and decision boundary hyperlane of a two-dimensional space. You do not need to understand how this function works; I have included it here so you can experiment on your own:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFhlJREFUeJzt3c+LHGd+x/HPaCQr6pFlpiM8sXYcOwnBi5gxa19mwSATMBhNbEwgBl12YQm5aI+BySH/gOir2EsOYdkcfNElbIZgCAYFkqCDc5lNNjoE1l7Zji1oRbI0Xq1H0zl0P6NSqau6fjz11PPj/QLTmnF3V/V016e+/TxPPc/SZDIRACAex/reAQCAXQQ7AESGYAeAyBDsABAZgh0AIkOwA0BkCHYAiAzBDgCRIdgBIDLHe9nq0tK5XrYLACGbTD6vcjcqdgDw3K62L7yvD35c9f79VOwAgEXWrmjn+x/rtXVJuqZL16s+kGAHAM9c0c57H+u19T29euumzl+X9Is6jyfYAcAfa+/rgz+XpGu6dE3Sl02ehGAHgJ5d0c57n+rF527r7LPXdGlP0r+0eT6CHQD6s/G+PnhTaleh5xHsANADM8pl1ilaqw19EYIdABzJNrmMdfbrj/TWz7rYDsEOAN3Ld4pKlppd5iHYAaBDuaGL/+BimwQ7ANh31Ckq2e0YrYJgBwCLLuvqD20NW2yKYAeAlna1fWFPG6sf67X1WafoT/rcH4IdAFrIDFvsvFO0KoIdABroo1O0KoIdAKrbuKyrr9/W2Wcl952iVRHsAFBBrlP0pjwMdINgB4By2flceu0UrYpgB4A5sisW9Tl0sQmCHQAydrV94af6wWaXc7l0jWAHgCkri1z4gGAHkDTTKSpJPg5dbIJgB5Cq4DpFqyLYkZLR7Han173oyarGI0m6o2GSrz/Lh/lcukSwA0iC6RQ1P8dWpWctTSaTHra6dM79RpEwU6mbg3pvdptE5Woq9X0NNiVpoP09Ka3Kvctl6FyaTPR5lftRsQOIkpnLRYqnU7QqKnakhDZ2JVGpRzNsMY+KHUByYu8UrYqKHUDQUuoUpWJ3L+mv+UAfYukUtY1gBxCcGOZz6RJNMe0lPZQOcOioU1RKs0qnKQZANHxehs5HVOz22Gpj76utnj4C+Cja+VyaoGJHX2yfIDjhJCg742LqQxebINjtsVWpb+Z+7jrQ+touItfkgqhcp2jyFXpTBDtssX2CKHo+gxNPPJLvFLWNYPeHCSrXFXNf20Wk8pOOlVXudIp2g2CHLbZPEPnnM2gyikO087n4gGD3T19BRUAqqYmyOmP+dvP+ljEuQ+cjgh222Q7E/PNRqYeJYYsOEexI3UiSVjWWVK1dGNWYvx3zubhHsCM0BG0AsotcMJ+Le1x5mhafmjH63pe5c/yYyp1KvTE6RTvElacAnGLooj+o2NPg0wyUPu2L1P83h9AddYrOmlw+FFV6Z6jYAXSKZej8RcWeFp+qU5/2BRVll6GjU9Q9Knb4KHsV6ZakG33tCOpj2GI4CPa0+FQd35Bf+4MCdIqGh2CHC6ZSv1jw/wl4/xx1ikpWhi521fRGk94cBDtQQwpXpNIpGj6CHS4UhWC04RioLuZz6WohFxaIKUGwAxXUmWM8NKZTVGIZulgQ7HAp+BCMSW4Zuq6GLXa1kAsLxJQg2IEKyuYYz/O8ml+TpMTnc4n+ZECwA4Foe8IwwxbHOvv1ZzrneuhiVyEabTi3QbADNVSp1D1sh2eRi6lkOlwJdsBzLU4YG5d19XWGLqaHYEeIvKy06rTDdynbKfqZzt29qfPM5zKVTIcrwY4qoj8QfFbnhNHRfC68/4Eh2BGSINpIXVfqLENXm1efly4wbS/K+LooRpv98fJk0NC8Zehsvj7f3v/kMW0vXHIVlsm0kS6wdllX36ZTFEUIdpSJKUiDaMYpk+0UlaSP9FZ26KKt15d9XEzvf1IIdiwyUvGiGH2FZZvnX5/d7pXeyzMscoE6aGPHImVhHWIb7PXZ7Zul9/LDxq62hzXnc2lbqYf0XiaHNna0VaUaD+mrutnH4ezW54A/6hSdrVpElY5aCPY0hRDEXbk1ux3YeDLbFyNZWIau6X6EdJLGAgQ7itQ50EMYbmi2Zyr1/dmtD0F21OQiJTvjos98+IzUQrCnJfiRIb6wNeFXbhm67NDFvk+ACBjBjqyydnRbz933ScW0qfd5Ulvb1fYrmU7RlGdcrMvl++bLZ7Y2gj0tKbajdvJaG074ddQpOgv0eZ2iwYYJ/EGwp22U+7nLMPHtpNLLfC4tOkVDY/t97uOE59tntjKCPU35D2yMnATBgkp9Q5Iyi1xU6RStEybBBQ7cINjTZK4mHWo6/M9cjGKGALqogrpUdrWsE6ZTdE+v3pKUSpUudXdC7bN6Du7ESbAjVjc0PSBdB4GtZeiqVOq0w2Mugj09JgT2Na3YB3pc3fo0truJeYHnpHLf1faFXV18eTZ0MeUrRbuurEP7TPaCYEfsTOXeCTPj4qxT9Fd6egrdLuZHD7ZTD24Q7OnJhsKengyFKhN+2Q4Rm8/rMvDmLXJRRSphHPvr81qswZ7KwQPHrmjnPUmqMHSxqB28ia7b1DleIhNrsGOxeQdxlal5bYVAneetu80uAirbKfo9Sb+RVKdzdGt2G3o/BgIQerAXtTnGOlogttcTgo3Luvp6bhm6suq76DNpbKq+rpqYYj9ekhV6sKN7XYVKleftLXiyy9D9m97409/o1CeSXpL0TsP9ocMTzoQa7IsO+NgOntArq6D2d84ydBsVHubiPepq6GBQ7w8WCzXY4V5XB32Ved47D55dbV/Y08ZqrlO0aLm4NsvGEZ7oXOhrnqZWaYT2em2to5m90Mj2a892iu5JuqnHQxfzi3LkF8B2OSwUYM3TAFVpa05VdmoAa3KLXOQvLJKevmKVsEYQQg/21A600CpEG00pVtuus00u0uP5XMZaHUnSUHeyJxGzzTrro/r4PiAxoQd7DMqCK/RO0zasV+jZTtFf6rvvrOnLO9d0qcpDTeXe598/pffeSPE1W0Gw+2lLi4PNt9Av236bfSqq+iu/3uzQxY/01s8kaU1f3pEeV+oD7W8e18H6Q53YeqDTN3KVe36bff+t8STejxyCvX/zgir24ZtlbF2Kv3ZZV9++rbPPSo+HLmaDXJKWdDiQpImO7Rc+U7ummbpsX3QX4mfHt6IlOAS7X0ylXiXUfAl9Vwdh5aDLLUP3oUom6Hqg0zdWdH8rH/BjrY5mVXt2G+u5hzd9ra7es+uSnpf08waP7ftzVQUngAIEuz/KRn6k9EFtc8JauMiFCetsZ+lYq6MV3d+ad//cfuTne7HJ9kV3I01D/eTsOUMKPV+KlmAR7H5p8oHuu1p0fRA+tb3LuvpDSYuGLhaaF/YFd207/NFVhWkq9RVJS5JekfSyqlXuIVXBnAAKEOzw1cKD9J6efeEv9Lc/nnWKlja55JWEt1EWcNefvntri0Kqbmh9Jek7kp6R9HD2c2jBF9r+eiP0K09Rna2rQPu2dkU738+MQ+9qGbqyv1fbCtHmFMVlsm3sfX+zgwVceeoPDhBLcp2iXa8rWjRaaV7nts8TfYVYqaMlgj0dIbdHzl2GznR6Zsadt1Khjd0WV3/7N1s81tU+hvh59B7B3p2QOqG8dEU7732qF59r2ilqyc6cf/f9XuZHT/GZwhMI9vSEEAIb7+uDN4s6RcdaHZ3R3XeXNDkpLR2e0b2LD3ViS5qOS69bcecvWnJYuaeMwqdDBHt3uq7uojwQ5ixy4aO+3ktzv4sF/7/o8VF+VlCMYEeROmHQKjiyTS7Z+VzmMdX0gU786rgO1qXJ8el/0rIOhwPtb9atuGuMY4c9vjRrRYlg715X1V3oX2FHkk69rw/+W3qyU3QeE7qRqfte1v19LJ8V1ESwI69OGDQOjs/1wktjra7s6dV/ni1DV0muuj5a0WjaPn44NHO+NKm6A6jUYzy5+f43DxLBHp6Qv8Ju/ELnr0jS7+nLF17WJ+N/1xtvSHqjKFSLOjYj0fS9rHu/ED8rTsXWDEewI69OGFS+r1mG7m905c5L+uSLgfaXJelbncjerXIA5Q9AU8HHcmDm0KSCWgj2cHl/UO9q+8KuLr6c6RT9ScFSdIVC69hsuJ+xXBAVnFiHuhLsKNIqmDLDFq/NflU2QVfrirTFgdhr9VsxSGhSQS0EO6zKzedS1Ck6GuqO5Ca4nfC58nuoE9cl6aS+bTPFQJRC+0ZYFcEOGzYu6+rrmWXoSocuztFHRdpru/WCtVaLHiNJmp0UnexfLEGXGoI9EKsajyTpjoZeHWimU3Q2l8tNddzk4kpfFznZDFRTqR/Xo+9kf3ZZuYdygvB9/+oi2NHUwmXoamoS8m1ODL22W5sl+fJrrZ7R3Xcfafmr7H1dNvP43KSE6gh2z5lKfV+DzezPfVXuplNUkhrMuOh9J2DbYGsagMd1sL6i+4MlTU4u63DYJFDzjzGVeZ+VOieIfhDsqGRX2xd+qh9sLprLpYEmV7oaNpp08icbZ7LNN2d0993pb5cOlzQ5lV1c22UHX6ydiakh2D1nKnMXlXrBNuYuctGSjdewJWko6VbNx5WeBBYFW1eB90jLXz3Q6RtndO+iNJ1+uOpjF1XHfYyG4QTRL4Idc5lOUUkqGLpoqzmlSvNM0fqj49m/2+xD0TcGZ7KhZ+aVnxeELsORIA4bwR4IF5X6vgabA+2fel7/+4+SdFtr73S1TTUP0HVJA02rdWlauY9UfS7zSgFeVKl33WZcp1I3fK6OfdqXlBDsOHJC3/7uQz2z/H9a/bWkLwruZnPI4pak5zVdcLmsuSBb1Q8k3dDT1XuZ0Wxb80KzUYdutg3cJoIQNhDsidvV9oW39OEnknRdf/LVba39l6qHtKme61SZ2VWAVjT9DL6o6sF6Y3afuieUpo+T9HRV7CNOCjAI9oTNWYbuJS1u2phXPdcNlC1Jz0k6JmlpdvuuppV7Vj6E625n3reLRZV7KVOpm3HnLpo/XIxb56QQF4K9B32MRTfb/Gtd+deP9dq6NLdTNFvRljEBOdQ0NOtUwWYbz0s6qSc/g3Uq/6Z/uyYnoiOmDdy0tfeNYMY8BHtkik4ap3X/1EM98wcf67VP5gxbzFayVdvOs/etyzynaWOXpJ/r6fbutu341i+IctlR2WWHLRcQxY1gd6iPq0hXNR79jr7547t67ve/0cr+NV16SdJfqV3I2QjMHUm/lHRK0td6svLHAgQzyhDskcifNP5Q//P397WyekK/1SMt6xut5NuvpScr4/3Zv5u2mzdh9mk/93vblbb11+IiQLv8duDzEEm0R7A75Ooq0mUdPH9fK49ua+0LSX85+3UX87Pkn6vqNsz9sieTKo+LUpNwJZhRhmCPxB0Nd3a1feFH+rvzv9Uzn97W2p9VeJjvk3L5tj+9cDE3DOKyNJlMetjq0jn3G43W0VwuknRNl74n6Y7cTX+bfXz+oqGqlXuS4ZJvJ9/XYE9qF7ZU8HGbTPR5lftRsQes4jJ0VaQWAtlRQKm9diSAij1Mthe5sCVfgftakTsP9rJK2kaV3UX1X2e7fENwg4o9QtkZFxssclGHr4HcVvbCqlOaTmtA5Y7oULEHoMNFLor01eZe9/nLpvgt+n/ZYP9G06l/W80jU8Z1Je1qtaS+viGkjoo9fPlOUTOfS5dsztzoo2wTEZU6okWwe8hip2hfytrY25wsyk48bU5KnZ3QXI03N8/vanIyxtH7jWD3SxfL0NXRZaenT9MFEEKIGm3sHqiwDJ1rXQT79dmtudq0Tbt70zb2Ns9bWZ9VLBV03GhjD4Ovwxa7qNSHudsqKx8FxedFOJAWKvaezFnkIlb59mszL0ynozb6YEakmHZuRorANip2D5lOUUlyOHSxb/l2++iYSn1Zh7NvI4fRfitBGAh2N/ruFPVJtNXrgY7fkqQlHQ4kKvUu0IdQDcHesQiGLtoS7YGYHfpn1kQ1S+gBfSDYC7ScM/2oU3TW5JJ6lZ6UBzp9g4rSLlaMqodgt8wMXex4Lhd4JnuB0ED7mwRPuGJ47wj2nCbrkpq5XKSjCt2noYvwQAhh4fM+cqVrPQR7SwkNW0SJsuAxbe+0u/stpuYegj2n6rqkdIp6zZvJy0yoL+tw6GsTTUiB5uM++Yhgr+eoU1Ri6GJkrJwM5lXqyzocLmly6rgO1ld0f0Dl7qeYmnsI9gL5Sp1O0V7UDVvvph1+oNM3BtrfPK6D9Uc6NrY1YibbvNP2+WIKNEwR7Iv5Op9LyHoP3AwnU/aaSp3Q9F8M7xHBXsB0is5GuVClu9U0bL1da9V2pW6ad87o3sWHOmGtch9rdTTW6iiGcEsZwZ7TwzJ0KfGuqUQOTgaEJFwj2KfWJIn5XLzRNmyjDdL89AU2vwlIYYyMwWLJB7sZtjjW2a8/0zmGLnbL26YS+bUvQCspz8eeUqeob0Hq2/5gJoRKPYR97ArzsRfbuKyrrzN0sRddrqWa3EEOFEkm2LOdop/p3N2bOp9Cx6itzkrCMxE+V8H0A1SXRLAzn0vv8gtZ2zhR+DjCBvBCtMGe7RSVlOrQxbadlUXhmX/+sscStLCi6ytkY/oGEGOwswydP7Y0DXhTqZuFrG0cOD6PsAF6FdOomLXLuvo2naKdyFfqpoI3izXvzLnvpqR1SWNJQ0m39DjY35Q9BDtaybfd72uwJ/lZuSczKibbKSpJLHLRuS09DupFbmka/luz2y4OFO8Ovipi+toP/wQd7HSKOpOvyMuCel4TSb7iR0WcALoX4+yWIQb7xq62h8zn4ly2iWWox+3nVQ6CLg+UoJpi6gzZY+UlNBVSsB91is5WLaJK749pYikTRND6yoT6cR28eEb3hmOtSoqjmvRVTH/bIDpPWYbOK75UyPmhmPM6chvr+mt5lUp9WYfDYzp8biIdHOj4r5nPHTF0nh41uUhJzOeC9uo0DwHR8rJizy1DJzF0MUSuKvt5nbSNt+nL0LdsU8xExx7c05l/6mM/4JcQK/a1XW2/kukUpUJHVaZSj2Z6AbOa0YruW1t9CenwoWI/6hSdBfp/iE7RkHXa9u1qm74MffNlP+CHICp2OkVhSbTTCxDoaKKvYM8ucsF8LnGJImQJVISsl2D/rv7zj27qPPO5wDbCGFBPbexLS/JhaTwACErVNvZjXe8IAMAtgh0AIkOwA0BkCHbYMhLT8wJeINgBIDKMikFbfVxpCiSJUTEAejfW6shMiwB3fJoEDGGK4kpTICYEOwDr6iwBCPsIdtjCAQt4gs5TAJ2hUreLzlMASBQVO/pGpytQERU7ACSKih194cImoCYqdgBIFBU7+kYbO1ARFTsAJIqKHQACQcUOAIki2AEgMgQ7AESGYAeAyBDsABAZgh0AIkOwA0BkCHYAiAzBDgCRIdgBIDIEOwBEhmAHgMgQ7AAQGYIdcGCs1dFYq6PF9wTaI9gBIDLMxw50yFTpA+1vStK+BnuSNNQdVoxCbczHDgCJomIHHDCVO5U62qBiB4BEUbEDQCCo2AEgUQQ7AESGYAeAyBDsABCZXjpPAQDdoWIHgMgQ7AAQGYIdACJDsANAZAh2AIgMwQ4AkSHYASAyBDsARIZgB4DIEOwAEBmCHQAiQ7ADQGQIdgCIDMEOAJEh2AEgMgQ7AESGYAeAyBDsABAZgh0AIkOwA0BkCHYAiAzBDgCRIdgBIDL/D+DeAiVTMU3zAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "((-3.0, 2.9800000000000058, -3.0, 2.9800000000000058), None)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_decision_regions(X, y, classifier):\n",
    "    cmap = ListedColormap((\"red\", \"blue\"))\n",
    "    xx1, xx2 = np.meshgrid(np.arange(-3, 3, 0.02), np.arange(-3, 3, 0.02))\n",
    "    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)\n",
    "    Z = Z.reshape(xx1.shape)\n",
    "    plt.contourf(xx1, xx2, Z, alpha=0.1, cmap=cmap)\n",
    "    \n",
    "    for idx, cl in enumerate(np.unique(y)):\n",
    "        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1], alpha=0.8, c=cmap(idx), marker=\"+\", label=cl)\n",
    "        \n",
    "svc_linear = SVC(kernel=\"linear\", random_state=0, C=1)\n",
    "\n",
    "svc_linear.fit(features, target)\n",
    "plot_decision_regions(features, target, classifier=svc_linear)\n",
    "plt.axis(\"off\"), plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAG99JREFUeJzt3U9sHOd9xvEvnTiRZBMOWdJEbDp2YBt0GVKAbBjKoSBRRakh0gGRYAmQAUT4kosYI0APhIqkhzY9qLwUgSDn0IsgoyABEgiEmDSKqCqkU32QXXgJxkTgxE4Yp4pYEg4T2a3TsIedIYer/TOzOzPvn3k+gEGvuLvzcnfnmd++8877duzt7SEiIv64z3QDREQkXQp2ERHPKNhFRDyjYBcR8YyCXUTEMwp2ERHPKNhFRDyjYBcR8YyCXUTEM582stWOjkeMbFfEQpMszC4zdcl0O3xyimszA7zDK7x8xXRbUrW390Gcu6liFzGnb5KFWdON8NF1Tl/Z4BkmWZhdYWzEdHvy1mFkrhhV7CJcYG7iJ3z1oeuc9quqtMtQicVRgCWm3f9WpIpdxGp9tzjRr1DP3NoyU5e26dktUvVupo9dpNiGJlkYLXN803RDiiI4gA4Bo+Os3jTdnqypYhcxoMzxzQ0Gr5puR8GsFaVyV8UukqMVxkYuc3YYULVuQLRyv8zZYS/63WvQyVORHK0wNvIiKwDedwfY7hTXZrrZ6lxiehm4bbo9sejkqYh1hoJqXSxwndNXgq6Zkum2pE3BLpKTc1x8dpmpG6hat0Yk3L3qd1cfu0gOLjA3cYeeTmDNdFvkMB/73VWxi+QkqNbFTmvLTF0qc3zzAnMTphvTLgW7SMbOcXHmFif6gTum2yKNbTD47i1O9Ls+1YOCXSQHy0y5M/Ki2PYrd5f73dXHLpIh1yu/otpg8OoGg31AKeh3d+rArIpdJGPBlLzOhILsux2p3ktAn+kGxaVgF8nIOS7OmG6DtG+DwauujXdXsItko+8OPZ1aQMMP1zl9Jex3d2HUjIJdJCPb9OyaboOkZ4PBq8tMLUdGzVjbNaNgF0lfn0tf2yWRQ/3utlbvmgRMJH19X+EnL2gRDe/1lVgsAeQ2aibmJGAa7iiSsqBaVzeM/24H51BGJqHUy9auLYtnqytGJF1D2/TsqlovlJvLTF26Q09n0Pc+ZLpBqthFUnSOi8/+mkc+NN0OyV+keh/tZetZk9W7+tilSOaDn3NZPHm4OpLFQxwz/fvlQInFWYBetnbHef291NZZVR+7SL7KDHUtM1U23Q4xLzi49wG9d+jJfTpgVexSBGGlGq5eFIZvapVrpFq3cU6RzP9+aWyA9Ylh3u5/jrc2zzPf+iLmWhpPJF9BtW5bqIsF8r64SRW7FElmfcznuDjzQ779HnYve6c+dguE1TtU+uATnWRVH7tIPi4wN7HBM2B3qIslgimBgcpJ1kkWZpeYvkGKyyYq2KVIMqtUr3P6zayeO0Wq1C2T1RBJdcWkR19zC+gcF2c0i6Ok4RTXZrrZ6nyJV8sANYdI6uSpSD6CkTAibbnO6SvLTN14kRUuc3a4ndW31BXTvuqhZKrcC+QOPZ2m2yBeWQNYZurmAOsTkyzMtjJEUhW7SIvOcXEmmHNdQxwlde0MkVQfe3rSqtRNVfz6ppHQJAuz6luXPIRDJJf2pr4b5/7qipG0pX2AsPKA007/p0hS0SGScSjY05NWpZ53X73OEbRI1XpT+iwZomCXtKR9gKj3fCGTYWHtWpcioGC3SRhUeVc5prbrqr5JFkpljm+abojF9C3QMAW7pCXtA0T184WMh8U2PbsbDLY+Q59IxhTs9jFV1aiaArrYngfYobve69GbY3NcpW+BhinYJW1p78TVz2cyLIYmWRi9zmktpiFWU7BL0c0DdLENwF2ODQe3a1buQd+6ZnGMR5W6IQp2cY2xsLjA3JOmti2ShIK9WGzq8zTdlkMjN3boLgN0sV0Oble3a+gWJ/o3GNSEX2I9zRUjElPQDaN5YcR6qtiLwaZxxba0pebIjR26a975HBefBT7Mvlki7VPFLtLECmMjGzyDxq6LKzS7Y7GY7teOsqktDa0wNvIiK6DRMGLY3h5azFqsE72K9CTwhqmGiPhMwV4sNlXHb2BXe2paYWzkMmeHAV2UJM5QsEsewkr9TJ3fWxvwZYa6lpm6QbBkmbQsq643Z7r08qSTpyLJzHPvxGQiVlHFLnmoV02pyvJfVsNbbRk2ayUFu0g8ChJxhoJd8uRUCK4wNnKLE/3Au6bb4rCspvDV1MANKNhF6ljhzBORE6dJgkRhYzfv3x8Fu0hjd0w3IMLlQMqqzS6+FplTsIskE6dSVz+8nQrz/ijYRexXmECSdCjYxUW2BptO6NmtMO+Pgl3i8H5HsJzpQNL77xgFu7gkty6Jc1ycuUNPJ60trKEAtJv374+CXRrxsW839t+wzJRty+DFed3TfI98fP8LQcEuachrhzfdJSHiBAW7NOJTkPpefab190Uf59P7XygKdmlmnvqLYpgKy3aevz/4qfnVxVsKdonDiUUxmgjbf7Lqti/ara4bHaR9e628p2CXeuJU4y59VQ/b2B38vBH8HDXQFpFMKdiLyYUgzspm8PNYSs9n22vZajtcOkhLEwp2qSfJjp4kBEwFR7i9sFK/G/xUkEkzzn1GFOzF4vvIkDxl/VqaPgCKwxTsEtWoHz2t5zZ9UAn71HVQc1Oe75stn9nEFOzFUsR+1Kz+1qxeS2fDROyhYC+2+arbWYaJbQcV09v3nQ8HPNs+s7Ep2Iup+gPro7yCwOQ3AecCR/KhYC+m8GrSbirD/8KrMMMhgHlUQVlqdLWsZCurA6rJ6tm5A6eCXXwVXi3ralWrJfikZQr24glD4C6Viv0YB9Wt62O7awWeKvd8ZV1Zu/aZNELBLr4zPc9NFvOjO3tST/KhYC+eaCiUORwKjYIiqxBJ83ldCDyb25Ym3/8+q/ka7EXZecRe9frB03wu169ylYz4GuzSXK2duFGlnnaoJHnepNvMOqBaeQ3C6YJdP48hDnA92Ov1Ofo6WsC3v8dqJRZLwbqnSRa0rveZDA2TnK5ylURcD3bJXlahEud5jQXPK7x85RwXZ6q2G2qlPS70/4snXA32Zju8bzuP65WVa+1tRR7vkSvz3Yhhrga75C+rnT7OPO9GgudHfOPrwPPAU8E/lat+ttIehadkrmNvb8/AVjseSemZilZpuPb3VlexrQZi9EKj3P72o9z9z485+ntgN/in6gWw8xwWKsLeHh/EuZ8qdnvE6WsuqujUALn5HDu/+C+ObgIfRdohYj3Xg71oO5prFWIaXSl5nV/Yf94utucBPuH+R4EHgc8keB4b3wcpGNeD3QeNgsv1k6btyLpCb/padrD3f0f5Q/dHPPAmsBPnMRkq0nsfKuLfnAoFu51O0jzYbAv9Rttvp031qv4sLpTqB07u0P0GQeX+OXYf3+SBHwBrkfuafq3lML0fVRTs5tUKKt+HbzaS5qX4cZ4/nIP+bo37hr5DpWKvfkwW0r7ozsXPjm1Fi3MU7HYJK/U4oWZL6Oe9UlHa23uDyuteHfDzO3TPXWBu4m/4xxeDf+uvemyr287rPbsBPAz8uIXHmv5cxaEDQB0Kdns0GvlRpA9qXvN5V39DOlnrzpeY/eZD7HzmQ7q+xb3zvaQp7Yvu5qmE+meD53Qp9GwpWmwyEvxcjHNnBbtdWvlAm64W894Js9he3ef8PZ3vf4b/HaSyY1Uv2NHqa591hRlW6g8AHcAA8ATxKneXquAiHACGSiyOArzEq2VYifUgBbvYKq8DRE3hkMe7HBs+wn0PAt8CfhM87kaG7akXUklfj98Cj1IZqvk/wW3Xgs+19qahD+gNwxwqgT7O6s0kT+L6lacSX1pXgRbC4WD/+MHf8dB7HAR7uxVimlMUNxLtYzf9zU4aGGB9AmCYt/uf461NgPPMX73njnt7uvLUEtpBHLRD9xxUAr6Lnc//Fa//xzJTX6b2yW2bJ/pysVIvipESi/vTOC8xfQP4D5JNE12Tgr04fOyPTHsOmZqvzQ5dv1li+sZRJkY+5mgKm8nttR9tfpe68mqjT5/HRobC/4l2swRhvpb2xhTs2XHpJJTUEVbuwNCLXL25zNQNIJyn3fQQ05A+U5Y6xbUZgG62OsMulmHWEveZJ6VgLx4fQmAe+BqVoXx/As5wMBSxleo9zkF47SVe7d6m59nrnE7eYqnmW+HTB3CKay90s9UZ/mMvW7uv8PK/kkL3ShIK9uxk3fXh+o7gnHFWb17m7HCJxfeDyj0tcd/L8H5n6vy+3uP1WUlfOK6cEovDvWyFUzvzCi9fMtOkAwp2qSdJGOQZHOG23qNyJeinOfgcd9PaxTixD8JLTF9aYWwEGA2q9ysxtyGHuXbO59CJTgjHlcM4q0nXxc2cgj17WVXqrn+FbXXOE+PGWb05zurNSRZmSyzOVlXvSU6EJX0vk/67L5+VrA1V3e6uDnGAJaatC/B6FOxSLUkYmAiOaKUXXdFomErFfqzqfq08dyxLTF8Chi7w1pMAtzjRD4yWOb4Z5/EbDB7lYBGPhgZYn3ifL36z1u8e5xcTtf79fb74OMAn3P85gPv55PFG949rg8FUhuQFsj7I9A2w/uV6vxzm7f5etna/wK8+jP77eeYzGa2SFwW7e1z7Clst6cEg69ke27V2nvloAFRXfzWtMNZ9mbPvANzgLwcBRvn394Nfz9Z+1H2/A3iHP/9BzLZ9H+AZfvqdhI+r6wJzT97iRKnd58lTcOXmdp1fv8vhKZld259qUrBLtSQHjlQPMuHVnpEhho1U3yes4E3vmLGqvHFWCYe8dbH9OOx/A7hH5CrY/uD2DMR+nbhN306StjVSdRATSynY3WU6wFqV9GDg2jeUxO2MG9Ctyvr5HefleQgFu9ST5IOdSqV+l2PD0dsJA6mlNrS4rTQ1DZLo9AbR2yL1KNgld3ECqoudzwfDCkOvBT9Hat2/mXFWN8h2RIPNlV84aqedKQZ85do3wlgU7JKWvnNcfCHOHY/w0dMA57g4AzDNv6wB/IhvPA3wF9x8H3j/xZhzTzczwHrXZc6WoheR/IhvfB3gE+5/8BPu/6OBajh6IOgnxpw3O3Tn0Kx9XgVd0SjY09GX9QYe45ffA/gVX/iHVh4/yULmIxnKHN8MhsLV873g54MAP+Tbzwe3w7/peYBlplIdL7zBIBsMwuH36fkjfPRwB3/q/hR/5AgfPf0xR38W+X1mFzk1kfa0vVCZlz16O8/K3ZUDhO3tS0TB3sAF5pqO9w3GLrNNz26z+7Zjlwf7Ab7CT2JVxdWuc7oMZDrxUAzhmO0/Vt2+TSUAPiLZDpY0NKIHi5eDmRrnAcb5cTjUcIh8xi+H87pXr7X6NSpT7Ubl2c1jc5eSxFT0YO9rVMlu07N7ndNvNnmOcBxsVsId6wjAdU6H46Rd3NGs7c9cZuoSMLLG9y8c4eNPnuLnR4JftVq5J9VPJeQ/S+VCq1Zeo+rHhJW5yUpdBwgDfA/2QxeLTLJwzwc76D64d6USyUsrV7qG0giNQwebR/lg/dc8MgjEviq0DdFtfy34/z8F2z5Z537R21m3KettSUZ8C/b9E3h36OmESnBHfu9iiOe5o+W9IHU7TlKpbGNdvh/R8G/sZmcOGCnzpbkh1tdr3C+r1+i3VE6ehjM3Vi+c3Uizg6OJ0TA6QBjkfLBPsnDo8utlpsrARnDTiQl7HJXWDhsnAOqt17pN+1ebNpqyIJXlkmKItv9kjX+rdb+sKYgd5lSwrzA2ssKZJ8JqHPb7wYswdWoelXqe/aGtzvkS9kWHY/9OBs8Vdy7zWHPOlFj652He7l9iuo+Dk7u1Hp/2a5SkUg/ZXB3b1JbCsD7Yo2EehPh7mB/dUWRpBtxJ4GEq3RCNuguiwXWMSvhVV++NRNdGbfTc+7c3GORRPpipcf/QyQa/a4eCUNpmY7D3weFx12mPa5Z7tFrxhdVzK/3BZ4AHqHwGH0uw7fAinqRtbfVxvVQ+e9WvkY10UBDAomC/wNzEL3nsoUhlbsO46yJq1rVRq3pOGigngYeA+4CO4Gej8dtzVT/jqvXtolnlvu86p9+cZGF0iWk4GNIaVurhuHPXT2rb2H0jbTIa7BeYm4he4BOMGS/CtKAmdqa40/DGqUjDgGxlKbpwGw9TGbMd/Qwmqfxbfe2SHIjWyhx/ssbj4eBgYZqCWe5hKtj3LwxSN0vqWr2wJaxk4/adR++bVPicYR87wI+5t7uj3X78LE4qmhh+msUJW11A5DEjwT7JQqmgFwaZHH2S5jbTCLc54KdUhhTucrjyt0rQHWPbN0kFs9RlJNiDy7clXYmG89V4TNhn3Gq/eSvCPvW7Vf+edlXc8uOD4mOCe+eQyeP1yfLbgc1DJKVN1pw8LQgTO1Me4RCKu43wftGDSZzHGbHB4M4KY93jrGa1iVbeGwWz1KVg90crO7rt4WBbe0zJY24Y8YiC3QwTO5NN/fi2H1DykkY/eV5z2YhDFOz+aWeGw6KIjgIq2t8uBaBglzTUq8DjVu5FUOu1SPObi6lRMvqGYKH7TDdArDSPpUMP2zRPZdGJM1SWizsT3G76t5YZ6sq2aSLpUcUuaUr7AqN64kzx28q26j325i1OzK4wNjLOatJpLuK8FmleOHWj6nZWNI7eYgp2ifJ9Z40eeBL1sS8zVX6JV7Nql0iqFOyShUZ9x2lU1LUOPO0clLI8oOU1Aqj62oCst6eRTRZTsEtUljurTX32CiHxWsfe3l7+G+3gkdw3KklkEexh329YUYYLZKTZF97sd+0878hrjBOzj91kFasK2md7ex/EuZsqdqkli0q9u+pnnJWPXGPTtxIpMAW75GUz+JnGvDDN5pTP4nnjMLEIRzVV6qJgl8xV99v7qEjfSsQBCnbJm88VZZrfSqQ2nUOIQcEuefF5R6weHw/JlvkTSZWCvT5VBtIqTS6WPt8vnkuV5ooRSUf0AqFwmT+fzyv4zPn3ThX7vVQZSBZc+BzZ3EZd6ZqAgl0kHc2mUQjnphF7eVPUKdjvpcrAfTa9d2God3PQRQN2tC3kUqDZ2CbrKNhFKtIKs1qVejdwFOinMhRSlbudvCnqFOz1OfumeiTpDmZj5fkGlfb0A9ukN2ImzeX9vAk0qVCwiwk2BUheU/aGlboNf7M05vx7pGAXG7UatjZXnmlX6mH3zhnSrdzDoX42vXaSkIJd8mRjV0keBwOFpORKwS42ajdsfQ7Slpf3a8LGg660SMEuebK5q8Smtoi0RcFeDDYGaRyutTdPab82Nh90q7nQRqMU7JKnLNdSzXInHymxODzO6o3mdxUxT8Hut7T6TQtfIb3Eq2VgzXQ7cmDze6zzADEp2CUP1QtZp7FDaicXqUPB7rd2+03rhWf18zd6rIJW0pL1eQBvPrMKdsnSSSo7S1ipp7lknEsn+0Ry1bG3t5f/Rjt4JPeNSjuqK/Wwgg8Xa66e+Cq8Tzg/SjeV9UDDYB/NoG2ZBfsprs38G199k2L0sRdR9TfTWp9rO+ztfRDnbqrYJanwcvbNZncM7lMOHlPGwas6B1if6Gark/RDXd80JDMKdomj1lS09YK6VheJ08uMLTFtcpijDgDZ865bT8EucUW/rnZz0H8eZyfIckdxbWdMMppHKy9JSxTs0oqwi6URV4LWVmGoP0blQBrS65odb15bBbvEZdvX1UzHsW/TNb9J/9N/z3evAe+m8ZyBOK9jdGreTwMPoMpdErjPdANEUhR2D6VmiellNBpGHKPhjpKVvCr7WidpW97mNl3zAMe4O/w7Ov+sl/9+A/ionedsUbQr5g/A68G/m/6mJCZpuKMUSFipp9ot0wGfaufxbYoeqLSkniSiil3SZuJij9S3WeZLrx3h40+e4udfb6tl7bPlnIbYQBW7FEgmJ3af4uc/S+N52qRAl8SMnDwtsTgLDJnYtmRuLvivzMFFTK6F09Df8bev4167RQBDFfsS0zcmYXSbnmevc/qKiTaIl1IJ4hKLo8H86yJOMjXccW2J6UsDvBNW7yOG2iHZcbFS3zfO6k3TbRBpldE+9ld4+QowNFmZ7W94malLJtsjgooM8YANFyitLTF9qZetXVXvYlhficXh4KIkEWdZMypG1bvYoJetXeC26XaItMOaYA+sLTG9doG5CWC2zPFNgA0GrxpulzTn/HjrU1x7wXQbRNJg5AIlOjriXKDUB/ROsjAKsMzUDTRnh81cD/ahEoujS0zrm6LYy4MLlG4Dt5eYXlthbISD4ZFaoswumc6ymJcB1p98jrfirAolYj2bg33fOKs3x1m9ucLYyADvPHuHntFlpsqAhqRJGoaGebv/PPM6aZo+Jw/0rrO5K6aRobCLBmCZqWV0wss0l3fgkdcY19j1bLj8ubCPB10xjawtMb0GEJxoLZU5vrnB4A6q4iWZkRKLw+OsmlzX1EdedNG5ytVg33ee+atA3wpjA5c5O0xlqKROtObP2R02mD5AnxfxhvPBHrg9zurtsB+eylj46GiaO6irRqoMsD4xzNv9NF+/VZKzbSnFQnG1jz2WoJuGW5zoD7pqNB5e9g2wPvEOX3oXVetZUrCnKWYfu9fBHtE3yUIpvKGuGqsY2fHDan2JaX0WxB2enzxN6nZ44YnGxAvAo3zwUDAnjLroxDtFCfZ9GhNvDZOjJka62erMYTsiRhQu2ENhwBOZeAw0Jr4ITnHtCVXr4rPCBntEvTHxOtGaLVOjJoZUrYvvinLyNJFJFmbD/9eJ1szlGuwlFmd1wlScpZOnratzolVrs2Yjr0q9r8RiODJKoS5es2EFJWuNs3pTa7P6o5etXU3LK0Wgij0Gre7kvLBa3zXdEJE8qGKPT2uzOmqA9S8/x1ubwQFaxHs6edqaocjKTqre7RaujKThjeK+mCdPVbG3Zm2J6UvP8dZmicXZAdYnTDdIahopsTgarIykUJfCUMXevv15aHRxk11KLM6+xKtlLaAh3lDFnpvbkeq9pOrdDgOsT/SytatQlyJSxZ6uvkkWSqrczdLMjeItXaBkxO2gP1fTEpgzpFCXolPFng31uxtyimszf80/vacuGPGS+tiNOtTvDgyZblARlFic7WarU6EuRadgz9B55q8G4T4K9Jluj8/Ck9aaMkBEwZ65SLhrxEx2+oJ+dYW6CDp5movzzF/loN99QidVU9VXYrEUnLS2mRZ1ltyoYs/P7SWml4d5u1+Ve2r6SiyWetnaDQ6eIoJGxZigETPpGCmxOPwcb21aHurVa7uWg5+q3CU5jYqxVvWIGZ1UbUGJxeGXeLVseaiLGKGK3aBzXJy5Q0+nKvf4wqtKHajUq6mPXdqnK0/t9wovX9EC2snoqlKR5lSx20H97s250qcukp2YFbuC3SIrjI1c5uzwNj27Wjz7sBKLs1osQwpPwe6uSRZmAZaZUpdDUKn3srWrpe2k8BTsblP1XqnSAdSnLhJQsPshUr0X6XL5oXBJO/Wni0RYHewiIpIZXaAkIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinlGwi4h4RsEuIuIZBbuIiGcU7CIinvl/2tsDRl6CrTYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "((-3.0, 2.9800000000000058, -3.0, 2.9800000000000058), None)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svc = SVC(kernel=\"rbf\", random_state=0, gamma=1, C=1)\n",
    "model = svc.fit(features, target)\n",
    "plot_decision_regions(features, target, classifier=svc)\n",
    "plt.axis(\"off\"), plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 17.3 Creating Predicted Probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00588822, 0.96874828, 0.0253635 ]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "features = iris.data\n",
    "target = iris.target\n",
    "\n",
    "scaler = StandardScaler()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "svc = SVC(kernel=\"linear\", probability=True, random_state=0)\n",
    "\n",
    "model = svc.fit(features_standardized, target)\n",
    "\n",
    "new_observation = [[.4, .4, .4, .4]]\n",
    "model.predict_proba(new_observation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discussion\n",
    "Many of the supervised learning algorithms we have covered used probability estimates to predict classes. For example, in k-nearest neighbor, an observation's k neighbor's classes were treated as votes to create a probability that an observation was of that class. THen the class with the highest probability was predicted.\n",
    "\n",
    "SVC's used of a hyperplane to create decision regions does not naturally output a probability estimate that an observation is a member of acertain class. However, we can in fact output calibrated class probabilities with a few aveats. In an SVC with two classes, Platt scaling can be used, wherein first the SVC is trained, and then a separate cross-validated logistic regression is trained to map the SVC outputs into probabilities:\n",
    "$$\n",
    "P(y=1 | x) = \\frac{1}{1+e^{(A * f(X) + B)}}\n",
    "$$\n",
    "where A and B are parameter vectors and f is the ith observation's signed distanced from the hyperplane. When we have more than two classes, an extension of Platt scaling is used.\n",
    "\n",
    "\n",
    "## 17.4 Identifying Support Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.5810659 ,  0.43490123, -0.80621461, -0.50581312],\n",
       "       [-1.52079513, -1.67626978, -1.08374115, -0.8607697 ],\n",
       "       [-0.89430898, -1.46515268,  0.30389157,  0.38157832],\n",
       "       [-0.5810659 , -1.25403558,  0.09574666,  0.55905661]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "features = iris.data[:100,:]\n",
    "target = iris.target[:100]\n",
    "\n",
    "scaler = StandardScaler()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "svc = SVC(kernel=\"linear\", random_state=0)\n",
    "\n",
    "model = svc.fit(features_standardized, target)\n",
    "model.support_vectors_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 17.5 Handling Imbalanced Classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "features = iris.data[:100,:]\n",
    "target = iris.target[:100]\n",
    "\n",
    "features = features[40:, :]\n",
    "target = target[40:]\n",
    "\n",
    "target = np.where((target == 0), 0, 1)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "svc = SVC(kernel=\"linear\", class_weight=\"balanced\", C=1.0, random_state=0)\n",
    "\n",
    "model = svc.fit(features_standardized, target)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:machine_learning_cookbook]",
   "language": "python",
   "name": "conda-env-machine_learning_cookbook-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}