SKN Statystyki - Python 11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wyklad 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine Learning cd."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Czytanie i Pisanie"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plików Excela"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame({'Data': [10, 20, 30, 20, 15, 30, 45]})\n",
    "writer = pd.ExcelWriter('pandas_simple.xlsx', engine='xlsxwriter')\n",
    "\n",
    "df.to_excel(writer, sheet_name='Sheet1')\n",
    "\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "path = ('Book1.xlsx')\n",
    "xl = pd.ExcelFile(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xl.sheet_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = xl.parse('Sheet1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- classification\n",
    "- regression\n",
    "- outlier detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVC Flavors in Scikit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SVC - many kernels to choose from. L2 error metric for regularization (strong penalization of big factors, sensitive to outliers)\n",
    "- NuSVC - slightly different implementation - worth checking against SVC\n",
    "- Linear SVC - only linear kernel, more settings, L1 error metric for regularization\n",
    "- OneClassSVM - good for detecting outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What are they good for"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Effective in high dimensional spaces.\n",
    "- Still effective in cases where number of dimensions is greater than the number of samples.\n",
    "- Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.\n",
    "- Versatile: different Kernel functions can be specified for the decision function. Common kernels are provided, but it is also possible to specify custom kernels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Between\n",
    "\\begin{equation}\n",
    "O(n_{features} \\times n_{samples}^2)\n",
    "\\end{equation}\n",
    "and\n",
    "\\begin{equation}\n",
    "O(n_{features} \\times n_{samples}^3)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "digits.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "clf = svm.SVC(C=100., kernel=\"linear\")\n",
    "X = digits.data[:-1]\n",
    "y = digits.target[:-1]\n",
    "clf.fit(X, y)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "shape = [n_class * (n_class-1) / 2, n_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.coef_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import datasets\n",
    "from sklearn import svm\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "iris.data.shape, iris.target.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn import svm, datasets\n",
    "\n",
    "# import some data to play with\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data[:, :2]  # we only take the first two features. We could\n",
    "                      # avoid this ugly slicing by using a two-dim dataset\n",
    "y = iris.target\n",
    "\n",
    "h = .02  # step size in the mesh\n",
    "\n",
    "# we create an instance of SVM and fit out data. We do not scale our\n",
    "# data since we want to plot the support vectors\n",
    "C = 1.0  # SVM regularization parameter\n",
    "svc = svm.SVC(kernel='linear', C=C).fit(X, y)\n",
    "rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(X, y)\n",
    "poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(X, y)\n",
    "lin_svc = svm.LinearSVC(C=C).fit(X, y)\n",
    "\n",
    "# create a mesh to plot in\n",
    "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "\n",
    "# title for the plots\n",
    "titles = ['SVC with linear kernel',\n",
    "          'LinearSVC (linear kernel)',\n",
    "          'SVC with RBF kernel',\n",
    "          'SVC with polynomial (degree 3) kernel']\n",
    "\n",
    "\n",
    "for i, clf in enumerate((svc, lin_svc, rbf_svc, poly_svc)):\n",
    "    # Plot the decision boundary. For that, we will assign a color to each\n",
    "    # point in the mesh [x_min, m_max]x[y_min, y_max].\n",
    "    plt.subplot(2, 2, i + 1)\n",
    "    plt.subplots_adjust(wspace=0.8, hspace=0.8)\n",
    "\n",
    "    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "\n",
    "    # Put the result into a color plot\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.6)\n",
    "\n",
    "    # Plot also the training points\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)\n",
    "    plt.xlabel('Sepal length')\n",
    "    plt.ylabel('Sepal width')\n",
    "    plt.xlim(xx.min(), xx.max())\n",
    "    plt.ylim(yy.min(), yy.max())\n",
    "    plt.xticks(())\n",
    "    plt.yticks(())\n",
    "    plt.title(titles[i])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- linear (use linear SVC if not grid-searching kernel)\n",
    "- RBF - default - Radial-basis function kernel (aka squared-exponential kernel).\n",
    "$$ k(x_i, x_j) = exp(\\frac{-1}{ 2 d(x_i / lengthscale, x_j / lengthscale)^2}) $$\n",
    "- poly\n",
    "- sigmoid\n",
    "$$ K(X, Y) = tanh( gamma ~\\times <X, Y> + coef_0)$$\n",
    "- precomputed, callable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "x = np.linspace(-20, 20, 100)\n",
    "plt.title('Logistic function')\n",
    "plt.plot(x, np.tanh(x))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- C - Penalty parameter for error\n",
    "- kernel\n",
    "- gamma - kernel coefficient for rbf, poly, sigmoid  \n",
    "  - if gamma='scale' is passed then it uses 1 / (n_features * X.std\n",
    "- class_weight : {dict, ‘balanced’}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## C - Penalty & Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img\\over.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img\\kernel.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weighting classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img\\unbalanced.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "digits = datasets.load_digits()\n",
    "\n",
    "n_samples = len(digits.images)\n",
    "X = digits.images.reshape((n_samples, -1))\n",
    "y = digits.target\n",
    "\n",
    "# Split the dataset in two equal parts\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.5, random_state=0)\n",
    "\n",
    "# Set the parameters by cross-validation\n",
    "tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],\n",
    "                     'C': [1, 10, 100, 1000]},\n",
    "                    {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]\n",
    "\n",
    "scores = ['precision', 'recall', 'f1']\n",
    "\n",
    "for score in scores:\n",
    "    print()\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "\n",
    "    clf = GridSearchCV(SVC(), tuned_parameters, cv=5,\n",
    "                       scoring='%s_macro' % score)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, recall_score, precision_score, SCORERS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$F_1 = \\frac{2 \\times precision \\times recall}{precision + recall} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = [0, 1, 0, 1, 0, 1, 0, 1]\n",
    "y_pred = [0, 1, 0, 0, 0, 0, 1, 1]\n",
    "\n",
    "precision_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recall_score(y_true, y_pred)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f1_score(y_true, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SCORERS.keys()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}