Gradiant_Boosting_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Boosting Classification\n",
    "\n",
    "The Gradient Boosting (GB) algorithm trains a series of weak learners and each focuses on the errors the previous learners have made and tries to improve it. Together, they make a better prediction.\n",
    "\n",
    "According to Wikipedia, Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion as other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function. \n",
    "\n",
    "Prerequisite\n",
    "\n",
    "    1. Linear regression and gradient descent\n",
    "    2. Decision Tree\n",
    "    3. Gradient Boosting Regression\n",
    "\n",
    "After studying this post, you will be able to:\n",
    "\n",
    "    1. Explain gradient boosting algorithm.\n",
    "    2. Explain gradient boosting classification algorithm.\n",
    "    3. Write a gradient boosting classification from scratch\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    \n",
    "# The algorithm\n",
    "\n",
    "The following plot illustrates the algorithm.\n",
    "\n",
    "![Gradient Boosting Regression](gradient_boosting/gradient_boosting_classification.png)\n",
    "\n",
    "(Picture taken from Youtube channel StatQuest)\n",
    "\n",
    "From the plot above, the first part is a stump, which is the log of odds of **y**. We then add several trees to it. In the following trees, the target is not y. Instead, the target is the residual or the true value subtracts the previous prediction.\n",
    "\n",
    "$$residual=true\\_value - previous\\_prediction$$\n",
    "\n",
    "That is why we say in Gradient Boosting trains a series of weak learners, each focuses on the errors of the previous one. The residual predictions are multiplied by the learning rate (0.1 here) before added to the average.\n",
    "\n",
    "Here the picture looks more complicated than the one on regression. The purple ones are log of odds (**l**). The green ones are probabilities. We firstly calculate log of odds of **y**, instead of average. We then calculate probabilities using log of odds. We build a regression tree. The leaves are colored green. The leaves have residuals. We use the probability residuals to produce log-of-odds residuals or $\\gamma$. $\\gamma$ is then used to update **l**. This continues until we are satisfied with the results or we are running out of iterations.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**The Steps**\n",
    "\n",
    "Step 1: Calculate the log of odds of y. This is also the first estimation of y. Here $n_1$ is the number of true values and $n_0$ of false values.\n",
    "$$l_0(x)=\\log \\frac{n_1}{n_0}$$\n",
    "For each $x_i$, the probability is:\n",
    "$$p_{0i}=\\frac{e^{l_{0i}}}{1+e^{l_{0i}}} $$\n",
    "The prediction is:\n",
    "$$\n",
    "f_{0i}=\\begin{cases}\n",
    "0 & p_{0i}<0.5 \\\\\n",
    "1 & p_{0i}>=0.5\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Step 2 for m in 1 to M: <br />\n",
    "  * Step 2.1: Compute so-call pseudo-residuals:\n",
    "    $$r_{im}=f_i-p_i$$\n",
    "  * Step 2.2: Fit a regression tree $t_m(x)$ to pseudo-residuals and create terminal regions (leaves) $R_{jm}$ for $j=1...Jm$ <br />\n",
    "\n",
    "  * Step 2.3: For each leaf of the tree, there are $p_j$ elements, compute $\\gamma$ as following equation. <br />\n",
    "\n",
    "$$\\gamma_{im}=\\frac{\\sum r_{im}}{\\sum (1-r_{im-1})(r_{im-1})} $$\n",
    "\n",
    "  * (In practise, the regression tree will do this for us.)\n",
    "\n",
    "  * Step 2.4: Update the log of odds with learning rate $\\alpha$:\n",
    "$$l_m(x)=l_{m-1}+\\alpha \\gamma_m$$\n",
    "\n",
    "For each $x_i$, the probability is:\n",
    "$$p_{mi}=\\frac{e^{l_{mi}}}{1+e^{l_{mi}}} $$\n",
    "The prediction is:\n",
    "$$\n",
    "f_{mi}=\\begin{cases}\n",
    "0 & p_{mi}<0.5 \\\\\n",
    "1 & p_{mi}>=0.5\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "\n",
    "Step 3. Output $$f_M(x)$$\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (Optional) From Gradient Boosting to Gradient Boosting Regression\n",
    "\n",
    "The above knowledge is enough for writing BGR code from scratch. But I want to explain more about gradient boosting. GB is a meta-algorithm that can be applied to both regression and classification. The above one is only a specific form for regression. In the following, I will introduce the general gradient boosting algorithm and deduce GBR from GB.\n",
    "\n",
    "Let's first look at the GB steps\n",
    "\n",
    "---\n",
    "\n",
    "**The Steps**\n",
    "\n",
    "Input: training set $\\{(x_i, y_i)\\}_{i=1}^{n}$, a differentiable loss function $L(y, F(x))$, number of iterations M\n",
    "\n",
    "Algorithm:\n",
    "\n",
    "Step 1: Initialize model with a constant value:\n",
    "\n",
    "$$F_0(x)=\\underset{\\gamma}{\\operatorname{argmin}}\\sum_{i=1}^{n}L(y_i, \\gamma)$$\n",
    "\n",
    "Step 2 for m in 1 to M: <br />\n",
    "  * Step 2.1: Compute so-call pseudo-residuals:\n",
    "    $$r_{im}=-[\\frac{\\partial L(y_i, F(x_i))}{\\partial F(x_i)}], {F(x)=F_{m-1}(x)}$$\n",
    "  * Step 2.2: Fit a weak learner $h_m(x)$ to pseudo-residuals. and create terminal regions $R_{jm}$, for $j=1...J_m$<br />\n",
    "\n",
    "  * Step 2.3: For each leaf of the tree, compute $\\gamma$ as the following equation. Here $\\hat{r}$ is the predicted residual produced by $h_m(x)$.<br />\n",
    "$$\\gamma_{jm}=\\underset{\\gamma}{\\operatorname{argmin}}\\sum_{x_i \\in R_{jm}}^{n}L(y_i, F_{m-1}(x_i)+\\gamma)$$\n",
    "\n",
    "  * Step 2.4: Update the model with learning rate $\\alpha$:\n",
    "$$F_m(x)=F_{m-1}+\\alpha\\gamma_m$$\n",
    "\n",
    "\n",
    "Step 3. Output $$F_M(x)$$\n",
    "\n",
    "Lost Function:\n",
    "\n",
    "To deduce the GB to GBC, I simply define a loss function and solve the loss function in step 1, 2.1 and 2.3. We use Log of Likelihood as the loss function:\n",
    "\n",
    "$$L(y, F(x))=-\\sum_{i=1}^{N}(y_i* log(p) + (1-y_i)*log(1-p))$$\n",
    "\n",
    "Since this is a function of probability and we need a function of log of odds($l$), let's focus on the middle part and transform it into a function of $l$.\n",
    "\n",
    "The middle part is:\n",
    "\n",
    "$$\n",
    "-(y*\\log(p)+(1-y)*\\log(1-p)) \\\\\n",
    "=-y * \\log(p) - (1-y) * \\log(1-p) \\\\\n",
    "=-y\\log(p)-\\log(1-p)+y\\log(1-p) \\\\\n",
    "=-y(\\log(p)-\\log(1-p))-\\log(1-p) \\\\\n",
    "=-y(\\log(\\frac{p}{1-p}))-\\log(1-p) \\\\\n",
    "=-y \\log(odds)-\\log(1-p)\n",
    "$$\n",
    "\n",
    "Since\n",
    "\n",
    "$$\n",
    "\\log(1-p)=log(1-\\frac{e^{log(odds)}}{1+e^{log(odds)}}) \\\\\n",
    "=\\log(\\frac{1+e^l}{1+e^l}-\\frac{e^l}{1+e^l})\\\\\n",
    "=\\log(\\frac{1}{1+e^l}) \\\\\n",
    "=\\log(1)+\\log(1+e^l) \\\\\n",
    "=-log(1+e^{\\log(odds)})\n",
    "$$\n",
    "\n",
    "We put this to the previous equation:\n",
    "$$\n",
    "-(y*\\log(p)+(1-y)*\\log(1-p)) \\\\\n",
    "=-y\\log(odds)+\\log(1+e^{\\log(odds)}) \\\\\n",
    "$$\n",
    "Thus, we will have the loss function over log of odds:\n",
    "\n",
    "$$L=-\\sum_{i=1}^{N}(y\\log(odds)-\\log(1+e^{\\log(odds)}))$$\n",
    "\n",
    "\n",
    "\n",
    "For Step 1:\n",
    "\n",
    "Because The lost function is convex and at the lowest point where the derivative is zero, we have the following:\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L(y, F_0)}{\\partial F_0}  \\\\\n",
    "=-\\frac{\\partial \\sum_{i=1}^{N}(y\\log(odds)-\\log(1+e^{\\log(odds)}))}{\\partial log(odds)} \\\\\n",
    "=-\\sum_{i=1}^{n} y_i+\\sum_{i=1}^{N} \\frac{\\partial log(1+e^{log(odds)})}{\\partial log(odds)} \\\\\n",
    "=-\\sum_{i=1}^{n} y_i+\\sum_{i=1}^{N} \\frac{1}{1+e^{\\log(odds)}} \\frac{\\partial (1+e^l)}{\\partial l} \\\\\n",
    "=-\\sum_{i=1}^{n} y_i+\\sum_{i=1}^{N} \\frac{1}{1+e^{\\log(odds)}} \\frac{\\partial (e^l)}{\\partial l} \\\\\n",
    "=-\\sum_{i=1}^{n} y_i+\\sum_{i=1}^{N} \\frac{e^l}{1+e^l} \\\\\n",
    "=-\\sum_{i=1}^{n} y_i+N\\frac{e^l}{1+e^l}=0\n",
    "$$\n",
    "\n",
    "And We have: (Here p is the real probability)\n",
    "\n",
    "\n",
    "$$\n",
    "\\frac{e^l}{1+e^l}=\\frac{\\sum_{i=1}^{N}y_i}{N}=p \\\\\n",
    "e^l=p+p*e^l \\\\\n",
    "(1-p)e^l=p \\\\\n",
    "e^l=\\frac{p}{1-p} \\\\\n",
    "\\log(odds)=log(\\frac{p}{1-p})\n",
    "$$\n",
    "\n",
    "Such that, when log(odds)=log(p/(1-p)) or the probability is the real probability, the lost function is minimized. \n",
    "\n",
    "For Step 2.1\n",
    "\n",
    "$$r_{im}=-[\\frac{\\partial L(y_i, F(x_i))}{\\partial F(x_i)}]_{F(x)=F_{m-1}(x)}$$\n",
    "\n",
    "$$=-[\\frac{\\partial (-(y_i* log(p)+(1-y_i)*log(1-p)))}{\\partial F_{m-1}(x_i)}]_{F(x)=F_{m-1}(x)}$$\n",
    "\n",
    "We have already taken the derivative.\n",
    "\n",
    "$$=y_i-F_{m-1}(x_i)$$\n",
    "\n",
    "For step 2.3:\n",
    "\n",
    "$$\\gamma_{jm}=\\underset{\\gamma}{\\operatorname{argmin}}\\sum_{x_i \\in R_{jm}}^{n}L(y_i, F_{m-1}(x_i)+\\gamma)$$\n",
    "\n",
    "I apply the lost function:\n",
    "\n",
    "$$\n",
    "\\gamma_{jm} \\\\\n",
    "=\\underset{\\gamma}{\\operatorname{argmin}}\\sum_{x_i \\in R_{jm}}^{n}L(y_i, F_{m-1}(x_i)+\\gamma) \\\\\n",
    "=\\underset{\\gamma}{\\operatorname{argmin}}\\sum_{x_i \\in R_{jm}}^{n} (-y_i * (F_{m-1}+\\gamma)+\\log(1+e^{F_{m-1}+\\gamma})) \\\\\n",
    "$$\n",
    "Let's focus on the middle part\n",
    "\n",
    "$$-y_i * (F_{m-1}+\\gamma)+\\log(1+e^{F_{m-1}+\\gamma})$$\n",
    "\n",
    "Let's use Second Order Taylor Polynomial:\n",
    "\n",
    "$$L(y,F+\\gamma) \\approx L(y, F)+ \\frac{d L(y, F+\\gamma)\\gamma}{d F}+\\frac{1}{2} \\frac{d^2 L(y, F+\\gamma)\\gamma^2}{d^2 F}$$\n",
    "\n",
    "Let's take the derivate:\n",
    "\n",
    "$$\n",
    "\\because \\frac{d L(y, F+\\gamma)}{d\\gamma} \\approx \\frac{d L(y, F)}{d F}+\\frac{d^2 L(y, F)\\gamma}{d^2 F}=0 \\\\\n",
    "\\therefore \\frac{d L(y, F)}{d F}+\\frac{d^2 L(y, F)\\gamma}{d^2 F}=0 \\\\\n",
    "\\therefore \\gamma=-\\frac{\\frac{d L(y, F)}{d F}}{\\frac{d^2 L(y, F)}{d^2 F}} \\\\\n",
    "\\therefore \\gamma = \\frac{y-p}{\\frac{d^2 (-y * l + \\log(1+e^l))}{d^2 l}} \\\\\n",
    "\\therefore \\gamma = \\frac{y-p}{\\frac{d (-y + \\frac{e^l}{1+e^l})}{d l}} \\\\\n",
    "\\therefore \\gamma = \\frac{y-p}{\\frac{d \\frac{e^l}{1+e^l}}{d l}} \\\\\n",
    "$$\n",
    "(The product rule (ab)'=a' b+a b'​)\n",
    "\n",
    "$$\n",
    "\\therefore \\gamma=\\frac{y-p}{\\frac{d e^l}{dl} * \\frac{1}{1+e^l} - e^l * \\frac{d }{d l} \\frac{1}{1+e^l}} \\\\\n",
    "=\\frac{y-p}{\\frac{e^l}{1+e^l}-e^l * \\frac{1}{(1+e^l)^2} \\frac{d}{dl} (1+e^l)} \\\\\n",
    "=\\frac{y-p}{\\frac{e^l}{1+e^l}- \\frac{(e^l)^2}{(1+e^l)^2}} \\\\\n",
    "=\\frac{y-p}{e^l+(e^l)^2-+(e^l)^2} \\\\\n",
    "=\\frac{y-p}{\\frac{e^l}{(1+e^l)^2}} \\\\\n",
    "=\\frac{y-p}{p(1-p)}\n",
    "$$\n",
    "\n",
    "\n",
    "Now we have\n",
    "\n",
    "$$\\gamma = \\frac{\\sum (y-p)}{\\sum p(1-p)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 291,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "import graphviz \n",
    "from sklearn import tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>likes_popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>favorite_color</th>\n",
       "      <th>loves_troll2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brunei</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Candy</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Blue</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>Red</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eric</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Felicity</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name  likes_popcorn  age favorite_color  loves_troll2\n",
       "0      Alex              1   10           Blue             1\n",
       "1    Brunei              1   90          Green             1\n",
       "2     Candy              0   30           Blue             0\n",
       "3     David              1   30            Red             0\n",
       "4      Eric              0   30          Green             1\n",
       "5  Felicity              0   10           Blue             1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>likes_popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>blue</th>\n",
       "      <th>green</th>\n",
       "      <th>red</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   likes_popcorn  age  blue  green  red\n",
       "0              1   10     1      0    0\n",
       "1              1   90     1      0    0\n",
       "2              0   30     1      0    0\n",
       "3              1   30     1      0    0\n",
       "4              0   30     1      0    0\n",
       "5              0   10     1      0    0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df=pd.DataFrame()\n",
    "df['name']=['Alex','Brunei','Candy','David','Eric','Felicity']\n",
    "df['likes_popcorn']=[1,1,0,1,0,0]\n",
    "df['age']=[10,90,30,30,30,10]\n",
    "df['favorite_color']=['Blue','Green','Blue','Red','Green','Blue']\n",
    "df['loves_troll2']=[1,1,0,0,1,1]\n",
    "print('Data:')\n",
    "display(df)\n",
    "\n",
    "X=df[['likes_popcorn','age']].copy()\n",
    "X['blue']=0\n",
    "X.loc[df['favorite_color']=='Blue','blue']=1\n",
    "X['green']=0\n",
    "X.loc[df['favorite_color']=='Green','blue']=1\n",
    "X['red']=0\n",
    "X.loc[df['favorite_color']=='Red','blue']=1\n",
    "\n",
    "y=df['loves_troll2']\n",
    "print('X:')\n",
    "display(X)\n",
    "\n",
    "n=df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 Log of Odds, Probabilities and Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the log_of_odds is : 0.6931471805599453\n",
      "the probability is : 0.6666666666666666\n",
      "the prediction is : 1\n"
     ]
    }
   ],
   "source": [
    "log_of_odds0=np.log(4 / 2)\n",
    "probability0=np.exp(log_of_odds0)/(np.exp(log_of_odds0)+1)\n",
    "print(f'the log_of_odds is : {log_of_odds0}')\n",
    "print(f'the probability is : {probability0}')\n",
    "predict0=1\n",
    "print(f'the prediction is : 1')\n",
    "n_samples=6\n",
    "\n",
    "loss0=-(y*np.log(probability0)+(1-y)*np.log(1-probability0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 293,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now let's get started\n",
    "n_times = 5\n",
    "learning_rate=0.8\n",
    "log_of_odds=np.zeros([n_times+1,n_samples])\n",
    "\n",
    "residuals = np.zeros([n_times+1,n_samples])\n",
    "loss = [0] * (n_times+1)\n",
    "probabilities = np.zeros([n_times+1,n_samples])\n",
    "predictions = np.zeros([n_times+1,n_samples])\n",
    "score = np.zeros([n_times+1])\n",
    "#calculation\n",
    "log_of_odds[0] = [log_of_odds0] * n_samples\n",
    "probabilities[0] = [probability0] * n_samples\n",
    "predictions[0] = (probabilities[0]>0.5)*1.0\n",
    "score[0]=np.sum(predictions[0]==y) / n_samples\n",
    "residuals[0] = y - probabilities[0]\n",
    "loss[0] = loss0.sum()\n",
    "trees = []\n",
    "gamma_value = np.zeros([n_times+1,8])\n",
    "gamma = np.zeros([n_times+1,n_samples])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 6)"
      ]
     },
     "execution_count": 294,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "residuals.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 6)"
      ]
     },
     "execution_count": 295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probabilities.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {},
   "outputs": [],
   "source": [
    "def iteration(i):\n",
    "    #step 2.1 calculate the residuals\n",
    "    residuals[i] = y - probabilities[i]\n",
    "    #step 2.2 Fit a regression tree\n",
    "    dt = DecisionTreeRegressor(max_depth=1, max_leaf_nodes=3)\n",
    "    dt=dt.fit(X, residuals[i])\n",
    "    \n",
    "    trees.append(dt.tree_)\n",
    "    \n",
    "    #Step 2.3 Calculate gamma\n",
    "    leaf_indeces=dt.apply(X)\n",
    "    print(leaf_indeces)\n",
    "    unique_leaves=np.unique(leaf_indeces)\n",
    "    n_leaf=len(unique_leaves)\n",
    "    #for leaf 1\n",
    "    for ileaf in range(n_leaf):\n",
    "        \n",
    "        leaf_index=unique_leaves[ileaf]\n",
    "        n_leaf=len(leaf_indeces[leaf_indeces==leaf_index])\n",
    "        previous_probability = probabilities[i][leaf_indeces==leaf_index]\n",
    "        denominator = np.sum(previous_probability * (1-previous_probability))\n",
    "        igamma = dt.tree_.value[ileaf+1][0][0] * n_leaf / denominator\n",
    "        gamma_value[i][ileaf]=igamma\n",
    "        print(f'for leaf {leaf_index}, we have {n_leaf} related samples. and gamma is {igamma}')\n",
    "\n",
    "    gamma[i] = [gamma_value[i][np.where(unique_leaves==index)] for index in leaf_indeces]\n",
    "    #Step 2.4 Update F(x) \n",
    "    log_of_odds[i+1] = log_of_odds[i] + learning_rate * gamma[i]\n",
    "\n",
    "    probabilities[i+1] = np.array([np.exp(odds)/(np.exp(odds)+1) for odds in log_of_odds[i+1]])\n",
    "    predictions[i+1] = (probabilities[i+1]>0.5)*1.0\n",
    "    score[i+1]=np.sum(predictions[i+1]==y) / n_samples\n",
    "    #residuals[i+1] = y - probabilities[i+1]\n",
    "    loss[i+1]=np.sum(-y * log_of_odds[i+1] + np.log(1+np.exp(log_of_odds[i+1])))\n",
    "    \n",
    "    new_df=df.copy()\n",
    "    new_df.columns=['name', 'popcorn','age','color','y']\n",
    "    new_df[f'$p_{i}$']=probabilities[i]\n",
    "    new_df[f'$l_{i}$']=log_of_odds[i]\n",
    "    new_df[f'$r_{i}$']=residuals[i]\n",
    "    new_df[f'$\\gamma_{i}$']=gamma[i]\n",
    "    new_df[f'$l_{i+1}$']=log_of_odds[i+1]\n",
    "    new_df[f'$p_{i+1}$']=probabilities[i+1]\n",
    "    display(new_df)\n",
    "    \n",
    "    dot_data = tree.export_graphviz(dt, out_file=None, filled=True, rounded=True,feature_names=X.columns) \n",
    "    graph = graphviz.Source(dot_data) \n",
    "    display(graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 299,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 2 2 2 1]\n",
      "for leaf 1, we have 2 related samples. and gamma is 1.5\n",
      "for leaf 2, we have 4 related samples. and gamma is -0.7499999999999998\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>color</th>\n",
       "      <th>y</th>\n",
       "      <th>$p_0$</th>\n",
       "      <th>$l_0$</th>\n",
       "      <th>$r_0$</th>\n",
       "      <th>$\\gamma_0$</th>\n",
       "      <th>$l_1$</th>\n",
       "      <th>$p_1$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1.50</td>\n",
       "      <td>1.893147</td>\n",
       "      <td>0.869114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brunei</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.75</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>0.523270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Candy</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Blue</td>\n",
       "      <td>0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>-0.666667</td>\n",
       "      <td>-0.75</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>0.523270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>Red</td>\n",
       "      <td>0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>-0.666667</td>\n",
       "      <td>-0.75</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>0.523270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eric</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>-0.75</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>0.523270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Felicity</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1.50</td>\n",
       "      <td>1.893147</td>\n",
       "      <td>0.869114</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name  popcorn  age  color  y     $p_0$     $l_0$     $r_0$  $\\gamma_0$  \\\n",
       "0      Alex        1   10   Blue  1  0.666667  0.693147  0.333333        1.50   \n",
       "1    Brunei        1   90  Green  1  0.666667  0.693147  0.333333       -0.75   \n",
       "2     Candy        0   30   Blue  0  0.666667  0.693147 -0.666667       -0.75   \n",
       "3     David        1   30    Red  0  0.666667  0.693147 -0.666667       -0.75   \n",
       "4      Eric        0   30  Green  1  0.666667  0.693147  0.333333       -0.75   \n",
       "5  Felicity        0   10   Blue  1  0.666667  0.693147  0.333333        1.50   \n",
       "\n",
       "      $l_1$     $p_1$  \n",
       "0  1.893147  0.869114  \n",
       "1  0.093147  0.523270  \n",
       "2  0.093147  0.523270  \n",
       "3  0.093147  0.523270  \n",
       "4  0.093147  0.523270  \n",
       "5  1.893147  0.869114  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: Tree Pages: 1 -->\r\n",
       "<svg width=\"236pt\" height=\"165pt\"\r\n",
       " viewBox=\"0.00 0.00 235.50 165.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 161)\">\r\n",
       "<title>Tree</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-161 231.5,-161 231.5,4 -4,4\"/>\r\n",
       "<!-- 0 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
       "<path fill=\"#f6d5bd\" stroke=\"black\" d=\"M148.5,-157C148.5,-157 75.5,-157 75.5,-157 69.5,-157 63.5,-151 63.5,-145 63.5,-145 63.5,-101 63.5,-101 63.5,-95 69.5,-89 75.5,-89 75.5,-89 148.5,-89 148.5,-89 154.5,-89 160.5,-95 160.5,-101 160.5,-101 160.5,-145 160.5,-145 160.5,-151 154.5,-157 148.5,-157\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">age &lt;= 20.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.222</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 6</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.0</text>\r\n",
       "</g>\r\n",
       "<!-- 1 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M90,-53C90,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,-0 12,-0 12,-0 90,-0 90,-0 96,-0 102,-6 102,-12 102,-12 102,-41 102,-41 102,-47 96,-53 90,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 2</text>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.333</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;1 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.6652,-88.9485C84.9963,-80.1664 78.8618,-70.6629 73.1504,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"75.977,-59.7402 67.6131,-53.2367 70.0959,-63.5365 75.977,-59.7402\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"62.3481\" y=\"-73.976\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\r\n",
       "</g>\r\n",
       "<!-- 2 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M215.5,-53C215.5,-53 132.5,-53 132.5,-53 126.5,-53 120.5,-47 120.5,-41 120.5,-41 120.5,-12 120.5,-12 120.5,-6 126.5,-0 132.5,-0 132.5,-0 215.5,-0 215.5,-0 221.5,-0 227.5,-6 227.5,-12 227.5,-12 227.5,-41 227.5,-41 227.5,-47 221.5,-53 215.5,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.25</text>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 4</text>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = &#45;0.167</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;2 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M133.685,-88.9485C139.446,-80.1664 145.681,-70.6629 151.486,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"154.555,-63.5178 157.115,-53.2367 148.703,-59.6779 154.555,-63.5178\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"162.198\" y=\"-74.0145\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1f165fa0688>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iteration(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Iteration 0, Let's look at each step.\n",
    "\n",
    "In Step 2.1, We calculate residuals, that is $y-p_0$. \n",
    "\n",
    "In step 2.2, we fit a regression tree as above.\n",
    "\n",
    "In step 2.3, we calculate $\\gamma$. \n",
    "\n",
    "  * For the first leaf, we have two samples (Alex and Felicity). $\\gamma$ is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 300,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.4999999999999998"
      ]
     },
     "execution_count": 300,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1/3+1/3)/((1-2/3)*2/3+(1-2/3)*2/3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  * For the second leaf, we have four samples. $\\gamma$ is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 301,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.75"
      ]
     },
     "execution_count": 301,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1/3-2/3-2/3+1/3)/(4*(1-2/3)*2/3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Step 2.4, F(x) is updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 302,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 1 1 1 1]\n",
      "for leaf 1, we have 5 related samples. and gamma is -0.31564962030401844\n",
      "for leaf 2, we have 1 related samples. and gamma is 1.9110594001952543\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>color</th>\n",
       "      <th>y</th>\n",
       "      <th>$p_1$</th>\n",
       "      <th>$l_1$</th>\n",
       "      <th>$r_1$</th>\n",
       "      <th>$\\gamma_1$</th>\n",
       "      <th>$l_2$</th>\n",
       "      <th>$p_2$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.869114</td>\n",
       "      <td>1.893147</td>\n",
       "      <td>0.130886</td>\n",
       "      <td>-0.315650</td>\n",
       "      <td>1.640627</td>\n",
       "      <td>0.837620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brunei</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.523270</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>0.476730</td>\n",
       "      <td>1.911059</td>\n",
       "      <td>1.621995</td>\n",
       "      <td>0.835070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Candy</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Blue</td>\n",
       "      <td>0</td>\n",
       "      <td>0.523270</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>-0.523270</td>\n",
       "      <td>-0.315650</td>\n",
       "      <td>-0.159373</td>\n",
       "      <td>0.460241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>Red</td>\n",
       "      <td>0</td>\n",
       "      <td>0.523270</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>-0.523270</td>\n",
       "      <td>-0.315650</td>\n",
       "      <td>-0.159373</td>\n",
       "      <td>0.460241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eric</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.523270</td>\n",
       "      <td>0.093147</td>\n",
       "      <td>0.476730</td>\n",
       "      <td>-0.315650</td>\n",
       "      <td>-0.159373</td>\n",
       "      <td>0.460241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Felicity</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.869114</td>\n",
       "      <td>1.893147</td>\n",
       "      <td>0.130886</td>\n",
       "      <td>-0.315650</td>\n",
       "      <td>1.640627</td>\n",
       "      <td>0.837620</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name  popcorn  age  color  y     $p_1$     $l_1$     $r_1$  $\\gamma_1$  \\\n",
       "0      Alex        1   10   Blue  1  0.869114  1.893147  0.130886   -0.315650   \n",
       "1    Brunei        1   90  Green  1  0.523270  0.093147  0.476730    1.911059   \n",
       "2     Candy        0   30   Blue  0  0.523270  0.093147 -0.523270   -0.315650   \n",
       "3     David        1   30    Red  0  0.523270  0.093147 -0.523270   -0.315650   \n",
       "4      Eric        0   30  Green  1  0.523270  0.093147  0.476730   -0.315650   \n",
       "5  Felicity        0   10   Blue  1  0.869114  1.893147  0.130886   -0.315650   \n",
       "\n",
       "      $l_2$     $p_2$  \n",
       "0  1.640627  0.837620  \n",
       "1  1.621995  0.835070  \n",
       "2 -0.159373  0.460241  \n",
       "3 -0.159373  0.460241  \n",
       "4 -0.159373  0.460241  \n",
       "5  1.640627  0.837620  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: Tree Pages: 1 -->\r\n",
       "<svg width=\"236pt\" height=\"165pt\"\r\n",
       " viewBox=\"0.00 0.00 235.50 165.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 161)\">\r\n",
       "<title>Tree</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-161 231.5,-161 231.5,4 -4,4\"/>\r\n",
       "<!-- 0 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
       "<path fill=\"#fbeade\" stroke=\"black\" d=\"M153.5,-157C153.5,-157 75.5,-157 75.5,-157 69.5,-157 63.5,-151 63.5,-145 63.5,-145 63.5,-101 63.5,-101 63.5,-95 69.5,-89 75.5,-89 75.5,-89 153.5,-89 153.5,-89 159.5,-89 165.5,-95 165.5,-101 165.5,-101 165.5,-145 165.5,-145 165.5,-151 159.5,-157 153.5,-157\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"114.5\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">age &lt;= 60.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"114.5\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.172</text>\r\n",
       "<text text-anchor=\"middle\" x=\"114.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 6</text>\r\n",
       "<text text-anchor=\"middle\" x=\"114.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.028</text>\r\n",
       "</g>\r\n",
       "<!-- 1 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M95,-53C95,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,-0 12,-0 12,-0 95,-0 95,-0 101,-0 107,-6 107,-12 107,-12 107,-41 107,-41 107,-47 101,-53 95,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"53.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.158</text>\r\n",
       "<text text-anchor=\"middle\" x=\"53.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 5</text>\r\n",
       "<text text-anchor=\"middle\" x=\"53.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = &#45;0.062</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;1 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M93.1652,-88.9485C87.4963,-80.1664 81.3618,-70.6629 75.6504,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"78.477,-59.7402 70.1131,-53.2367 72.5959,-63.5365 78.477,-59.7402\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"64.8481\" y=\"-73.976\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\r\n",
       "</g>\r\n",
       "<!-- 2 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M215.5,-53C215.5,-53 137.5,-53 137.5,-53 131.5,-53 125.5,-47 125.5,-41 125.5,-41 125.5,-12 125.5,-12 125.5,-6 131.5,-0 137.5,-0 137.5,-0 215.5,-0 215.5,-0 221.5,-0 227.5,-6 227.5,-12 227.5,-12 227.5,-41 227.5,-41 227.5,-47 221.5,-53 215.5,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"176.5\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"176.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 1</text>\r\n",
       "<text text-anchor=\"middle\" x=\"176.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.477</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;2 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M136.185,-88.9485C141.946,-80.1664 148.181,-70.6629 153.986,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"157.055,-63.5178 159.615,-53.2367 151.203,-59.6779 157.055,-63.5178\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"164.698\" y=\"-74.0145\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1f1661b8188>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iteration(1)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Iteration 2, we have two leaves. <br />\n",
    "For Leaf 1, there are 5 samples. And $\\gamma$ is\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 303,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.3156498224562022"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sum_of_residual=0.130886+-0.523270+-0.523270+0.476730+0.130886\n",
    "denominator=2*0.869114*(1-0.869114)+3*0.523270*(1-0.523270)\n",
    "display(sum_of_residual/denominator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Leaf 2, there is only 1 sample. And $\\gamma$ is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 304,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.9110593001700842"
      ]
     },
     "execution_count": 304,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "0.476730/(0.523270*(1-0.523270))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 305,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 2 2 2 1]\n",
      "for leaf 1, we have 2 related samples. and gamma is 1.1938583610371274\n",
      "for leaf 2, we have 4 related samples. and gamma is -0.24439014590890917\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>color</th>\n",
       "      <th>y</th>\n",
       "      <th>$p_2$</th>\n",
       "      <th>$l_2$</th>\n",
       "      <th>$r_2$</th>\n",
       "      <th>$\\gamma_2$</th>\n",
       "      <th>$l_3$</th>\n",
       "      <th>$p_3$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.837620</td>\n",
       "      <td>1.640627</td>\n",
       "      <td>0.162380</td>\n",
       "      <td>1.193858</td>\n",
       "      <td>2.595714</td>\n",
       "      <td>0.930585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brunei</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.835070</td>\n",
       "      <td>1.621995</td>\n",
       "      <td>0.164930</td>\n",
       "      <td>-0.244390</td>\n",
       "      <td>1.426483</td>\n",
       "      <td>0.806353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Candy</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Blue</td>\n",
       "      <td>0</td>\n",
       "      <td>0.460241</td>\n",
       "      <td>-0.159373</td>\n",
       "      <td>-0.460241</td>\n",
       "      <td>-0.244390</td>\n",
       "      <td>-0.354885</td>\n",
       "      <td>0.412198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>Red</td>\n",
       "      <td>0</td>\n",
       "      <td>0.460241</td>\n",
       "      <td>-0.159373</td>\n",
       "      <td>-0.460241</td>\n",
       "      <td>-0.244390</td>\n",
       "      <td>-0.354885</td>\n",
       "      <td>0.412198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eric</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.460241</td>\n",
       "      <td>-0.159373</td>\n",
       "      <td>0.539759</td>\n",
       "      <td>-0.244390</td>\n",
       "      <td>-0.354885</td>\n",
       "      <td>0.412198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Felicity</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.837620</td>\n",
       "      <td>1.640627</td>\n",
       "      <td>0.162380</td>\n",
       "      <td>1.193858</td>\n",
       "      <td>2.595714</td>\n",
       "      <td>0.930585</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name  popcorn  age  color  y     $p_2$     $l_2$     $r_2$  $\\gamma_2$  \\\n",
       "0      Alex        1   10   Blue  1  0.837620  1.640627  0.162380    1.193858   \n",
       "1    Brunei        1   90  Green  1  0.835070  1.621995  0.164930   -0.244390   \n",
       "2     Candy        0   30   Blue  0  0.460241 -0.159373 -0.460241   -0.244390   \n",
       "3     David        1   30    Red  0  0.460241 -0.159373 -0.460241   -0.244390   \n",
       "4      Eric        0   30  Green  1  0.460241 -0.159373  0.539759   -0.244390   \n",
       "5  Felicity        0   10   Blue  1  0.837620  1.640627  0.162380    1.193858   \n",
       "\n",
       "      $l_3$     $p_3$  \n",
       "0  2.595714  0.930585  \n",
       "1  1.426483  0.806353  \n",
       "2 -0.354885  0.412198  \n",
       "3 -0.354885  0.412198  \n",
       "4 -0.354885  0.412198  \n",
       "5  2.595714  0.930585  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: Tree Pages: 1 -->\r\n",
       "<svg width=\"236pt\" height=\"165pt\"\r\n",
       " viewBox=\"0.00 0.00 235.50 165.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 161)\">\r\n",
       "<title>Tree</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-161 231.5,-161 231.5,4 -4,4\"/>\r\n",
       "<!-- 0 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
       "<path fill=\"#f6d5bd\" stroke=\"black\" d=\"M151,-157C151,-157 73,-157 73,-157 67,-157 61,-151 61,-145 61,-145 61,-101 61,-101 61,-95 67,-89 73,-89 73,-89 151,-89 151,-89 157,-89 163,-95 163,-101 163,-101 163,-145 163,-145 163,-151 157,-157 151,-157\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">age &lt;= 20.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.132</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 6</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.018</text>\r\n",
       "</g>\r\n",
       "<!-- 1 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M90,-53C90,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,-0 12,-0 12,-0 90,-0 90,-0 96,-0 102,-6 102,-12 102,-12 102,-41 102,-41 102,-47 96,-53 90,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 2</text>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.162</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;1 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.6652,-88.9485C84.9963,-80.1664 78.8618,-70.6629 73.1504,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"75.977,-59.7402 67.6131,-53.2367 70.0959,-63.5365 75.977,-59.7402\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"62.3481\" y=\"-73.976\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\r\n",
       "</g>\r\n",
       "<!-- 2 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M215.5,-53C215.5,-53 132.5,-53 132.5,-53 126.5,-53 120.5,-47 120.5,-41 120.5,-41 120.5,-12 120.5,-12 120.5,-6 126.5,-0 132.5,-0 132.5,-0 215.5,-0 215.5,-0 221.5,-0 227.5,-6 227.5,-12 227.5,-12 227.5,-41 227.5,-41 227.5,-47 221.5,-53 215.5,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.183</text>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 4</text>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = &#45;0.054</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;2 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M133.685,-88.9485C139.446,-80.1664 145.681,-70.6629 151.486,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"154.555,-63.5178 157.115,-53.2367 148.703,-59.6779 154.555,-63.5178\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"162.198\" y=\"-74.0145\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1f165db6088>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iteration(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 306,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 1 1 1 1]\n",
      "for leaf 1, we have 5 related samples. and gamma is -0.11420357619360062\n",
      "for leaf 2, we have 1 related samples. and gamma is 1.2401521534064346\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>color</th>\n",
       "      <th>y</th>\n",
       "      <th>$p_3$</th>\n",
       "      <th>$l_3$</th>\n",
       "      <th>$r_3$</th>\n",
       "      <th>$\\gamma_3$</th>\n",
       "      <th>$l_4$</th>\n",
       "      <th>$p_4$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.930585</td>\n",
       "      <td>2.595714</td>\n",
       "      <td>0.069415</td>\n",
       "      <td>-0.114204</td>\n",
       "      <td>2.504351</td>\n",
       "      <td>0.924446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brunei</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.806353</td>\n",
       "      <td>1.426483</td>\n",
       "      <td>0.193647</td>\n",
       "      <td>1.240152</td>\n",
       "      <td>2.418604</td>\n",
       "      <td>0.918235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Candy</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Blue</td>\n",
       "      <td>0</td>\n",
       "      <td>0.412198</td>\n",
       "      <td>-0.354885</td>\n",
       "      <td>-0.412198</td>\n",
       "      <td>-0.114204</td>\n",
       "      <td>-0.446247</td>\n",
       "      <td>0.390253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>Red</td>\n",
       "      <td>0</td>\n",
       "      <td>0.412198</td>\n",
       "      <td>-0.354885</td>\n",
       "      <td>-0.412198</td>\n",
       "      <td>-0.114204</td>\n",
       "      <td>-0.446247</td>\n",
       "      <td>0.390253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eric</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.412198</td>\n",
       "      <td>-0.354885</td>\n",
       "      <td>0.587802</td>\n",
       "      <td>-0.114204</td>\n",
       "      <td>-0.446247</td>\n",
       "      <td>0.390253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Felicity</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.930585</td>\n",
       "      <td>2.595714</td>\n",
       "      <td>0.069415</td>\n",
       "      <td>-0.114204</td>\n",
       "      <td>2.504351</td>\n",
       "      <td>0.924446</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name  popcorn  age  color  y     $p_3$     $l_3$     $r_3$  $\\gamma_3$  \\\n",
       "0      Alex        1   10   Blue  1  0.930585  2.595714  0.069415   -0.114204   \n",
       "1    Brunei        1   90  Green  1  0.806353  1.426483  0.193647    1.240152   \n",
       "2     Candy        0   30   Blue  0  0.412198 -0.354885 -0.412198   -0.114204   \n",
       "3     David        1   30    Red  0  0.412198 -0.354885 -0.412198   -0.114204   \n",
       "4      Eric        0   30  Green  1  0.412198 -0.354885  0.587802   -0.114204   \n",
       "5  Felicity        0   10   Blue  1  0.930585  2.595714  0.069415   -0.114204   \n",
       "\n",
       "      $l_4$     $p_4$  \n",
       "0  2.504351  0.924446  \n",
       "1  2.418604  0.918235  \n",
       "2 -0.446247  0.390253  \n",
       "3 -0.446247  0.390253  \n",
       "4 -0.446247  0.390253  \n",
       "5  2.504351  0.924446  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: Tree Pages: 1 -->\r\n",
       "<svg width=\"226pt\" height=\"165pt\"\r\n",
       " viewBox=\"0.00 0.00 226.00 165.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 161)\">\r\n",
       "<title>Tree</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-161 222,-161 222,4 -4,4\"/>\r\n",
       "<!-- 0 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
       "<path fill=\"#fbeade\" stroke=\"black\" d=\"M147,-157C147,-157 69,-157 69,-157 63,-157 57,-151 57,-145 57,-145 57,-101 57,-101 57,-95 63,-89 69,-89 69,-89 147,-89 147,-89 153,-89 159,-95 159,-101 159,-101 159,-145 159,-145 159,-151 153,-157 147,-157\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"108\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">age &lt;= 60.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"108\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.122</text>\r\n",
       "<text text-anchor=\"middle\" x=\"108\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 6</text>\r\n",
       "<text text-anchor=\"middle\" x=\"108\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.016</text>\r\n",
       "</g>\r\n",
       "<!-- 1 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M86,-53C86,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,-0 12,-0 12,-0 86,-0 86,-0 92,-0 98,-6 98,-12 98,-12 98,-41 98,-41 98,-47 92,-53 86,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.139</text>\r\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 5</text>\r\n",
       "<text text-anchor=\"middle\" x=\"49\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = &#45;0.02</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;1 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M87.3647,-88.9485C81.8817,-80.1664 75.9483,-70.6629 70.4242,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"73.3333,-59.8656 65.0684,-53.2367 67.3955,-63.5728 73.3333,-59.8656\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"59.4353\" y=\"-73.8938\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\r\n",
       "</g>\r\n",
       "<!-- 2 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M206,-53C206,-53 128,-53 128,-53 122,-53 116,-47 116,-41 116,-41 116,-12 116,-12 116,-6 122,-0 128,-0 128,-0 206,-0 206,-0 212,-0 218,-6 218,-12 218,-12 218,-41 218,-41 218,-47 212,-53 206,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"167\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = &#45;0.0</text>\r\n",
       "<text text-anchor=\"middle\" x=\"167\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 1</text>\r\n",
       "<text text-anchor=\"middle\" x=\"167\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.194</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;2 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M128.635,-88.9485C134.118,-80.1664 140.052,-70.6629 145.576,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"148.604,-63.5728 150.932,-53.2367 142.667,-59.8656 148.604,-63.5728\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"156.565\" y=\"-73.8938\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1f1661bbbc8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iteration(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 2 1 2 1 1]\n",
      "for leaf 1, we have 3 related samples. and gamma is 0.5406201302738854\n",
      "for leaf 2, we have 3 related samples. and gamma is -0.6083743125192569\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>popcorn</th>\n",
       "      <th>age</th>\n",
       "      <th>color</th>\n",
       "      <th>y</th>\n",
       "      <th>$p_4$</th>\n",
       "      <th>$l_4$</th>\n",
       "      <th>$r_4$</th>\n",
       "      <th>$\\gamma_4$</th>\n",
       "      <th>$l_5$</th>\n",
       "      <th>$p_5$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alex</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.924446</td>\n",
       "      <td>2.504351</td>\n",
       "      <td>0.075554</td>\n",
       "      <td>-0.608374</td>\n",
       "      <td>2.017652</td>\n",
       "      <td>0.882638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Brunei</td>\n",
       "      <td>1</td>\n",
       "      <td>90</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.918235</td>\n",
       "      <td>2.418604</td>\n",
       "      <td>0.081765</td>\n",
       "      <td>-0.608374</td>\n",
       "      <td>1.931905</td>\n",
       "      <td>0.873460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Candy</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Blue</td>\n",
       "      <td>0</td>\n",
       "      <td>0.390253</td>\n",
       "      <td>-0.446247</td>\n",
       "      <td>-0.390253</td>\n",
       "      <td>0.540620</td>\n",
       "      <td>-0.013751</td>\n",
       "      <td>0.496562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>Red</td>\n",
       "      <td>0</td>\n",
       "      <td>0.390253</td>\n",
       "      <td>-0.446247</td>\n",
       "      <td>-0.390253</td>\n",
       "      <td>-0.608374</td>\n",
       "      <td>-0.932947</td>\n",
       "      <td>0.282327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Eric</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>Green</td>\n",
       "      <td>1</td>\n",
       "      <td>0.390253</td>\n",
       "      <td>-0.446247</td>\n",
       "      <td>0.609747</td>\n",
       "      <td>0.540620</td>\n",
       "      <td>-0.013751</td>\n",
       "      <td>0.496562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Felicity</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>Blue</td>\n",
       "      <td>1</td>\n",
       "      <td>0.924446</td>\n",
       "      <td>2.504351</td>\n",
       "      <td>0.075554</td>\n",
       "      <td>0.540620</td>\n",
       "      <td>2.936847</td>\n",
       "      <td>0.949638</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name  popcorn  age  color  y     $p_4$     $l_4$     $r_4$  $\\gamma_4$  \\\n",
       "0      Alex        1   10   Blue  1  0.924446  2.504351  0.075554   -0.608374   \n",
       "1    Brunei        1   90  Green  1  0.918235  2.418604  0.081765   -0.608374   \n",
       "2     Candy        0   30   Blue  0  0.390253 -0.446247 -0.390253    0.540620   \n",
       "3     David        1   30    Red  0  0.390253 -0.446247 -0.390253   -0.608374   \n",
       "4      Eric        0   30  Green  1  0.390253 -0.446247  0.609747    0.540620   \n",
       "5  Felicity        0   10   Blue  1  0.924446  2.504351  0.075554    0.540620   \n",
       "\n",
       "      $l_5$     $p_5$  \n",
       "0  2.017652  0.882638  \n",
       "1  1.931905  0.873460  \n",
       "2 -0.013751  0.496562  \n",
       "3 -0.932947  0.282327  \n",
       "4 -0.013751  0.496562  \n",
       "5  2.936847  0.949638  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: Tree Pages: 1 -->\r\n",
       "<svg width=\"236pt\" height=\"165pt\"\r\n",
       " viewBox=\"0.00 0.00 235.50 165.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 161)\">\r\n",
       "<title>Tree</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-161 231.5,-161 231.5,4 -4,4\"/>\r\n",
       "<!-- 0 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
       "<path fill=\"#f2c09c\" stroke=\"black\" d=\"M175,-157C175,-157 49,-157 49,-157 43,-157 37,-151 37,-145 37,-145 37,-101 37,-101 37,-95 43,-89 49,-89 49,-89 175,-89 175,-89 181,-89 187,-95 187,-101 187,-101 187,-145 187,-145 187,-151 181,-157 175,-157\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-141.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">likes_popcorn &lt;= 0.5</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-126.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.116</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 6</text>\r\n",
       "<text text-anchor=\"middle\" x=\"112\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.01</text>\r\n",
       "</g>\r\n",
       "<!-- 1 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
       "<path fill=\"#e58139\" stroke=\"black\" d=\"M90,-53C90,-53 12,-53 12,-53 6,-53 0,-47 0,-41 0,-41 0,-12 0,-12 0,-6 6,-0 12,-0 12,-0 90,-0 90,-0 96,-0 102,-6 102,-12 102,-12 102,-41 102,-41 102,-47 96,-53 90,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.167</text>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 3</text>\r\n",
       "<text text-anchor=\"middle\" x=\"51\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = 0.098</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;1 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M90.6652,-88.9485C84.9963,-80.1664 78.8618,-70.6629 73.1504,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"75.977,-59.7402 67.6131,-53.2367 70.0959,-63.5365 75.977,-59.7402\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"62.3481\" y=\"-73.976\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\r\n",
       "</g>\r\n",
       "<!-- 2 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
       "<path fill=\"#ffffff\" stroke=\"black\" d=\"M215.5,-53C215.5,-53 132.5,-53 132.5,-53 126.5,-53 120.5,-47 120.5,-41 120.5,-41 120.5,-12 120.5,-12 120.5,-6 126.5,-0 132.5,-0 132.5,-0 215.5,-0 215.5,-0 221.5,-0 227.5,-6 227.5,-12 227.5,-12 227.5,-41 227.5,-41 227.5,-47 221.5,-53 215.5,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-37.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">mse = 0.049</text>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">samples = 3</text>\r\n",
       "<text text-anchor=\"middle\" x=\"174\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">value = &#45;0.078</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;2 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M133.685,-88.9485C139.446,-80.1664 145.681,-70.6629 151.486,-61.815\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"154.555,-63.5178 157.115,-53.2367 148.703,-59.6779 154.555,-63.5178\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"162.198\" y=\"-74.0145\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1f165da3fc8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iteration(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 308,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAbEklEQVR4nO3de5Bc9Xnm8e8joQsgQBIaCVkSSKBBF9trYY+1iTEkCIbIXi+wtS5Hqji2t1xhs2WcxE45hlqHsKxdtfln2doUSRVxsNeXoLDErFVreaWWBfgSYTQy4qIeJAYJrEFSz4Du1nWkd//o09Aa9UgtTfec7j7Pp6pr5vz6d47eUzD9dP/69NuKCMzMLHtGpV2AmZmlwwFgZpZRDgAzs4xyAJiZZZQDwMwsoy5Ku4DzMWXKlJg9e3baZZiZNZWNGze+FRFtg8ebKgBmz55NV1dX2mWYmTUVSW9UGvcSkJlZRjkAzMwyygFgZpZRDgAzs4xyAJiZZZQDwMwsoxwAZmYZ1VSfAzA7mx9uepPX+g6lXYZZXXz2I7O5csK4mh7TAWAtof/gMf7snzYRAVLa1ZjV3h2LZjgAzCpZ90qBCPjRn3yU977nirTLMWsKfg/AWkIuX2DGxItZOP3ytEsxaxoOAGt6h48P8LNX36Jz4TTk9R+zqjkArOn97NW3ODZwis6F09IuxaypOACs6eXyBS4bfxGL50xOuxSzpuIAsKZ28lSw7pU+bpk3lTGj/b+z2fmo6i9G0lJJWyT1SLq3wv1XS3pK0vOSXpT08WS8U9JGSS8lP5eU7fN0csxNyW1q7U7LsuJXv97Lnt8c9/KP2QU452WgkkYDDwOdQC+wQdLKiMiXTfsa8HhE/J2khcAqYDbwFvBvI2KnpPcBq4EZZfv9QUT4G17sguXyBcaMFr8774wvOzKzc6jmFcBioCcitkXEcWAFcOegOQGUrr+7AtgJEBHPR8TOZHwzMF5SbT/JYJkVEeTyBX7r2iu5bPyYtMsxazrVBMAMYEfZdi+nP4sHeAD4tKReis/+v1jhOP8eeD4ijpWNfStZ/vlLDXH9nqS7JXVJ6urv76+iXMuK1/oPsf2t33C7l3/MLkg1AVDpgTkGbS8Hvh0RM4GPA9+V9M6xJb0X+GvgP5bt8wcR8X7gpuT2h5X+8Yh4JCI6IqKjrc0v8+1da/IFAG5d4AAwuxDVBEAvMKtseybJEk+ZzwOPA0TEemA8MAVA0kzgSeAzEfFaaYeIeDP5eRD4R4pLTWZVW5sv8L4Zl/OeiRenXYpZU6omADYA7ZLmSBoLLANWDprza+BWAEkLKAZAv6SJwI+A+yLiF6XJki6SVAqIMcAngJeHezKWHf0Hj/H8jn10Lrgq7VLMmtY5AyAiBoB7KF7B003xap/Nkh6UdEcy7c+BP5L0AvAY8LmIiGS/ucBfDrrccxywWtKLwCbgTeDva31y1rp+0l1s/ubLP80uXFXdQCNiFcU3d8vH7i/7PQ/cWGG/rwNfH+KwH6q+TLPTlZq/LZh+WdqlmDUtf3TSms7h4wP8vMfN38yGywFgTeenW938zawWHADWdNZ2F7jczd/Mhs0BYE3lneZv8938zWy4/BdkTWXjG27+ZlYrDgBrKrn8bsaMFr9zvT8VbjZcDgBrGqXmb7993RQ3fzOrAQeANY2evkO8/vZhOhf4qyPMasEBYE0j111s/nab1//NasIBYE0jly/w/hlXMP0KN38zqwUHgDWFvoNH2bRjn6/+MashB4A1hZ9097n5m1mNOQCsKeTyBWZOupj5V7n5m1mtOACs4f3mmJu/mdWDA8Aa3s9efYvjA6fo9Fc/mtWUA8AaXi5fbP72YTd/M6spB4A1tIGTp1j3SoElbv5mVnP+i7KGtvGNvew9fILOhf7uX7NacwBYQ8vlC4wdPYrfmefmb2a15gCwhhUR5LoL/PZ1VzJhXFVfX21m58EBYA2rp+8Qb7x92L1/zOqkqgCQtFTSFkk9ku6tcP/Vkp6S9LykFyV9vOy++5L9tkj6vWqPabYmX2z+5ss/zerjnAEgaTTwMPAxYCGwXNLCQdO+BjweETcAy4C/TfZdmGy/F1gK/K2k0VUe0zIuly/wr2ZewVVXjE+7FLOWVM0rgMVAT0Rsi4jjwArgzkFzArg8+f0KYGfy+53Aiog4FhHbgZ7keNUc0zKs70DS/M3P/s3qppoAmAHsKNvuTcbKPQB8WlIvsAr44jn2reaYAEi6W1KXpK7+/v4qyrVWsLa7D4DO9zoAzOqlmgCo1HwlBm0vB74dETOBjwPflTTqLPtWc8ziYMQjEdERER1tbb4UMCty+d3Mmnwx86a5+ZtZvVQTAL3ArLLtmby7xFPyeeBxgIhYD4wHppxl32qOaRn1m2MD/OK1t7ltgZu/mdVTNQGwAWiXNEfSWIpv6q4cNOfXwK0AkhZQDID+ZN4ySeMkzQHageeqPKZl1M9e7S82f/Pln2Z1dc5P10TEgKR7gNXAaODRiNgs6UGgKyJWAn8O/L2kL1FcyvlcRASwWdLjQB4YAL4QEScBKh2zDudnTWhNvsAVF49h8Ww3fzOrJxUfp5tDR0dHdHV1pV2G1dHAyVN0fGMtt8ybykO/vyjtcsxagqSNEdExeNyfBLaG0vXGXvYdPuHlH7MR4ACwhlJq/nbz9b7iy6zeHADWMCKCtW7+ZjZiHADWMF5Nmr95+cdsZDgArGHkSs3fHABmI8IBYA1jTb7AB2ZewbTL3fzNbCQ4AKwhFA4c5YUd+/zs32wEOQCsIfyk1PzN3/1rNmIcANYQSs3frp82Ie1SzDLDAWCpKzV/61xwlZu/mY0gB4Cl7qdb3fzNLA0OAEtdLl9g4iVj+PDsSWmXYpYpDgBL1cDJU6zb0seSeVO5aLT/dzQbSf6Ls1S5+ZtZehwAlqpS87eb3PzNbMQ5ACw1EUEuX+Ajc938zSwNDgBLzdbCIX69x83fzNLiALDU5PK7AbhtgQPALA0OAEtNLl/gA7MmuvmbWUocAJaKwoGjvNC7n9u9/GOWmqoCQNJSSVsk9Ui6t8L9D0nalNy2StqXjN9SNr5J0lFJdyX3fVvS9rL7/A3gGbK2u9j738s/Zuk556UXkkYDDwOdQC+wQdLKiMiX5kTEl8rmfxG4IRl/CliUjE8GeoA1ZYf/SkQ8UYPzsCaTyxe4evIlbv5mlqJqXgEsBnoiYltEHAdWAHeeZf5y4LEK458EfhwRh8+/TGslh44N8C89b9O5cJqbv5mlqJoAmAHsKNvuTcbOIOkaYA6wrsLdyzgzGL4h6cVkCWncEMe8W1KXpK7+/v4qyrVG99Ot/Rw/6eZvZmmrJgAqPUWLIeYuA56IiJOnHUCaDrwfWF02fB8wH/gwMBn4aqUDRsQjEdERER1tbf60aCsoNX/ruMbN38zSVE0A9AKzyrZnAjuHmFvpWT7Ap4AnI+JEaSAidkXRMeBbFJearMUNnDzFulf6WDLfzd/M0lbNX+AGoF3SHEljKT7Irxw8SdI8YBKwvsIxznhfIHlVgIqLwHcBL59f6daMNry+l/1HTtDpq3/MUnfOq4AiYkDSPRSXb0YDj0bEZkkPAl0RUQqD5cCKiDhteUjSbIqvIJ4ZdOjvS2qjuMS0Cfjj4ZyINYdcvsDYi0Zxs5u/maWuqg5cEbEKWDVo7P5B2w8Mse/rVHjTOCKWVFuktYaIINe9mxuvu5JL3fzNLHVehLURs6VwkB17jtC58Kq0SzEzHAA2gnKbS5/+nZpyJWYGDgAbQWu7CyyaNZGpbv5m1hAcADYiSs3f/OEvs8bhALARkcsXl38cAGaNwwFgIyKXL3DNlZfQPtXN38wahQPA6u7QsQHWv/Y2nQvc/M2skTgArO6e2eLmb2aNyAFgdbe2u8CkS8bwITd/M2soDgCrqxNJ87db3PzNrOH4L9LqasPre9h/5IS/+9esATkArK5Kzd9uanfzN7NG4wCwuokIcvkCH507xc3fzBqQA8Dq5pXdB+nde8RX/5g1KAeA1c3a5NO/t7r5m1lDcgBY3eRKzd8uc/M3s0bkALC62L3/KC+6+ZtZQ3MAWF3kuovLP77806xxOQCsLnL5ArOvvIS5bv5m1rAcAFZzB4+eYP1rb9G50M3fzBqZA8Bq7qdb3+LEyfB3/5o1uKoCQNJSSVsk9Ui6t8L9D0nalNy2StpXdt/JsvtWlo3PkfRLSa9K+idJY2tzSpa2XH43ky4Zwwevnph2KWZ2FucMAEmjgYeBjwELgeWSFpbPiYgvRcSiiFgE/A3wg7K7j5Tui4g7ysb/GngoItqBvcDnh3ku1gBKzd+WzJ/m5m9mDa6av9DFQE9EbIuI48AK4M6zzF8OPHa2A6q4MLwEeCIZ+l/AXVXUYg1uw/Y9HDg64Ms/zZpANQEwA9hRtt2bjJ1B0jXAHGBd2fB4SV2SnpVUepC/EtgXEQNVHPPuZP+u/v7+Ksq1NK3JFxh30Shuvn5K2qWY2TlU06Gr0mUcMcTcZcATEXGybOzqiNgp6VpgnaSXgAPVHjMiHgEeAejo6Bjq37UGUN787ZKxbv5m1uiqeQXQC8wq254J7Bxi7jIGLf9ExM7k5zbgaeAG4C1goqTSo8TZjmlN4pXdB3lzn5u/mTWLagJgA9CeXLUzluKD/MrBkyTNAyYB68vGJkkal/w+BbgRyEdEAE8Bn0ymfhb44XBOxNKXyxeQ4NYFDgCzZnDOAEjW6e8BVgPdwOMRsVnSg5LKr+pZDqxIHtxLFgBdkl6g+ID/3yIin9z3VeDLknoovifwD8M/HUtTLl9s/tZ22bi0SzGzKlS1UBsRq4BVg8buH7T9QIX9/gV4/xDH3EbxCiNrAbv2H+GlN/fzF0vnpV2KmVXJF2pbTZR6/7v5m1nzcABYTazJF5gz5VKua3PzN7Nm4QCwYTt49ATPbnvbzd/MmowDwIbtma39SfM3L/+YNRMHgA1bLl9g8qVj+eDVk9IuxczOgwPAhuXEyVM89UofS+ZPZfQoL/+YNRMHgA3Lc27+Zta0HAA2LLmk+dtN7W7+ZtZsHAB2wUrN325qd/M3s2bkALAL1r3Lzd/MmpkDwC5YqfnbkvkOALNm5ACwC5br3s0Nbv5m1rQcAHZBdu47wstvHqBz4VVpl2JmF8gBYBfkJ93F5m9e/zdrXg4AuyBr8gWunXIpc6e6+ZtZs3IA2Hk7UNb8zcyalwPAztszW4rN325zAJg1NQeAnbdcvsCVbv5m1vQcAHZeTpw8xVNb3PzNrBU4AOy8PLd9Dwfd/M2sJTgA7Lzk8gXGjxnFTe1taZdiZsNUVQBIWippi6QeSfdWuP8hSZuS21ZJ+5LxRZLWS9os6UVJv1+2z7clbS/bb1HtTsvqodT87aNz27h47Oi0yzGzYTpnC0dJo4GHgU6gF9ggaWVE5EtzIuJLZfO/CNyQbB4GPhMRr0p6D7BR0uqI2Jfc/5WIeKJG52J1lt91gDf3HeFPbp2bdilmVgPVvAJYDPRExLaIOA6sAO48y/zlwGMAEbE1Il5Nft8J9AFeO2hSbv5m1lqqCYAZwI6y7d5k7AySrgHmAOsq3LcYGAu8Vjb8jWRp6CFJFTuKSbpbUpekrv7+/irKtXrJ5Qt88OpJbv5m1iKqCYBK1/rFEHOXAU9ExMnTDiBNB74L/IeIOJUM3wfMBz4MTAa+WumAEfFIRHREREdbm188pGXnviNs3nnAV/+YtZBqAqAXmFW2PRPYOcTcZSTLPyWSLgd+BHwtIp4tjUfErig6BnyL4lKTNai1bv5m1nKqCYANQLukOZLGUnyQXzl4kqR5wCRgfdnYWOBJ4DsR8b8HzZ+e/BRwF/DyhZ6E1V8uX+Datku5rs3N38xaxTkDICIGgHuA1UA38HhEbJb0oKQ7yqYuB1ZERPny0KeAm4HPVbjc8/uSXgJeAqYAX6/B+VgdvNP8bYGf/Zu1kqq+yTsiVgGrBo3dP2j7gQr7fQ/43hDHXFJ1lZaqp5Pmb17+MWst/iSwnVOp+dsNbv5m1lIcAHZWxwdO8fSWPm5d4OZvZq3GAWBn9W7zN3/3r1mrcQDYWeXyuxk/ZhQfnTsl7VLMrMYcADYkN38za20OABvS5p0H2Ln/KLf76h+zluQAsCG90/xtwdS0SzGzOnAA2JDWdhf40NWTmDLBzd/MWpEDwCp6083fzFqeA8AqWpt38zezVucAsIpKzd+udfM3s5blALAz7D+SNH/zs3+zluYAsDM8vaWPgVPhyz/NWpwDwM6wtruPKRPGsmiWm7+ZtTIHgJ3m+MApnn6lj1vnT3PzN7MW5wCw0/xy+9scPDbg9X+zDHAA2Gly+QLjx4ziRjd/M2t5DgB7R0SwNl/gpnY3fzPLAgeAvaPU/M3LP2bZ4ACwd+TyBUYJbp3v5m9mWVBVAEhaKmmLpB5J91a4/yFJm5LbVkn7yu77rKRXk9tny8Y/JOml5Jj/U5IvOUlZLl/gQ9dM4ko3fzPLhHMGgKTRwMPAx4CFwHJJC8vnRMSXImJRRCwC/gb4QbLvZOCvgH8NLAb+SlLp4vK/A+4G2pPb0pqckV2Q3r2Hye9y8zezLKnmFcBioCcitkXEcWAFcOdZ5i8HHkt+/z0gFxF7ImIvkAOWSpoOXB4R6yMigO8Ad13wWdiwlZq/3bbAAWCWFdUEwAxgR9l2bzJ2BknXAHOAdefYd0byezXHvFtSl6Su/v7+Ksq1C5HrLnCdm7+ZZUo1AVBpbT6GmLsMeCIiTp5j36qPGRGPRERHRHS0tbWds1g7f/uPnOCX2/bQufCqtEsxsxFUTQD0ArPKtmcCO4eYu4x3l3/Otm9v8ns1x7Q6KzV/8/q/WbZUEwAbgHZJcySNpfggv3LwJEnzgEnA+rLh1cDtkiYlb/7eDqyOiF3AQUm/lVz98xngh8M8F7tAuXyBKRPGccOsiWmXYmYj6JwBEBEDwD0UH8y7gccjYrOkByXdUTZ1ObAieVO3tO8e4L9SDJENwIPJGMB/Ar4J9ACvAT+uwfnYeTo+cIpntvRz24KpjHLzN7NMuaiaSRGxClg1aOz+QdsPDLHvo8CjFca7gPdVW6jVx7Pbis3ffPWPWfb4k8AZl8sXuHjMaD7a7uZvZlnjAMiwiGBtd4Gb2qcwfoybv5lljQMgwzbvPMAuN38zyywHQIatKTV/8/q/WSY5ADIsly/Qcc1kJl86Nu1SzCwFDoCM2rHnMN27DnDbQrd+NssqB0BGre0uNn9z+wez7HIAZNTa7gJzp05gzpRL0y7FzFLiAMigd5u/+c1fsyxzAGSQm7+ZGTgAMmlNvkDbZeNYNNPN38yyzAGQMccGTrr5m5kBDoDMeXbbHg65+ZuZ4QDInLVJ87cb57r5m1nWOQAypNT87ebr3fzNzBwAmfLym6Xmb/7wl5k5ADIll9/NKMGS+W7/YGYOgExZky/QMdvN38ysyAGQETv2HOaV3Qfp9NU/ZpZwAGTEu83fHABmVuQAyIhcvkD71AnMdvM3M0tUFQCSlkraIqlH0r1DzPmUpLykzZL+MRm7RdKmsttRSXcl931b0vay+xbV7rSs3P7DJ/jldjd/M7PTXXSuCZJGAw8DnUAvsEHSyojIl81pB+4DboyIvZKmAkTEU8CiZM5koAdYU3b4r0TEE7U6GavsqS19nHTzNzMbpJpXAIuBnojYFhHHgRXAnYPm/BHwcETsBYiIvgrH+STw44g4PJyC7fzlkuZvH3DzNzMrU00AzAB2lG33JmPlrgeul/QLSc9KWlrhOMuAxwaNfUPSi5IekjSu0j8u6W5JXZK6+vv7qyjXyh0bOMnTW/rc/M3MzlBNAFR61IhB2xcB7cDvAsuBb0p65+mmpOnA+4HVZfvcB8wHPgxMBr5a6R+PiEcioiMiOtra2qoo18o9u20Pvzl+0ss/ZnaGagKgF5hVtj0T2Flhzg8j4kREbAe2UAyEkk8BT0bEidJAROyKomPAtyguNVmN5fK7uWTsaD5ynZu/mdnpqgmADUC7pDmSxlJcylk5aM7/AW4BkDSF4pLQtrL7lzNo+Sd5VYAkAXcBL1/ICdjQIoK1+T5ubm9z8zczO8M5rwKKiAFJ91BcvhkNPBoRmyU9CHRFxMrkvtsl5YGTFK/ueRtA0myKryCeGXTo70tqo7jEtAn449qckpW89OZ+dh846uUfM6vonAEAEBGrgFWDxu4v+z2ALye3wfu+zplvGhMRS86zVjtPuXzBzd/MbEj+JHALyyXN3ya5+ZuZVeAAaFGl5m+3e/nHzIbgAGhRubybv5nZ2VX1HkCz+89PvsRz2/ekXcaI2n3gKNdPm8A1V7r5m5lVlokAeM/Ei2mfNiHtMkZU+7QJ/LsbZqZdhpk1sEwEwBdumZt2CWZmDcfvAZiZZZQDwMwsoxwAZmYZ5QAwM8soB4CZWUY5AMzMMsoBYGaWUQ4AM7OMUrGTc3OQ1A+8cYG7TwHeqmE5zcDnnA0+59Y33PO9JiLO+E7dpgqA4ZDUFREdadcxknzO2eBzbn31Ol8vAZmZZZQDwMwso7IUAI+kXUAKfM7Z4HNufXU538y8B2BmZqfL0isAMzMr4wAwM8uoTASApKWStkjqkXRv2vXUm6RHJfVJejntWkaCpFmSnpLULWmzpD9Nu6Z6kzRe0nOSXkjO+b+kXdNIkTRa0vOS/m/atYwESa9LeknSJkldNT12q78HIGk0sBXoBHqBDcDyiMinWlgdSboZOAR8JyLel3Y99SZpOjA9In4l6TJgI3BXi/83FnBpRBySNAb4OfCnEfFsyqXVnaQvAx3A5RHxibTrqTdJrwMdEVHzD75l4RXAYqAnIrZFxHFgBXBnyjXVVUT8FNiTdh0jJSJ2RcSvkt8PAt3AjHSrqq8oOpRsjklurf1sDpA0E/g3wDfTrqUVZCEAZgA7yrZ7afEHhyyTNBu4AfhlupXUX7IUsgnoA3IR0fLnDPwP4C+AU2kXMoICWCNpo6S7a3ngLASAKoy1/DOlLJI0Afhn4M8i4kDa9dRbRJyMiEXATGCxpJZe7pP0CaAvIjamXcsIuzEiPgh8DPhCssRbE1kIgF5gVtn2TGBnSrVYnSTr4P8MfD8ifpB2PSMpIvYBTwNLUy6l3m4E7kjWxFcASyR9L92S6i8idiY/+4AnKS5r10QWAmAD0C5pjqSxwDJgZco1WQ0lb4j+A9AdEf897XpGgqQ2SROT3y8GbgNeSbeq+oqI+yJiZkTMpvh3vC4iPp1yWXUl6dLkwgYkXQrcDtTs6r6WD4CIGADuAVZTfHPw8YjYnG5V9SXpMWA9ME9Sr6TPp11Tnd0I/CHFZ4SbktvH0y6qzqYDT0l6keKTnFxEZOKyyIyZBvxc0gvAc8CPIuL/1ergLX8ZqJmZVdbyrwDMzKwyB4CZWUY5AMzMMsoBYGaWUQ4AM7OMcgCYmWWUA8DMLKP+P03oENd+TZq3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(score)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXSdZbn38e+VqWnSKWnSgTZp0oEWKG3aho6KWBGLKCBjK0Jxqvh6GPR4FD2+IqjvQXExeY5ARaSIUgZREGSoMglp06bzSOmQzrRpk6ZN26QZrvePvdsTY9LsNMOzs/P7rLVX936e+8m+9nL5y8OVe9+3uTsiIhK74oIuQERE2peCXkQkxinoRURinIJeRCTGKehFRGJcQtAFNCYjI8NzcnKCLkNEpNNYunTpfnfPbOxcVAZ9Tk4ORUVFQZchItJpmNm2ps6pdSMiEuMU9CIiMU5BLyIS4xT0IiIxTkEvIhLjFPQiIjFOQS8iEuNiJuhr65z/eXMTK3ccDLoUEZGoEjNBX1FVw+8XbeOW+cupqKoJuhwRkagRM0Hfu3si988cx47So/zwhTVBlyMiEjViJugBJuam82/TR/D8sl28sGJX0OWIiESFmAp6gFumDyd/SBr/+ac1bD9wNOhyREQCF3NBnxAfx/0z8zCDW+Yvp7q2LuiSREQC1WzQm1mymS02s5VmttbM7mxkzH1mtiL82GhmB+udq6137sW2/gCNGZyWwt1XjGHFjoPct2BjR7yliEjUimSZ4ipgurtXmFki8K6ZveLui04McPdvnnhuZjcD4+pdf8zd89qs4ghdMmYg72zM4qG3N/OR4RlMHZ7R0SWIiESFZu/oPaQi/DIx/PBTXDILeKoNamu1Oy49m9yMVL75zApKjxwPuhwRkUBE1KM3s3gzWwHsAxa4e2ET44YAucAb9Q4nm1mRmS0ys8tP8R5zwuOKSkpKWvARmpaSlMCDM8dRdqSa7zy3CvdT/X4SEYlNEQW9u9eG2y+DgYlmNrqJoTOB59y9tt6xbHfPBz4P3G9mw5p4j7nunu/u+ZmZje6GdVpGD+rNdy8exd/W7+XJRU1uwCIiErNaNOvG3Q8CbwEzmhgykwZtG3ffHf53S/jacf96Wfv60rQcLhiZyY9fXs+GDw919NuLiAQqklk3mWbWJ/y8O3AhsKGRcSOBNGBhvWNpZtYt/DwDmAasa5vSI2dm/OLqsfRKTuSWp5ZTWV3b/EUiIjEikjv6gcCbZrYKWEKoR/+Smd1lZpfWGzcLmO//3Ag/Cygys5XAm8Dd7t7hQQ+Q0aMb914zlo17K/jJy4GUICISiGanV7r7Khppt7j7Dxu8/lEjYwqAc1tRX5s6/8xM5pw/lLnvbOGjIzL51DkDgi5JRKTdxdw3Y5vz7YtGcu6g3nz3j6vYU34s6HJERNpdlwv6pIQ4Hpw1juM1ddw2fwW1dZpyKSKxrcsFPUBuRip3XnoOhVtLeeitTUGXIyLSrrpk0ANcNWEwnx17Bvf97QOWbisLuhwRkXbTZYPezPjp50YzsHcyt85fzqHK6qBLEhFpF1026AF6JSfy4Kxx7Cmv5D//tEZLJIhITOrSQQ8wPjuNb144gr+s3M1zS3cGXY6ISJvr8kEP8PULhjN5aDp3vLiWLSUVzV8gItKJKOiB+DjjvmvzSEqI49b5Kzheo12pRCR2KOjDBvbuzs+uHMPqXeX84vX3gy5HRKTNKOjr+dQ5A/jC5GzmvrOFdza2zZr4IiJBU9A38INLzubM/j341jMr2V9RFXQ5IiKtpqBvIDkxngdnjeNQZTXffnYldVoiQUQ6OQV9I0YN6MUPLjmLt94v4bcFxUGXIyLSKgr6Jlw/eQgXntWfn72ygTW7yoMuR0TktCnom2Bm/PyqMaSlJnLL/OUcPV4TdEkiIqdFQX8K6alJ3HdtHlv3H+HOF7UrlYh0Tgr6ZkwdlsHXPzaMp4t28PKqPUGXIyLSYpFsDp5sZovNbKWZrTWzOxsZc6OZlZjZivDjK/XOzTazD8KP2W39ATrCNz95JnlZfbj9+VXsLDsadDkiIi0SyR19FTDd3ccCecAMM5vcyLin3T0v/HgUwMzSgTuAScBE4A4zS2uj2jtMYnwcD84chzvcNn8FNbVaIkFEOo9mg95DTqz0lRh+RDq5/FPAAncvdfcyYAEw47QqDVh23xR++rnRFG0r45dvaFcqEek8IurRm1m8ma0A9hEK7sJGhl1pZqvM7DkzywofGwTsqDdmZ/hYY+8xx8yKzKyopCQ6lx+4LG8QV4wfxC/f+IDFW0uDLkdEJCIRBb2717p7HjAYmGhmoxsM+QuQ4+5jgL8B88LHrbEf18R7zHX3fHfPz8zMjKz6ANx12Wiy01O4bf5yyo9qVyoRiX4tmnXj7geBt2jQfnH3A+5+YmGYXwMTws93Aln1hg4Gdp9WpVGiR7cEHpg5jn2Hq7j9+VXalUpEol4ks24yzaxP+Hl34EJgQ4MxA+u9vBRYH37+GnCRmaWF/wh7UfhYpzY2qw//8amRvLLmQ+Yv2dH8BSIiAUqIYMxAYJ6ZxRP6xfCMu79kZncBRe7+InCLmV0K1AClwI0A7l5qZj8GloR/1l3uHhPN7a9+dCjvbtrPnX9Zy3k5aQzv1zPokkREGmXR2HrIz8/3oqKioMto1r5Dlcx44B/069mNP39jGsmJ8UGXJCJdlJktdff8xs7pm7Gt0K9XMr+4egwbPjzM3a9saP4CEZEAKOhbafqo/nxxWg6PFxTzxoa9QZcjIvIvFPRt4PaLR3HWwF58+9lV7DtUGXQ5IiL/REHfBrolxPPLWXkcPV7Dt57RrlQiEl0U9G1keL+e3PHZc3h3037m/mNL0OWIiJykoG9DM8/L4uLRA/jFa++zcsfBoMsREQEU9G3KzLj7ijH069mNW+Yvp6JKu1KJSPAU9G2sd0oiD8wax47So/zwhTVBlyMioqBvD+flpHPz9BE8v2wXf16+K+hyRKSLU9C3k5unDyd/SBo/+PMath/QrlQiEhwFfTtJiI/j/pl5mMHN85dTrV2pRCQgCvp2NDgthbuvGMPKHQe5b8HGoMsRkS5KQd/OLhkzkJnnZfHQ25sp2LQ/6HJEpAtS0HeAH372bIZmpPLNZ1ZQeuR40OWISBejoO8AKUkJPDhrHGVHqvnOcyu1K5WIdCgFfQc554zefPfiUfxt/T5+t2hb0OWISBeioO9AX5qWw8dHZvKTl9ez4cNDQZcjIl2Egr4DmRn3XD2WXsmJ3PLUciqra4MuSUS6AAV9B8vo0Y17rxnLxr0V/OTldUGXIyJdQLNBb2bJZrbYzFaa2Vozu7ORMd8ys3VmtsrM/m5mQ+qdqzWzFeHHi239ATqj88/MZM75Q3ly0XZeXfNh0OWISIyL5I6+Cpju7mOBPGCGmU1uMGY5kO/uY4DngJ/XO3fM3fPCj0vbpOoY8O2LRnLuoN7c/vwq9pQfC7ocEYlhzQa9h1SEXyaGH95gzJvufmJBl0XA4DatMgYlJcTx4KxxHK+p47b5K6jVrlQi0k4i6tGbWbyZrQD2AQvcvfAUw78MvFLvdbKZFZnZIjO7/BTvMSc8rqikpCSi4ju73IxU7rpsNIVbS3norU1BlyMiMSqioHf3WnfPI3SnPtHMRjc2zsy+AOQD99Q7nO3u+cDngfvNbFgT7zHX3fPdPT8zM7NFH6Izu3L8IC4dewb3/e0Dlm4rC7ocEYlBLZp14+4HgbeAGQ3PmdmFwH8Cl7p7Vb1rdof/3RK+dtzplxt7zIyffG40A3snc+v85RyqrA66JBGJMZHMusk0sz7h592BC4ENDcaMAx4hFPL76h1PM7Nu4ecZwDRAcwob6JWcyIOzxrGnvJL//NMaLZEgIm0qkjv6gcCbZrYKWEKoR/+Smd1lZidm0dwD9ACebTCN8iygyMxWAm8Cd7u7gr4R47PT+NYnz+QvK3fz3NKdQZcjIjHEovHuMT8/34uKioIuo8PV1jnXPbqIVTvLeenmjzA0s0fQJYlIJ2FmS8N/D/0X+mZsFImPM+6/dhxJCXHcMn85x2u0K5WItJ6CPsoM6J3Mz68cw5pdh7jntQ3NXyAi0gwFfRS66JwBfGFyNr/+x1be3tg1vlMgIu1HQR+lfnDJ2ZzZvwf//sxK9ldUNX+BiEgTFPRRKjkxnl/OGs/hymq+/exK6rREgoicJgV9FBs5oCc/uOQs3nq/hN8WFAddjoh0Ugr6KPeFyUP45Nn9ufuV9azZVR50OSLSCSnoo5yZ8bMrx5CemsQt85dz9HhN0CWJSCejoO8E0lOTuO/aPLbuP8KdL+qLxSLSMgr6TmLqsAz+zwXDeLpoBy+v2hN0OSLSiSjoO5HbLjyTvKw+3P78KnaWHW3+AhERFPSdSmJ8HA/OHIc73Dp/BTW1WiJBRJqnoO9ksvum8NPPjWbptjIefEO7UolI8xT0ndBleYO4cvxg/vuND1i8tTTockQkyinoO6k7LzuH7PQUbpu/nINHjwddjohEMQV9J9WjWwIPzhpHSUUV1z1aqPVwRKRJCvpObMzgPvz6hnw2l1RwzcML2XXwWNAliUgUUtB3cheM7MeTX55ESUUVVz9UwOaSiqBLEpEoo6CPAfk56Tw9ZwrHa+u45uGFWhNHRP5Js0FvZslmttjMVprZWjO7s5Ex3czsaTPbZGaFZpZT79z3wsffN7NPtW35csLZZ/Ti2ZumkpwYz6y5iyjcciDokkQkSkRyR18FTHf3sUAeMMPMJjcY82WgzN2HA/cBPwMws7OBmcA5wAzgV2YW31bFyz/LzUjlua9PoV+vbtzw2GLe3LAv6JJEJAo0G/QecqLxmxh+NNwF4zJgXvj5c8AnzMzCx+e7e5W7bwU2ARPbpHJp1MDe3Xn2pqmc2b8nX32iiBdW7Aq6JBEJWEQ9ejOLN7MVwD5ggbsXNhgyCNgB4O41QDnQt/7xsJ3hY429xxwzKzKzopIS7ZPaGumpSfzhq5OYMCSN255ewe8WbQu6JBEJUERB7+617p4HDAYmmtnoBkOssctOcbyx95jr7vnunp+ZmRlJWXIKPZMTmfeliXxiVD/+75/X8D9vbsJd2xGKdEUtmnXj7geBtwj12+vbCWQBmFkC0BsorX88bDCw+zRrlRZKToznoS9M4PK8M7jntfe5+5UNCnuRLiiSWTeZZtYn/Lw7cCGwocGwF4HZ4edXAW94KFFeBGaGZ+XkAiOAxW1VvDQvMT6Oe6/J44YpQ3jknS187/nV1GqjcZEuJSGCMQOBeeHZMnHAM+7+kpndBRS5+4vAb4DfmdkmQnfyMwHcfa2ZPQOsA2qAb7h7bXt8EGlaXJxx56Xn0Kd7Ig++sYnDlTXce+1YuiVoApRIV2DR+J/y+fn5XlRUFHQZMenRf2zhJy+v56MjMnjk+gmkJEXyu15Eop2ZLXX3/MbO6ZuxXcxXPjqUn185hvc27ef63yym/Gh10CWJSDtT0HdB15yXxa+uG8/qneVcO3ch+w5XBl2SiLQjBX0XNWP0QB678Ty2lx7lmocXsqNUe9CKxCoFfRf2kREZPPmVSZQdreaqhwv4YO/hoEsSkXagoO/ixmen8fTXJlPncM0jC1m542DQJYlIG1PQC6MG9OK5m6bQIzmBz/96EQs3a+VLkViioBcAhvRN5bmbpjIorTuzf7uYBev2Bl2SiLQRBb2c1L9XMk/PmcJZA3tx05NLeX7ZzqBLEpE2oKCXf5KWmsTvvzKJSbnpfOuZlTz+3tagSxKRVlLQy7/o0S2Bx248j4vO7s+P/rKOB//+gRZDE+nEFPTSqOTEeH513XiuHD+Yexds5McvradOi6GJdEpa6ESalBAfxz1XjaFX9wQee28rhyqrufuKc0mI1/2BSGeioJdTioszfviZs+nTPYn7/raRw5XVPDBzHMmJWvlSpLPQrZk0y8y49cIR3PHZs3lt7V6+PG8JR6pqgi5LRCKkoJeIfXFaLvdeM5ZFW0q57tFCDh49HnRJIhIBBb20yBXjB/PQdeNZt+cQ1zyykL2HtPKlSLRT0EuLXXTOAB7/4nnsKjvGVQ8XsP2AVr4UiWYKejktU4dl8IevTqaisoarHi7g/Q+18qVItIpkc/AsM3vTzNab2Vozu7WRMf9hZivCjzVmVmtm6eFzxWa2OnxO+wPGkLFZfXjma1MwC618uXx7WdAliUgjIrmjrwH+3d3PAiYD3zCzs+sPcPd73D3P3fOA7wFvu3tpvSEfD59vdD9D6bxG9O/JczdNpU9KItc9Wsi7H+wPuiQRaaDZoHf3Pe6+LPz8MLAeGHSKS2YBT7VNedIZZKWn8OxNU8hOT+FLjy/h1TV7gi5JROppUY/ezHKAcUBhE+dTgBnAH+sdduB1M1tqZnNO8bPnmFmRmRWVlJS0pCyJAv16hla+HD2oF//n98t4tmhH0CWJSFjEQW9mPQgF+G3ufqiJYZ8F3mvQtpnm7uOBiwm1fc5v7EJ3n+vu+e6en5mZGWlZEkV6pyTy5FcmMW14Bv/x3Cp+865WvhSJBhEFvZklEgr537v786cYOpMGbRt33x3+dx/wJ2Di6ZUqnUFKUgKPzs7n0+cO4McvrePe19/XypciAYtk1o0BvwHWu/u9pxjXG/gY8EK9Y6lm1vPEc+AiYE1ri5bo1i0hnl/OGs+1+Vk8+MYmfvTiWq18KRKgSBY1mwZcD6w2sxXhY98HsgHc/eHwsc8Br7v7kXrX9gf+FPpdQQLwB3d/tS0Kl+gWH2fcfeW59E5JZO47WzhUWcPPrxpDola+FOlwzQa9u78LWATjHgceb3BsCzD2NGuTTs7M+N7Fo+jdPZF7Xnufw5XV/Pfnx2vlS5EOptsraVdmxjc+PpwfXz6av2/Yx42/XczhyuqgyxLpUhT00iGunzyE+6/No6i4jOseLaT0iFa+FOkoCnrpMJflDWLuDRN4/8PDXPPIQvaUHwu6JJEuQUEvHWr6qP488aWJ7C2v5KqHFrJ1/5HmLxKRVlHQS4ebNLQvT82ZzLHqWq5+eCHrdjf1/TsRaQsKegnE6EG9eeZrU0iKN66du5Ci4tLmLxKR06Kgl8AM79eDZ78+lcwe3fjCbwp5e6PWOBJpDwp6CdSgPt155qYpDM3owVfmLeHlVVr5UqStKeglcBk9ujH/a5PJy+rDzU8tY/7i7UGXJBJTFPQSFXolJ/LElyZx/pmZ3P78ah55e3PQJYnEDAW9RI3uSfHMvT6fz449g/96ZQM/e3WDVr4UaQORLGom0mGSEuK4/9o8eiUn8NBbmzl0rJq7LhtNfFyzyy2JSBMU9BJ14uOMn1w+mt7dE/nVW5s5VFnDvdeM1cqXIqdJQS9Rycz4zoxR9OqeyN2vbKCisppfXTeB7kla+VKkpXSLJFHtpo8N47+uOJe3NpYw+7HFHNLKlyItpqCXqDdrYjb/PWs8y3eUMWvuIvZXVAVdkkinoqCXTuGSMQP59Q35bC6p4JqHF/L2xhJtTygSIQW9dBoXjOzHk1+exOGqGmY/tpgL73ubeQXFVFTVBF2aSFSzaJynnJ+f70VFRUGXIVGqqqaWv67ew+PvFbNyZzk9uiVw1YTBzJ6aQ25GatDliQTCzJa6e36j55oLejPLAp4ABgB1wFx3f6DBmAuAF4Ct4UPPu/td4XMzgAeAeOBRd7+7uYIV9BKp5dvLmFdQzMur91Bd63x8ZCazp+Zw/ohM4jT3XrqQ1gb9QGCguy8zs57AUuByd19Xb8wFwLfd/TMNro0HNgKfBHYCS4BZ9a9tjIJeWmrf4Ur+ULidJxdtZ39FFUMzUrlhyhCunDCYnsmJQZcn0u5OFfTN9ujdfY+7Lws/PwysBwZF+N4TgU3uvsXdjwPzgcsivFYkYv16JnPbhWdScPt0HpiZR6/uifzoL+uY8l9v8KMX17KlpCLoEkUC06IvTJlZDjAOKGzk9BQzWwnsJnR3v5bQL4Qd9cbsBCY18bPnAHMAsrOzW1KWyElJCXFcljeIy/IGsWLHQeYVFPP7wm08XlDMx87M5MZpOXxMbR3pYiL+Y6yZ9QDeBn7q7s83ONcLqHP3CjP7NPCAu48ws6uBT7n7V8LjrgcmuvvNp3ovtW6kLe07XMlThTt4snAbJYeryA23da5SW0diSKtaN+EfkAj8Efh9w5AHcPdD7l4Rfv5XINHMMgjdwWfVGzqY0B2/SIfp1zOZWy8cwXvfDbV1+qQkcudf1jH5//2dO15Yw2a1dSTGNdu6MTMDfgOsd/d7mxgzANjr7m5mEwn9AjkAHARGmFkusAuYCXy+rYoXaYn6bZ2V4bbOHxZvZ97CbaG2ztQcPnam2joSeyKZdfMR4B/AakLTKwG+D2QDuPvDZvZvwNeBGuAY8C13Lwhf/2ngfkLTKx9z9582V5RaN9JRSg5X8dTi7Ty5aBv7DleR0zeFG6bkcFX+YHqprSOdSKumVwZBQS8d7XhNHa+u/ZB5BcUs3VZGalI8V04YzA1Tchjer0fQ5Yk0S0Ev0gKrdh7k8YJiXlq5h+O1dXx0RAZfnJbDBWf2U1tHopaCXuQ07K+o4qnC7TxZuI29h6oYEm7rXK22jkQhBb1IK1TX1vHqmlBbp2hbGSlJ8Vw5fjCzpw5heL+eQZcnAijoRdrMml3lPF5QzIsrdp9s69w4NYcLRvbTvrYSKAW9SBs7UFHF/CU7+N3CbXx4qJLs9BRumDKEq/Oz6N1dbR3peAp6kXZSXVvH62v38njBVpYUh9o6V4wfxOwpOYzor7aOdBwFvUgHWLOrnHkFxbywcjfHa+r4yPAMZk/NYfootXWk/SnoRTrQibbOk4u2sae8kqz07twwOYdr8rPonaK2jrQPBb1IAGpq63h93V4ef6+YxcWldE8MtXVunKq2jrQ9Bb1IwNbuDrd1VuymqqaOacP7MntKDp84q7/aOtImFPQiUaL0yHHmL9nO7xaG2jqD07pzw5QhXJufrbaOtIqCXiTK1NTWsWDdXn5bUMziraG2zuXjQm2dkQPU1pGWU9CLRLF1uw8xr6CYP6/YRVVNHVOH9WX21BwuVFtHWkBBL9IJlB05fnK2zq6Dxxic1p3rJw/h2vOy6JOSFHR5EuUU9CKdSE1tHX9bv5fHC4pZtKWU5MQ4Lh17BheM7MfE3HQyenQLukSJQgp6kU5q/Z5DPLEwNFvn6PFaAIb368HE3HQm5aYzeWhf+vdKDrZIiQoKepFOrrq2jtW7ylm8tZTCLQcoKi7jcFUNADl9U5iU25dJQ9OZmJvO4LSUgKuVICjoRWJMbZ2zbvchCrceYNGWUpYUl1J+rBqAQX26M2loOpPD4Z+dnkJo62eJZa0KejPLAp4ABhDaM3auuz/QYMx1wHfDLyuAr7v7yvC5YuAwUAvUNFVIfQp6kZapq3M2fHiYwq0HQnf9W0spPXIcgAG9kk/e7U/K7cuwzFQFfwxqbdAPBAa6+zIz6wksBS5393X1xkwF1rt7mZldDPzI3SeFzxUD+e6+P9KCFfQirePubNpXwaJwq6dwayklh6sAyOjRjUm56UwaGgr+Ef16aIvEGHCqoE9o7mJ33wPsCT8/bGbrgUHAunpjCupdsggY3KqKRaRVzIwR/Xsyon9Prp88BHdn6/4jJ+/2C7cc4OXVewBIS0nkvJx0Jg3ty6TcdM4a2Evz92NMs0Ffn5nlAOOAwlMM+zLwSr3XDrxuZg484u5zW1ijiLSSmTE0swdDM3swc2I27s7OsmMsCt/tF249wOvr9gLQMzmBiTn/e8d/zhm9SIiPC/gTSGtEHPRm1gP4I3Cbux9qYszHCQX9R+odnubuu82sH7DAzDa4+zuNXDsHmAOQnZ3dgo8gIi1lZmSlp5CVnsLV+VkA7D54LHzHf4DCLaX8fcM+AFKT4pmQc2I6ZzrnDupDUoKCvzOJaNaNmSUCLwGvufu9TYwZA/wJuNjdNzYx5kdAhbv/4lTvpx69SPD2Hao8ebdfuKWUD/ZVAJCcGMeEIWlMyu3LxNx08rL6kJwYH3C10to/xhowDyh199uaGJMNvAHcUL9fb2apQFy4t58KLADucvdXT/WeCnqR6HOgooolxaUs2hLq82/48BDukJQQR15WHybnhvr847PT6J6k4O9orQ36jwD/AFYTml4J8H0gG8DdHzazR4ErgW3h8zXunm9mQwnd5UOoTfQHd/9pcwUr6EWi38Gjx1lSXHZyVs/a3eXUOSTGG2MG9zn57d38nHR6dGvRnwPlNOgLUyLS7g5VVrN0WxmFW0LtntU7y6mpc+LjjNFn9Do5qyc/J53e3bX2fltT0ItIhztSVcOy7f8b/Ct3lHO8tg4zOGtAr5OzeiblppOWqtU5W0tBLyKBq6yuZdn2svB6PaUs215GVU2oGzyyf8+TwT8xN53Mnlqhs6UU9CISdapqalm1s/xkj3/ptrKTK3QOzUxl8tC+TB3WlylD+9JXSzM3S0EvIlGvuraONbvKT35zd0lxGRXhFTpHDejJ1GEZTBseuuPvmawef0MKehHpdGpq61i1q5yFmw/w3qb9FG0r43hNHfFxxrmDejNteF+mDstgwpA0zeNHQS8iMeBEj79g0wEKNu9n5c5yauucpIQ4JmSnMXVYX6YOz2DM4N4kdsElGxT0IhJzDldWs6S4lIJNB3hv8wHW7wmtzJKaFM+kcH9/6rAMRg3o2SVW52zV6pUiItGoZ3Ii00f1Z/qo/gCUHjnOws2hu/2Fmw/wRnitnrSURKaEQ3/qsL7kZnS99fgV9CISE9JTk7hkzEAuGTMQCC3StnDzAd7bvJ+CTQf46+oPARjYO/lk8E8b3peBvbsHWXaHUOtGRGKeu1N84Cjvbdp/8q6/7Gho68XcjNSTbZ7JQ9M77VRO9ehFROo5sfViweb9FGw+QOGWAxwJz+E/a2Avpg7ry7ThfTkvp/NM5VTQi4icQnVtHat2lrMwHPz1p3KOGdybaeH+/vgonsqpoBcRaYHK6lqWbSujINzjX1VvKmf+kHpTOQf1jprdtxT0IiKtcLiymsVbSynYfICCelM5e3RLYFJuOlOG9WXa8AxG9g9uKqemV4qItELP5EQ+cVZ/PnFWaCrngYoqFm0p5b3wVM4T2y6mpyYxZcCWt+cAAAUKSURBVGhfpoa/tZvTNyUqpnLqjl5EpJV2HzwWvtsPTeX88FAlAGf0TmZKuL8/bXgGA3ont1sNat2IiHQQd2fr/iO8t/kAC8N3/Cemcg7NSD15tz95aF/S23AdfgW9iEhA6uqc9R8eOrk42+KtpSencp59cipnBufltm7LRQW9iEiUCE3lPBhenO0AS7eVcbw2NJVzQnYaT82ZTPxp/EG3VX+MNbMs4AlgAKHNwee6+wMNxhjwAPBp4Chwo7svC5+bDfwgPPQn7j6vxZ9ARCRGJMbHMWFIOhOGpHPzJ0ZQWV3L0m1lFGzez4GK46cV8s2J5L8TaoB/d/dlZtYTWGpmC9x9Xb0xFwMjwo9JwEPAJDNLB+4A8gEPX/uiu5e16acQEemkkhPjmTY8g2nDM9rtPZqd6e/ue07cnbv7YWA9MKjBsMuAJzxkEdDHzAYCnwIWuHtpONwXADPa9BOIiMgptegrXWaWA4wDChucGgTsqPd6Z/hYU8cb+9lzzKzIzIpKSkpaUpaIiJxCxEFvZj2APwK3ufuhhqcbucRPcfxfD7rPdfd8d8/PzMyMtCwREWlGREFvZomEQv737v58I0N2Aln1Xg8Gdp/iuIiIdJBmgz48o+Y3wHp3v7eJYS8CN1jIZKDc3fcArwEXmVmamaUBF4WPiYhIB4lk1s004HpgtZmtCB/7PpAN4O4PA38lNLVyE6HplV8Mnys1sx8DS8LX3eXupW1XvoiINKfZoHf3d2m8115/jAPfaOLcY8Bjp1WdiIi0WnQspCwiIu0mKpdAMLMSYNtpXp4B7G/DcjoDfebY19U+L+gzt9QQd290ymJUBn1rmFlRU+s9xCp95tjX1T4v6DO3JbVuRERinIJeRCTGxWLQzw26gADoM8e+rvZ5QZ+5zcRcj15ERP5ZLN7Ri4hIPQp6EZEYFzNBb2YzzOx9M9tkZrcHXU9HMLPHzGyfma0JupaOYGZZZvamma03s7VmdmvQNbU3M0s2s8VmtjL8me8MuqaOYmbxZrbczF4KupaOYGbFZrbazFaYWZvupRoTPXoziwc2Ap8ktGLmEmBWg12wYo6ZnQ9UENr0ZXTQ9bS38GY2A+vvdgZcHsv/O4cXFUx194rwKrLvAreGN/iJaWb2LUK70/Vy988EXU97M7NiIN/d2/xLYrFyRz8R2OTuW9z9ODCf0K5XMc3d3wG6zCJxEe52FlPCu7ZVhF8mhh+d/+6sGWY2GLgEeDToWmJBrAR9xDtZSWw4xW5nMSfcwlgB7CO0NWfMf2bgfuA7QF3QhXQgB143s6VmNqctf3CsBH3EO1lJ59fMbmcxx91r3T2P0MY9E80sptt0ZvYZYJ+7Lw26lg42zd3HAxcD3wi3ZttErAS9drLqIiLY7SxmuftB4C1gRsCltLdpwKXhnvV8YLqZPRlsSe3P3XeH/90H/IlQS7pNxErQLwFGmFmumSUBMwnteiUxJMLdzmKKmWWaWZ/w8+7AhcCGYKtqX+7+PXcf7O45hP6//Ia7fyHgstqVmaWGJxhgZqmEduNrs9l0MRH07l4D/BuhbQrXA8+4+9pgq2p/ZvYUsBAYaWY7zezLQdfUzk7sdjY9PAVthZl9Ouii2tlA4E0zW0XohmaBu3eJ6YZdTH/gXTNbCSwGXnb3V9vqh8fE9EoREWlaTNzRi4hI0xT0IiIxTkEvIhLjFPQiIjFOQS8iEuMU9CIiMU5BLyIS4/4/acuh1TaRNbUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 2\n",
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "357 out of 569 persons have cancer\n",
      "features 30\n"
     ]
    }
   ],
   "source": [
    "cancer=load_breast_cancer()\n",
    "n_samples = len(cancer.target)\n",
    "X = cancer.data\n",
    "y = cancer.target\n",
    "total=len(y)\n",
    "has_cancer=np.sum(y)\n",
    "no_cancer=total-has_cancer\n",
    "print(f'{has_cancer} out of {total} persons have cancer')\n",
    "print(f'features {X.shape[1]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 Log of Odds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the log_of_odds is : 0.5211495071076265\n",
      "the probability is : 0.6274165202108963\n"
     ]
    }
   ],
   "source": [
    "log_of_odds0=np.log(has_cancer / no_cancer)\n",
    "probability0=np.exp(log_of_odds0)/(np.exp(log_of_odds0)+1)\n",
    "print(f'the log_of_odds is : {log_of_odds0}')\n",
    "print(f'the probability is : {probability0}')\n",
    "predict0=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 For Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss is 375.7200026920845\n",
      "loss is 375.72000269208445\n",
      "gamma1 1.2213642015774735\n",
      "gamma2 -2.4363001705150817\n",
      "new loss is 338.13703383104206\n"
     ]
    }
   ],
   "source": [
    "#explore\n",
    "residuals0=y - probability0\n",
    "loss0=-(y*np.log(probability0)+(1-y)*np.log(1-probability0))\n",
    "print(f'loss is {loss0.sum()}')\n",
    "loss0=-y * log_of_odds0 + np.log(1+np.exp(log_of_odds0))\n",
    "print(f'loss is {loss0.sum()}')\n",
    "\n",
    "tree0 = DecisionTreeRegressor(max_depth=1)\n",
    "tree0.fit(X, residuals0)\n",
    "stem = tree0.tree_\n",
    "\n",
    "leaf_index=tree0.apply(X)\n",
    "#for leaf 1\n",
    "n_leaf1=len(leaf_index[leaf_index==1])\n",
    "previous_probability1 = np.array([probability0] * n_leaf1)\n",
    "denominator1 = np.sum(previous_probability1 * (1-previous_probability1))\n",
    "gamma1 = stem.value[1][0][0] * n_leaf1 / denominator1\n",
    "print(f'gamma1 {gamma1}')\n",
    "#for leaf 2\n",
    "n_leaf2=len(leaf_index[leaf_index==2])\n",
    "previous_probability2 = np.array([probability0] * n_leaf2)\n",
    "denominator2 = np.sum(previous_probability2 * (1-previous_probability2))\n",
    "gamma2 = stem.value[2][0][0] * n_leaf2 / denominator2\n",
    "print(f'gamma2 {gamma2}')\n",
    "\n",
    "gamma_value=[gamma1, gamma2]\n",
    "gamma = np.array([gamma_value[index-1] for index in leaf_index])\n",
    "new_log_of_odds = log_of_odds0 + 0.1 * gamma\n",
    "new_loss=-y * new_log_of_odds + np.log(1+np.exp(new_log_of_odds))\n",
    "print(f'new loss is {new_loss.sum()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [],
   "source": [
    "#now let's get started\n",
    "n_times = 20\n",
    "learning_rate=0.1\n",
    "log_of_odds=np.zeros([n_times+1,n_samples])\n",
    "\n",
    "residuals = np.zeros([n_times+1,n_samples])\n",
    "loss = [0] * (n_times+1)\n",
    "probabilities = np.zeros([n_times+1,n_samples])\n",
    "score = np.zeros([n_times+1])\n",
    "#calculation\n",
    "log_of_odds[0] = [log_of_odds0] * n_samples\n",
    "probabilities[0] = [probability0] * n_samples\n",
    "score[0]=np.sum((probabilities[0]>0.5)*1.0==y) / n_samples\n",
    "residuals[0] = y - probabilities[0]\n",
    "loss[0] = loss0.sum()\n",
    "trees = []\n",
    "gamma_value = np.zeros([n_times+1,8])\n",
    "gamma = np.zeros([n_times+1,n_samples])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(n_times):\n",
    "    dt = DecisionTreeRegressor(max_depth=2)\n",
    "    dt=dt.fit(X, residuals[i])\n",
    "    tree = dt.tree_\n",
    "    trees.append(tree)\n",
    "    leaf_indeces=dt.apply(X)\n",
    "    unique_leaves=np.unique(leaf_indeces)\n",
    "    n_leaf=len(unique_leaves)\n",
    "    #for leaf 1\n",
    "    for ileaf in range(n_leaf):\n",
    "        leaf_index=unique_leaves[ileaf]\n",
    "        n_leaf=len(leaf_indeces[leaf_indeces==leaf_index])\n",
    "        previous_probability = probabilities[i][leaf_indeces==leaf_index]\n",
    "        denominator = np.sum(previous_probability * (1-previous_probability))\n",
    "        igamma = tree.value[ileaf+1][0][0] * n_leaf / denominator\n",
    "        gamma_value[i][ileaf]=igamma\n",
    "\n",
    "    gamma[i] = [gamma_value[i][np.where(unique_leaves==index)] for index in leaf_indeces]\n",
    "    #next \n",
    "    log_of_odds[i+1] = log_of_odds[i] + learning_rate * gamma[i]\n",
    "\n",
    "    probabilities[i+1] = np.array([np.exp(odds)/(np.exp(odds)+1) for odds in log_of_odds[i+1]])\n",
    "    score[i+1]=np.sum((probabilities[i+1]>0.5)*1.0==y) / n_samples\n",
    "    residuals[i+1] = y - probabilities[i+1]\n",
    "    loss[i+1]=np.sum(-y * log_of_odds[i+1] + np.log(1+np.exp(log_of_odds[i+1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5b3H8c8vO4GQEBJCgLCHXdaIIEWoVo1LxVattlZR2+JarbdX7XLb2u32qldtvXXDpWLdl6q44S6himBAwCAQwh4gC2uAQEKS5/4xB5tiQhLI5Exmvu/Xa15z5jnnZH45M/nmzDPnPMecc4iISHiJ8rsAERFpfQp3EZEwpHAXEQlDCncRkTCkcBcRCUMxfhcAkJaW5vr27et3GSIi7cqiRYu2OefSG5oXEuHet29f8vPz/S5DRKRdMbMNjc1Tt4yISBhSuIuIhCGFu4hIGFK4i4iEIYW7iEgYUriLiIQhhbuISBhq1+G+cXslt85ezsHaOr9LEREJKe063AtL9/DYx+t5euFGv0sREQkp7TrcTxnajQn9U/nzu6upOHDQ73JEREJGuw53M+OXZw5jx75qHvhwjd/liIiEjHYd7gDH9UrmW2N68sg/17Fl136/yxERCQntPtwBfnraIBzwv2+v8rsUEZGQEBbh3qtLIldM6sdLn22mYPNuv8sREfFdWIQ7wDVfH0BKh1j++PoKnHN+lyMi4quwCffOCbHccEo289du54NVZX6XIyLiq7AJd4DvndCHfmkd+dMbK6nRiU0iEsHCKtzjYqK4JXcIq8v28lx+sd/liIj4JqzCHeD04Rnk9OnCXe8Usreqxu9yRER8EXbhbmb88qyhbNtbxcy8tX6XIyLii7ALd4Axvbtw9shMZuatoWT3Ab/LERFpc2EZ7gC35A6hrg7uekcnNolI5AnbcM9KTeTSiX14flExK7ZW+F2OiEibCttwB7ju5IF0TojlT2+u9LsUEZE21WS4m1mCmS00s6VmttzMfuu1P2Zm68xsiXcb7bWbmd1jZkVmtszMxgb7l2hMSmIcPz55IHmF5cwtLPerDBGRNtecPfcq4GTn3ChgNJBrZhO8eTc550Z7tyVe2xlAtnebAdzf2kW3xCUT+5CV2oE/vbGC2joNSyAikaHJcHcBe72Hsd7tSCk5DXjcW+8TIMXMMo+91KMTHxPNLblDWFmyhxcX68QmEYkMzepzN7NoM1sClAHvOOcWeLP+6HW93G1m8V5bT2BTvdWLvbbDf+YMM8s3s/zy8uB2mZx1XCajs1K48+1VVFbrxCYRCX/NCnfnXK1zbjTQCxhvZiOAnwNDgOOBVOAWb3Fr6Ec08DNnOudynHM56enpR1V8c5kZ/3XWUEorqnh43rqgPpeISCho0dEyzrldwIdArnNuq9f1UgX8DRjvLVYMZNVbrRewpRVqPSY5fVPJHd6dB+auoWyPTmwSkfDWnKNl0s0sxZvuAHwDWHmoH93MDDgXKPBWmQ1c6h01MwHY7ZzbGpTqW+iWM4ZQXVPHn99d7XcpIiJB1Zw990zgAzNbBnxKoM/9NeBJM/sc+BxIA/7gLf8GsBYoAh4Crmn1qo9Sv7SOfH9CH55ZuJHVpXv8LkdEJGgsFK5alJOT4/Lz89vkuXbsq2bK7R9wfL9UHr3s+DZ5ThGRYDCzRc65nIbmhfUZqg1J7RjHtScP5P2VZXxctM3vckREgiLiwh3gshP70jOlA398YwV1OrFJRMJQRIZ7Qmw0N50+mOVbKnh+0aamVxARaWciMtwBzhnVg/F9U/nj6ysordChkSISXiI23KOijNvOH0lVTR2/fKmAUPhiWUSktURsuEPg0MifnjaId1eU8uqykDgUX0SkVUR0uAP84Gv9GZWVwq2zl7N9b5Xf5YiItIqID/foKOOO80ey58BBbn31C7/LERFpFREf7gCDMpL48cnZvLp0C28vL/G7HBGRY6Zw91w9dQBDuifxXy8XsHv/Qb/LERE5Jgp3T2x0FHecP4rt+6r54+vqnhGR9k3hXs9xvZK58qT+PJdfTJ6uuSoi7ZjC/TDXn5LNgPSO/Pwfn7O3SldtEpH2SeF+mITYaG4/fyRbdu/n9jkr/S5HROSoKNwbMK5PKped2JfH529gwdrtfpcjItJiCvdG3HT6YLJSO3DLi8vYX13rdzkiIi2icG9EYlwMt317JOu3V3L3u4V+lyMi0iIK9yM4cWAa3x3fm4fnrWXJpl1+lyMi0mwK9yb8/MwhZHRO4OYXllJVo+4ZEWkfFO5N6JwQy39/6zgKS/dy7wdr/C5HRKRZFO7N8PUh3fj2mJ7c90ERX2yp8LscEZEmKdyb6dffHEZKYhw3v7iUmto6v8sRETkihXszpSTG8ftpwynYXMHMeWv9LkdE5IgU7i1wxnGZnHlcd/787mqKyvb6XY6ISKMU7i3023NGkBgXzc0vLKW2TtddFZHQpHBvofSkeH7zzWEs3riLR/+5zu9yREQapHA/CueO7slpwzK4bc5KFm3Y6Xc5IiJfoXA/CmbGHReMokdKB659cjHbdGFtEQkxTYa7mSWY2UIzW2pmy83st157PzNbYGarzexZM4vz2uO9x0Xe/L7B/RX8kdwhlvu/P5adldXc8Mxn6n8XkZDSnD33KuBk59woYDSQa2YTgNuAu51z2cBO4Afe8j8AdjrnBgJ3e8uFpeE9kvn9uSP4qGg7d7+jwcVEJHQ0Ge4u4NBxf7HezQEnAy947bOAc73pad5jvPmnmJm1WsUh5js5WVx0fBZ//aCI91aU+l2OiAjQzD53M4s2syVAGfAOsAbY5Zw7dB26YqCnN90T2ATgzd8NdG3gZ84ws3wzyy8vb9/XK731nOGM6NmZG59dwqYdlX6XIyLSvHB3ztU650YDvYDxwNCGFvPuG9pL/0qHtHNupnMuxzmXk56e3tx6Q1JCbDT3XzwOgKueWMSBgxo9UkT81aKjZZxzu4APgQlAipnFeLN6AVu86WIgC8CbnwzsaI1iQ1lWaiJ3Xzia5VsquHX2cr/LEZEI15yjZdLNLMWb7gB8A1gBfACc7y02HXjFm57tPcab/75zLiIOJTllaAbXfn0Az3y6iefyN/ldjohEsJimFyETmGVm0QT+GTznnHvNzL4AnjGzPwCfAY94yz8C/N3MigjssV8UhLpD1n+cOpglm3bxq5cLGN6jM8N7JPtdkohEIAuFneqcnByXn5/vdxmtZtveKs6+55/ExUTx6o+/RnKHWL9LEpEwZGaLnHM5Dc3TGapBkNYpnnsvHsuWXfv56XNLqdMJTiLSxhTuQTKuTxd+edZQ3l1RyoN5Gv9dRNqWwj2ILjuxL2ePzOSOt1by8ZptfpcjIhFE4R5EZsZt542kX1pHrn/6M0p2H/C7JBGJEAr3IOsYH8MD3x9HZXUt1z21mIO6/qqItAGFexvIzkjif84bSf6GnfzPmyv9LkdEIoDCvY2cM6oHl53Yl0f+uY7Xl231uxwRCXMK9zb0izOHMqZ3Cje/sJQ15brAtogEj8K9DcXFRHHfxWOJj41mxuP57Kqs9rskEQlTCvc2lpncgfsvHsumHfv54ax8jSApIkGhcPfBCf27cteFo1i0cSc/eWaJLtEnIq1O4e6Ts0f24FdnDWPO8hJ+++pyQmGMHxEJH80ZFVKC5Iqv9WPr7v08NG8dmckduHrqAL9LEpEwoXD32c/PGEpJRRW3zVlJ9+R4vjWml98liUgYULj7LCrK+N8LRrJtTxU3Pb+MtE7xTM5u35cdFBH/qc89BMTHRPPgpeMY2K0TV/19EQWbd/tdkoi0cwr3ENE5IZbHLh9PcodYLn/sUzbtqPS7JBFpxxTuIaR7cgKPXTGeqoO1TP/bQnbu00lOInJ0FO4hZlBGEg9PP57infv54eM6yUlEjo7CPQSN75fKny8czeKNO7n+6c90kpOItJjCPUSdeVwmvz57GG9/Ucqts3WSk4i0jA6FDGGXT+pHye4DPJi3lsyUBK6ZOtDvkkSknVC4h7hbcodQUnGA2+esIiMpgfPG6SQnEWmawj3ERUUZd5w/ivI9Vdzy4jLSk+I5aZBOchKRI1OfezsQFxPFA5cETnK6+gmd5CQiTVO4txOdE2KZdcV4UhLjuOxvn7KqZI/fJYlICFO4tyMZnROYdcV4YqKM7zw4n8Ubd/pdkoiEKIV7OzOwWyeev2oiXRJjufihBcxbXe53SSISgpoMdzPLMrMPzGyFmS03sxu89lvNbLOZLfFuZ9Zb5+dmVmRmq8zs9GD+ApEoKzWR566aSJ+uiVzx2Ke8+flWv0sSkRDTnD33GuCnzrmhwATgWjMb5s272zk32ru9AeDNuwgYDuQC95lZdBBqj2jdkhJ49sqJjOyVwrVPLea5Tzf5XZKIhJAmw905t9U5t9ib3gOsAHoeYZVpwDPOuSrn3DqgCBjfGsXKv0vuEMvffzCeydnp3PziMh7KW+t3SSISIlrU525mfYExwAKv6TozW2Zmj5pZF6+tJ1B/N7KYBv4ZmNkMM8s3s/zycvUbH63EuBgeujSHs0Zm8sc3VnDHWys1VIGIND/czawT8CLwE+dcBXA/MAAYDWwF7jy0aAOrfyVtnHMznXM5zrmc9HSdlHMs4mKiuOeiMXx3fG/u/WANv3qlgDoNNiYS0Zp1hqqZxRII9iedc/8AcM6V1pv/EPCa97AYyKq3ei9gS6tUK42KjjL++1sjSO4QywNz11Cxv4Y7vzOK2GgdECUSiZpztIwBjwArnHN31WvPrLfYt4ACb3o2cJGZxZtZPyAbWNh6JUtjzIyfnTGEW3KHMHvpFmY8ns/+ao0HLxKJmrPnPgm4BPjczJZ4bb8Avmtmowl0uawHrgRwzi03s+eALwgcaXOtc04J04aunjqAlMRYfvHS51z66AIenn48yR1i/S5LRNqQhcKXbzk5OS4/P9/vMsLOa8u2cOOzS8julsSsK8aTnhTvd0ki0orMbJFzLqeheeqQDWNnj+zBw9OPZ922fXznwfkU79RFt0UihcI9zE0ZlM4TPxzP9r1VXPDAfIrK9vpdkoi0AYV7BBjXJ5Vnr5zIwVqnAcdEIoTCPUIMzezMC1dNJCkhhotmfsKrS3V0qkg4U7hHkL5pHXnpmkmM6pXMj5/+jHveW62zWUXClMI9wqR2jOOJH57At8f05K53CvmP55ZSVaMjVUXCja6hGoHiY6K58zuj6JfWkTvfKaR4ZyUPXpJDasc4v0sTkVaiPfcIZWb8+JRs/vq9MSwt3s25936kI2lEwojCPcKdPbIHz8yYQGV1Dd+67yM+Ktrmd0ki0goU7sLY3l146ZpJZCYnMP3RhTy9cKPfJYnIMVK4CxC4dN+LV5/IpIFp/Pwfn/Pfb6ygVsMGi7RbCnf5UlJCLI9Mz+HSiX2YmbeWq55YRGV1jd9lichRULjLv4mJjuJ300Zw6zeH8d6KUi54YD4luw/4XZaItJDCXRp02aR+PDL9eNZv28e0e/9JwebdfpckIi2gcJdGfX1IN164+kSizbjggfm8vbzE75JEpJkU7nJEQzM78/J1kxiU0Ykrn1jEPe+t1hetIu2Awl2a1C0pgWdmTGTaqB7c9U4hFz/8ifrhRUKcwl2apUNcNHdfOJo7zh/J0k27OeMveby/srTpFUXEFwp3aTYz44KcLF798dfontyBKx7L53evfqGBx0RCkMJdWmxgt068dM2JXHZiXx79aB3n3f8x67bt87ssEalH4S5HJSE2mlvPGc7MS8ZRvHM/Z98zj5c+K/a7LBHxKNzlmJw2vDtvXD+Z4T2SufHZpfzHc0vYV6WzWkX8pnCXY9YjpQNP/egEbjglm5c/28zZ/6eTnkT8pnCXVhETHcWNpw7iqR9NYH91Ld++72Me/ec6XcZPxCcKd2lVE/p35c0bJnPSoDR+99oX/OjxfHbsq/a7LJGIo3CXVtelYxwPXZrDb745jLzCbZzxlzzmr9nud1kiEUXhLkFhZlw+qR//uOZEOsbF8L2HP+H2OSs5cFDHxIu0BYW7BNWInsm8+uOvccG4Xtz34RrOumceizbs8LsskbDXZLibWZaZfWBmK8xsuZnd4LWnmtk7Zrbau+/itZuZ3WNmRWa2zMzGBvuXkNDWMT6G288fxawrxnPgYB3nPzCfW2cv1yGTIkHUnD33GuCnzrmhwATgWjMbBvwMeM85lw285z0GOAPI9m4zgPtbvWppl6YMSuetG0/i0gl9mDV/Paf/OY95q8v9LkskLDUZ7s65rc65xd70HmAF0BOYBszyFpsFnOtNTwMedwGfAClmltnqlUu71Ck+ht9OG8FzV04kLiaKSx5ZyE3PL2V35UG/SxMJKy3qczezvsAYYAGQ4ZzbCoF/AEA3b7GewKZ6qxV7bSJfOr5vKm9cP5lrpg7gH59t5ht3z2VOwVa/yxIJG80OdzPrBLwI/MQ5V3GkRRto+8qZLGY2w8zyzSy/vFwfzSNRQmw0N+cO4ZVrJ5HeKZ6rnljM1U8somyPxooXOVbNCncziyUQ7E865/7hNZce6m7x7su89mIgq97qvYAth/9M59xM51yOcy4nPT39aOuXMDCiZzKvXDeJm3MH897KMk69K48XFhXr7FaRY9Cco2UMeARY4Zy7q96s2cB0b3o68Eq99ku9o2YmALsPdd+INCY2Ooprpg7kzRsmMyijE//5/FKm/+1TindW+l2aSLtkTe0dmdnXgHnA50Cd1/wLAv3uzwG9gY3ABc65Hd4/g78CuUAlcLlzLv9Iz5GTk+Py84+4iESQujrHEws2cNubK3HALblDuGRCH6KiGurxE4lcZrbIOZfT4LxQ+OircJeGFO+s5BcvFZBXWM6Y3in86uxhjO3dxe+yRELGkcJdZ6hKyOrVJZFZlx/PnReMonjnfr5938dc99RiNm5XV41IU2L8LkDkSMyM88b1IndEd2bmrWVm3lreWl7C9Il9ue7kgaQkxvldokhI0p67tAsd42O48dRBfHjTVL49phePfLSOKXd8yMPz1uoC3SINULhLu5LROYHbzh/JG9dPZlRWCn94fQWn3pXH68u26tBJkXoU7tIuDc3szONXjGfWFeNJjIvm2qcWc979H2vESRGPwl3atSmD0nn9+sncft5Iinfu57z753PNk4tYv22f36WJ+EqHQkrYqKyu4aG8dTyYt4aDtXV8f0Ifrj85my4d9aWrhCcdCikRITEuhhu+kc2H/zmV88f1YtbH6znpjg+YmbdGV4CSiKM9dwlbq0r28Kc3V/DhqnIyOsdz3cnZXJiTRVyM9mkkPGjPXSLS4O5JPHb5eJ7+0QSyuiTyq5cLOPnOD3k+fxM1tXVN/wCRdkzhLmFv4oCuPH/VRB67/Hi6JMZx0wvLOO3uPGYv3UJdnf+fXEWCQeEuEcHMmDq4G7Ovm8QD3x9HTLRx/dOfceY983h7eYmOkZewo3CXiGJm5I7ozps3nMRfLhpNVU0dM/6+iHPv/Yi8wnKFvIQNhbtEpOgoY9ronrxz40ncft5Itu2t5tJHF3Lhg5+wYO12v8sTOWY6WkYEqKqp5blPN/F/7xdRtqeKydlp/PS0wYzOSvG7NJFGaTx3kWbaX13LE59s4P65a9ixr5pvDM3gqin9GdenC4Hr0IiEDoW7SAvtrarhsY/WMTNvLRUHahia2ZlLJ/Zh2ugeJMZppGwJDQp3kaNUWV3Dy59t4fH561lZsofOCTFckJPFJRP60Deto9/lSYRTuIscI+ccn67fyePz1zOnoISaOsdJg9KZPrEPUwd3I1rXdxUfHCnc9flSpBnMjPH9UhnfL5WyigM8tXAjTy3YyA9m5dOrSwe+P6EPF+ZkaZAyCRnacxc5Sgdr63h7eSmz5q9n4bodxMdE8c1RPbh0Yh9G9tJRNhJ86pYRCbKVJRX8ff4GXvpsM5XVtYzOSuHSiX04a2Qm8THRfpcnYUrhLtJGKg4c5MVFxfx9/gbWbttHZnICV08dwHdyskiIVchL61K4i7SxujpH3upy/vp+EfkbdtK9cyDkLzxeIS+tR+Eu4hPnHB+v2c6f3y3k0/U7yegcz9VTBnDR+N4KeTlmCncRnznnmL9mO39+dzUL1+8go3M8V00ZwHcV8nIMFO4iIcI5x/y1Xsiv20G3pEDIf+8Ehby0nMJdJATN97prFqzbQboX8hcr5KUFjukye2b2qJmVmVlBvbZbzWyzmS3xbmfWm/dzMysys1Vmdnrr/Aoi4WfigK48e+VEnv7RBAakd+T3r33B5Ns/4OF5a9lfrQt6y7Fpcs/dzE4C9gKPO+dGeG23Anudc/972LLDgKeB8UAP4F1gkHPuiO9U7bmLwIK12/nLe6v5eM120jrFc/mkvkwb3YNeXRL9Lk1C1DENP+CcyzOzvs18rmnAM865KmCdmRURCPr5zVxfJGKd0L8rT/XvysJ1O7jnvdXc8dYq7nhrFcf37cK00T0567hMDW8gzXYsV2K6zsyWed02Xby2nsCmessUe20i0kzj+6XyxA9PYN7NX+em0wezq/Ig//VyAcf/8V1+8NinzF66hcrqGr/LlBB3tAOH3Q/8HnDe/Z3AFUBDQ+M12O9jZjOAGQC9e/c+yjJEwldWaiLXfn0g10wdwBdbK5i9ZAuzl27hvZVlJMZFc/rw7pwzugeTB6YRE60rZsq/O6pwd86VHpo2s4eA17yHxUBWvUV7AVsa+RkzgZkQ6HM/mjpEIoGZMbxHMsN7JHNL7hAWrt/BK0s28/qyrbz02Wa6dozj7JGZnDO6J2N7p+iKUQI081BIr8/9tXpfqGY657Z60zcCJzjnLjKz4cBT/OsL1feAbH2hKtL6qmpqmbuqnFeWbOHdFaVU1dSRldqBaaN6Mm10D7IzkvwuUYLsmI5zN7OngalAGlAK/MZ7PJpAl8t64Mp6Yf9LAl00NcBPnHNvNlWgwl3k2Ow5cJC3lpfyypLNfFS0jToHA9I7csaITHJHdGd4j87aow9DOolJJIKU7TnAnIIS5hSUsGDdDmrrHL26dCB3eHdyR3RnbO8uROnKUWFB4S4SoXbsq+bdL0qZs7yEf67eRnVtHelJ8Zw+PIPc4Zmc0D+VWH0Z224p3EWEigMH+WBlGXMKSvhwVTn7D9aSkhjLN4ZmcMaI7kwamKahD9oZhbuI/Jv91bXMLSznreUlvLuilD0HaugYF83Xh3Qjd0R3Jmenk9wh1u8ypQm6QLaI/JsOcdHkjgj0wVfX1PHxmm28tbyEt5eX8tqyrURHGWN7pzB1cDemDEpnWGZn9dO3M9pzF5Ev1dY5Fm3YydzCMuYWllOwuQKAtE7xnDQojSmD0jkpO13DIIQIdcuIyFEp23OAeYXbmFtYTt7qcnZVHsQMRvVKYcqgdKYOTmdkrxSitVfvC4W7iByz2jrHsuJdzC0s58NV5Swt3oVzkJIYy+TsdKYOSmfyoDS6JSX4XWrEULiLSKvbua+aeUXb+HBVGXmF5WzbWw3A8B6dA903g9IZ16eLDrUMIoW7iARVXZ3ji60VzC0sZ25hOYs27KS2ztEpPoYTB3RlyuBAX31Wqsamb00KdxFpUxUHDvJx0fZAX31hOZt37Qegf3pHpgxKZ8qgdCb076rj6o+Rwl1EfOOcY035vi/36hes3U5VTR3xMVGM75f65RezA9I7afybFlK4i0jIOHCwlgXrdjB3VTlzC8tYU74PgB7JCYzrm0p2t04M7NaJ7G6d6NO1I3Ex6rNvjE5iEpGQkRAb/WXXDAyjeGcleYXbmFtYxmcbd/Lq0n9dAiImyujTNZHsbkmBwM/oxID0wK1DnLp0jkR77iISUiqra1hbvo/VZXsoKtvL6tK9FJXtZcOOSmrrAnllBlldEr/cwx/QrRMjeyUzOCMporp2tOcuIu1GYlwMI3omM6Jn8r+1V9XUsn5bZSDwveAvKtv75WiXAH26JpI7vDunj+jO6F4pET1kgvbcRaRdq6mtY+OOShas28GcghI+XrONg7WO7p0TOH14BqeP6M74vqlheZ1ZfaEqIhFj9/6DvL+ylDkFJcwtLOfAwTpSO8Zx6tAMckd058SBXYmPCY/+eoW7iESkyuoa5q4qZ87yEt5fUcaeqhqS4mM4eWg3cod3Z8rgdBLj2m/vtPrcRSQiJcbFcMZxmZxxXCZVNbV8vGY7cz4v4Z0VpbyyZAvxMVFMGZTOqcMyGNM7hX5pncJmEDTtuYtIxKmprePT9TuZU7CVt5aXUlJxAIDEuGiG9+jM8B7JHNczmeN6JdM/rWPI9terW0ZEpBF1dY7Csj18Xrybgs27KdhSwRdbKth/sBaAhNgohmV25jjvCJ4RPZPJ7tYpJAJf4S4i0gK1dY415Xsp2LybzzcHQn/5lgoqqwOBHx8TxdDMzozoGQj9QRlJZGck0Sm+bXu6Fe4iIseots6xbtu+wN69F/rLt1Swt6rmy2V6JCeQnZFEdrdODMpIYmBG4CSrpITgXI9WX6iKiByj6ChjoDfuzbljegKBLp0NOypZXbqH1WV7v7z/xBsc7ZDM5AQGeoGf3a1T4B9ARic6Byn0QeEuInLUoqKMfmkd6ZfWkdOG/6u9ts5RvLOSwlLvbNrSvRSW7eHJBRs4cPBfod+9cwI/nNyPH07u3+q1KdxFRFpZdJTRp2tH+nTtyKnDMr5sr6tzbN61n0JvD7+wdA/pSfFBqUHhLiLSRqKijKzURLJSEzllaEbTKxzLcwX1p4uIiC8U7iIiYajJcDezR82szMwK6rWlmtk7Zrbau+/itZuZ3WNmRWa2zMzGBrN4ERFpWHP23B8Dcg9r+xnwnnMuG3jPewxwBpDt3WYA97dOmSIi0hJNhrtzLg/YcVjzNGCWNz0LOLde++Mu4BMgxcwyW6tYERFpnqPtc89wzm0F8O67ee09gU31liv22r7CzGaYWb6Z5ZeXlx9lGSIi0pDW/kK1obEyGxzfwDk30zmX45zLSU9Pb+UyREQi29GGe+mh7hbvvsxrLway6i3XC9iCiIi0qaM9iWk2MB34H+/+lXrt15nZM8AJwO5D3TdHsmjRom1mtuEoa0kDth3lusEUqnVB6NamulpGdbVMONbVp7EZTY4KaWZPA1O9AkqB3wAvA88BvVmAlMcAAAUNSURBVIGNwAXOuR1mZsBfCRxdUwlc7pwL6nCPZpbf2KhofgrVuiB0a1NdLaO6WibS6mpyz905991GZp3SwLIOuPZYixIRkWOjM1RFRMJQOIT7TL8LaESo1gWhW5vqahnV1TIRVVdIXIlJRERaVzjsuYuIyGEU7iIiYajdhLuZ5ZrZKm/EyZ81MD/ezJ715i8ws75tUFOWmX1gZivMbLmZ3dDAMlPNbLeZLfFuvw52Xd7zrjezz73n/MrhqH6M4Glmg+tthyVmVmFmPzlsmTbbXi0Z8bSBdad7y6w2s+ltUNcdZrbSe61eMrOURtY94usehLpuNbPN9V6vMxtZ94h/v0Go69l6Na03syWNrBuU7dVYNrTp+8s5F/I3IBpYA/QH4oClwLDDlrkGeMCbvgh4tg3qygTGetNJQGEDdU0FXvNhm60H0o4w/0zgTQJDRkwAFvjwmpYAffzaXsBJwFigoF7b7cDPvOmfAbc1sF4qsNa77+JNdwlyXacBMd70bQ3V1ZzXPQh13Qr8ZzNe6yP+/bZ2XYfNvxP4dVtur8ayoS3fX+1lz308UOScW+ucqwaeITACZX31R6p8ATjFO6kqaJxzW51zi73pPcAKGhkoLQT5PYLnKcAa59zRnpl8zFzLRjyt73TgHefcDufcTuAdvjosdqvW5Zx72zlX4z38hMDQHm2qke3VHM35+w1KXV4GfAd4urWer5k1NZYNbfb+ai/h3pzRJr9cxvsj2A10bZPqAK8baAywoIHZE81sqZm9aWbDG5gfDA5428wWmdmMBuY3ewTPILmIxv/g/NhehzQ24ml9fm+7Kwh86mpIU697MFzndRc92kg3g5/bazJQ6pxb3cj8oG+vw7Khzd5f7SXcmzPaZLNHpGxtZtYJeBH4iXOu4rDZiwl0PYwC/o/A0A1tYZJzbiyBC6hca2YnHTbfz+0VB5wDPN/AbL+2V0v4ue1+CdQATzaySFOve2u7HxgAjAa2EugCOZxv2wv4Lkfeaw/q9moiGxpdrYG2Fm+v9hLuzRlt8stlzCwGSOboPkK2iJnFEnjxnnTO/ePw+c65CufcXm/6DSDWzNKCXZdzbot3Xwa8ROCjcX1+juB5BrDYOVd6+Ay/tlc9jY14Wp8v2877Yu1s4GLndc4erhmve6tyzpU652qdc3XAQ408n1/bKwb4NvBsY8sEc3s1kg1t9v5qL+H+KZBtZv28vb6LCIxAWd+hkSoBzgfeb+wPoLV4/XmPACucc3c1skz3Q33/ZjaewDbfHuS6OppZ0qFpAl/GFRy22GzgUguYQDNH8Gwlje5N+bG9DlP/fVR/xNP63gJOM7MuXjfEaV5b0JhZLnALcI5zrrKRZZrzurd2XfW/p/lWI8/XnL/fYPgGsNI5V9zQzGBuryNkQ9u9v1r7W+Jg3Qgc3VFI4Fv3X3ptvyPwZgdIIPAxvwhYCPRvg5q+RuDj0jJgiXc7E7gKuMpb5jpgOYEjBD4BTmyDuvp7z7fUe+5D26t+XQbc623Pz4GcNnodEwmEdXK9Nl+2F4F/MFuBgwT2ln5A4Hua94DV3n2qt2wO8HC9da/w3mtFBEY/DXZdRQT6YQ+9zw4dGdYDeONIr3uQ6/q79/5ZRiC4Mg+vy3v8lb/fYNbltT926H1Vb9k22V5HyIY2e39p+AERkTDUXrplRESkBRTuIiJhSOEuIhKGFO4iImFI4S4iEoYU7iIiYUjhLiIShv4fmNh13DBIjkoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xcZ33n8c9Pd8mWZdmSncTyLYkNCYTc9HISAiElJDF5sSRcCg6QhpZtForTJUB3w7abpqZdUuhCdyFcAnUJtxjK1bDuhiy5mFKHWM7FiR1iK44vio1mLFnWWLJGt9/+MUfyWB5Zo2jmnNHo+3695qWZc54z88yZ0VePnvM855i7IyIixask6gqIiEh+KehFRIqcgl5EpMgp6EVEipyCXkSkyJVFXYGxGhoafNmyZVFXQ0RkWtm2bdthd2/MtK7ggn7ZsmW0tLREXQ0RkWnFzPaNt05dNyIiRU5BLyJS5BT0IiJFTkEvIlLkFPQiIkVOQS8iUuQU9CIiRa7gxtGLiMw0B7uOs3lXnCF33n/Z0pw/v4JeRCRkfQNDPL6ng827DrN5d5zW2DEALl4yN7qgN7PVwP8CSoFvuPs9Y9YvBdYDjUAn8AF3bwvWDQHPBkX3u/vbc1R3EZFpwd3ZHTvG5l1xHtsV54mXOkkODlNRVsJly+fx3ubFXLWykZULZ+fl9ScMejMrBe4FrgXagK1mttHdd6YV+wfgW+5+v5m9GfgMcEuw7ri7X5TjeouI5FSib4B9Hb283HWc6vJS6qrLmVtTztzqCmqryigpsUk9X1dvP//WepjNu+L8evdhDh3tA+DcBbN5/2VLuWplA5ctn091RWk+3s5JsmnRrwJa3X0PgJltAG4E0oP+fOCO4P4jwE9zWUkRmVkOdPZy4Egvc6qCsK2pYFZFKWaTC9t07k5nTz/7OnvZ19HDvo5e9nX0srejh/0dvXT09I+7rRkn6lJdTl1NReoPQfDHoK46dautKuf5Q91s3h3nmQNdDDvUVpXxhnMb+PNrGrlqZSOL5la/4vfwSmUT9IuAA2mP24DLxpR5BngXqe6ddwC1Zjbf3TuAKjNrAQaBe9z9lD8CZnYbcBvAkiVLJv0mRGR66+0fHO2zfmxXnJcO95xSpqzETgrVuTUVQeimWt2j62rKqSgtoe1I72iY7+vsYd/hXhLJwdHnM4Oz6qpZOr+G616zkKXzZ7F0Xg1N9TX0Dw3R1TuQuh0f4OjxAY729tN1PLXs6PEBDnT20tXbz9HjAwynXXrbDC5smsvaN6/gTSsbuLBpLmWl0Q5wzCboM/0JHXtF8U8CXzKzDwKbgZdJBTvAEnc/aGZnAw+b2bPu/uJJT+Z+H3AfQHNzs65WLlLk3J3nDyXYvDvO5l1xWvYeoX9omKryEi4/ez63XL6UV59RS3ffIEeP94+Ga9fxAY72DtB1vJ9Yoo9d7QmO9g6cFODpykqMxfNqWDKvhkuX1KfCfH4NS+fPoqm+mqryqXebDA87x/oHORrUcdHcaupnVUz5eXMpm6BvAxanPW4CDqYXcPeDwDsBzGw28C53P5q2DnffY2aPAhcDJwW9iBQed+dI7wBlpUZtZdmUuk0AOo4l+bfWVIv917sPE08kAXj1GbV88MplXLWikeZl9a8ofAeHhunuG6QraHX3DQyxuL6GM+uq8t6aLikx5lSVM6eq/KSgLCTZBP1WYIWZLSfVUl8DvC+9gJk1AJ3uPgx8itQIHMysHuh192RQ5krgszmsv8iMkBwcoqK0ZMphO9bwsPP77r7Rfuq9Hen91z309A8BUFpio33Sc6pP9FXPralIPR5ZNtq1kurDfulwz+hIk+cOHsUd5taU84ZzG7hqZSNXrWjkjLqqKb+PstIS5s2qYF6BtaQLxYRB7+6DZrYWeJDU8Mr17r7DzNYBLe6+Ebga+IyZOamum48Gm58HfM3MhknNwr1nzGgdERlH38AQv9h+iG9v2cszbUepKCsZc/Cv4sTBwZHlI/3WaSNGqipKONQVhHlnL3sP97K/s4e9Hb3s7+ylf3B49DXLS43F9TUsmV/DquXzaKqvxh26jqf6oke6UDqO9fNi/BhHewfo7svcbTKitMS4ePFc7njLSq5a2cgFi+ooneQIFpkacy+sLvHm5mbXFaZkJjvQ2ct3Ht/H91sO0NU7wLkLZnPDa88gOTQ82g88epAwOBg40vLORnV5adBPXXOiz3pe6udZc6snHcJDw0738RMHLUcOUHb1DrBwTiVXnNNAXXX5ZHeDTJKZbXP35kzrNDNWpAAMDzuP7YrzrS17eXRXnBIzrjt/IbdcsZQrzp4/YZdN/+BwamRIWst75I9Bb3KQM+qqWDp/Fsvm19BYW5nTLqDSEqN+VkXBHYCUExT0IhE60tPPv2w7wHce38/+zl4aZldy+x+cy82XLeHMuuzHW1eUldBYW0ljbWUeayvTlYJeQuXufP3Xe/jxky9HXZVXZN6sihNdHvNOdH3Mqpzcr9L2ti6+tWUfP3/mIMnBYVYtn8dfXP8qrn/NGVSU6aSyklsKeglN38AQd/5oOz99+iCXLq2nYfb0+lffHeLHkvxyR/spsygbZleybH7qIOaytLHay+bXMLcm9T5HD64+vo9nDnRRU1HKuy9t4pYrlvLqM+ZE8ZZkhlDQSyhiiT7+07e38dT+Lj5x7UrWvvncnA8VDNPIeVHSp9Dv7ehhy4sdp/y3MqeqjKXzZ9F2pJcjvQOc0ziLv3n7a3jHJYuYU6WDlJJ/CnrJux0Hj/Kn97fQ2dvPV95/CW+94MyoqzRltVXlvHZRHa9dVHfKur6BIQ50njwmfW9HD0vnN/C+VUu44pyJD66K5JKCXvLqwR2/52Mbnqauupwffvj1GYOx2FSVl7JiYS0rFtZGXRURQEEveeLufPnRF/ncgy9wYVMdX/+jZhbMmfoMSBGZPAW95FzfwBCf+vGz/OSpl/kPF57F5979upycPEpEXhkFveRUsR10FSkGCnrJmWI86CpSDBT0khMz8aCryHShoJ/B9nf00j80zOJ51VSWvbI+dB10FSl8CvoZyN3559/s5W//z06G/eRLqk1mer8OuopMDwr6GaZ/cJi/3vgcDzxxgGvPX8gNF5wRnJ88NalnvOn9o38E5s1iWUMNZ8yp4p7/+zue2t/Fx69dye066CpSsBT0M8iRnn4+/J1t/PalTv7s6nP45HWvoiTDucfTp/ePXFR5X+ep0/urykt00FVkGlDQzxCtsQQfur+FQ119fP49F/LOS5rGLZvN9P59Hb2cu2A2yxpm5bPaIpIDCvoZ4NEXYtz+vaeoLC/hgdsu59Kl9a/4uTS9X2T6UdAXsfSDrisX1vKNW5tpqq+JuloiEjIFfZEaGBrmrp/t4IEn9nPd+Qv5wnsvmvTFMUSkOOg3vwgd6ennI9/dxuN7Tn/QVURmhqyuWWZmq83sBTNrNbM7M6xfama/MrPtZvaomTWlrbvVzHYHt1tzWXk5VWsswU1f/g1P7uvi8++5kP+y+tUKeZEZbsIWvZmVAvcC1wJtwFYz2+juO9OK/QPwLXe/38zeDHwGuMXM5gF/DTQDDmwLtj2S6zci8NiuOGu/+2RODrqKSPHIpkW/Cmh19z3u3g9sAG4cU+Z84FfB/UfS1l8PPOTunUG4PwSsnnq1JV3qoOtL/PE/P8Gi+mp++tErFfIiMiqboF8EHEh73BYsS/cM8K7g/juAWjObn+W2mNltZtZiZi3xeDzbugupg67/7SfP8Tc/38k15y3kRx95vUbWiMhJsgn6TB28PubxJ4E3mdlTwJuAl4HBLLfF3e9z92Z3b25sbMyiSgLQGjvGLf/0Wx54Yj8fufocvvaBSzWyRkROkU0qtAGL0x43AQfTC7j7QeCdAGY2G3iXux81szbg6jHbPjqF+grw/KFuvvRIK5uePURVWemEM11FZGbLJui3AivMbDmplvoa4H3pBcysAeh092HgU8D6YNWDwP8ws5EO4+uC9fIKbG/r4osPt/LQznZmV5bxkTedw4fesJz5syujrpqIFLAJg97dB81sLanQLgXWu/sOM1sHtLj7RlKt9s+YmQObgY8G23aa2adJ/bEAWOfunXl4H0WtZW8nX3y4lcd2xamrLueOt6zkg69fRl1NedRVE5FpwNxP6TKPVHNzs7e0tERdjci5O1te7OCLD7eyZU8H82ZV8B/fuJxbLl9KbZUCXkROZmbb3L050zoduSsw7s6ju+J86eFWtu07woLaSv77287n5lWLqanQxyUik6fkKBDDw85Dz7fzpYdbefbloyyaW82nb3otf3hpk67aJCJToqCPmLvzi+2HuPeRVn73+wRL59fw2Xe9jpsuXkRFWVZnqBAROS0FfcR+vv0Qf/7AU5y7YDb/+N6LeNvrzqSsVAEvIrmjoI/Y7w51U1ZiPPixqyjVycdEJA/UdIxYLJGksbZSIS8ieaOgj1gskWRBrSY8iUj+KOgjFuvuY8GcqqirISJFTEEfMbXoRSTfFPQR6h8cprOnnwW1atGLSP4o6CN0+FgSgAVz1KIXkfxR0EeovbsPgIUKehHJIwV9hGKJoEWvrhsRySMFfYROBL1a9CKSPwr6CMW7+ygxdOEQEckrBX2E2ruTNMzWrFgRyS8FfYRiiT6NuBGRvFPQRyg1WUoHYkUkvxT0EdKsWBEJg4I+IoNDw3QcU9CLSP4p6CPS0dPPsKMTmolI3mUV9Ga22sxeMLNWM7szw/olZvaImT1lZtvN7IZg+TIzO25mTwe3r+b6DUxXsW6NoReRcEx4hSkzKwXuBa4F2oCtZrbR3XemFfsr4Afu/hUzOx/YBCwL1r3o7hflttrTXyyROv2BWvQikm/ZtOhXAa3uvsfd+4ENwI1jyjgwJ7hfBxzMXRWLk2bFikhYsgn6RcCBtMdtwbJ0dwMfMLM2Uq3529PWLQ+6dB4zszdmegEzu83MWsysJR6PZ1/7aWzkhGaNCnoRybNsgj7TtE0f8/hm4Jvu3gTcAHzbzEqAQ8ASd78Y+DjwPTObM2Zb3P0+d2929+bGxsbJvYNpKpZIMn9WBeWlOh4uIvmVTcq0AYvTHjdxatfMh4AfALj7FqAKaHD3pLt3BMu3AS8CK6da6WIQ606qNS8iocgm6LcCK8xsuZlVAGuAjWPK7AeuATCz80gFfdzMGoODuZjZ2cAKYE+uKj+dxRO6VqyIhGPCUTfuPmhma4EHgVJgvbvvMLN1QIu7bwQ+AXzdzO4g1a3zQXd3M7sKWGdmg8AQ8GF378zbu5lG2ruTrFxYG3U1RGQGmDDoAdx9E6mDrOnL7kq7vxO4MsN2PwJ+NMU6Fp3hYefwsaROaCYiodCRwAh09vYzOOw6oZmIhEJBHwHNihWRMCnoI9CuWbEiEiIFfQTiatGLSIgU9BEYOc+NxtGLSBgU9BGIJZLUVZdTVV4adVVEZAZQ0EegvbuPhRpaKSIhUdBHQNeKFZEwKegjEOvWJQRFJDwK+pC5O/FEkkZ13YhISBT0IevqHaB/aJiF6roRkZAo6EM2emUptehFJCQK+pCNXitWLXoRCYmCPmQ6z42IhE1BH7IT57lR0ItIOBT0IYt1J6mtLKOmIqtLAYiITJmCPmQaWikiYVPQhyyW6FP/vIiESkEfsvZunf5ARMKloA+RuxNL6IRmIhIuBX2IEslB+gaG1aIXkVBlFfRmttrMXjCzVjO7M8P6JWb2iJk9ZWbbzeyGtHWfCrZ7wcyuz2Xlp5vRMfRq0YtIiCYc42dmpcC9wLVAG7DVzDa6+860Yn8F/MDdv2Jm5wObgGXB/TXAa4CzgP9nZivdfSjXb2Q6iHXrylIiEr5sWvSrgFZ33+Pu/cAG4MYxZRyYE9yvAw4G928ENrh70t1fAlqD55uRRs5zs1AXBReREGUT9IuAA2mP24Jl6e4GPmBmbaRa87dPYlvM7DYzazGzlng8nmXVp58T57lRi15EwpNN0FuGZT7m8c3AN929CbgB+LaZlWS5Le5+n7s3u3tzY2NjFlWanmLdSarLS5ldqVmxIhKebBKnDVic9riJE10zIz4ErAZw9y1mVgU0ZLntjNGeSLJgTiVmmf7+iYjkRzYt+q3ACjNbbmYVpA6ubhxTZj9wDYCZnQdUAfGg3BozqzSz5cAK4IlcVX66iXX36YIjIhK6CVv07j5oZmuBB4FSYL277zCzdUCLu28EPgF83czuINU180F3d2CHmf0A2AkMAh+dqSNuIHWem/POmjNxQRGRHMqqs9jdN5E6yJq+7K60+zuBK8fZ9u+Av5tCHYtGLJHkTToQKyIh08zYkPQkBzmWHNSsWBEJnYI+JCfG0KtFLyLhUtCHZGRWrFr0IhI2BX1IRlr0Os+NiIRNQR+S9m7NihWRaCjoQxJPJKkoK6GuujzqqojIDKOgD0kskWRBrWbFikj4FPQh0bViRSQqCvqQxHStWBGJiII+JO3dulasiERDQR+CvoEhuvsGWaALjohIBBT0IYgHY+h1CUERiYKCPgS6spSIRElBH4L2bl0rVkSio6APQUyzYkUkQgr6EMQSScpKjPqaiqirIiIzkII+BLFEksbaSkpKNCtWRMKnoA9Be7dmxYpIdBT0IYgnkhpDLyKRUdCHYOSEZiIiUVDQ51n/4DCdPf06z42IRCaroDez1Wb2gpm1mtmdGdZ/wcyeDm67zKwrbd1Q2rqNuaz8dBA/pitLiUi0yiYqYGalwL3AtUAbsNXMNrr7zpEy7n5HWvnbgYvTnuK4u1+UuypPLyNj6HVCMxGJSjYt+lVAq7vvcfd+YANw42nK3ww8kIvKFYPRa8Wq60ZEIpJN0C8CDqQ9bguWncLMlgLLgYfTFleZWYuZPW5mN42z3W1BmZZ4PJ5l1aeHE0GvFr2IRCOboM80y8fHKbsG+KG7D6UtW+LuzcD7gH80s3NOeTL3+9y92d2bGxsbs6jS9BHr7qPEYP5sBb2IRCOboG8DFqc9bgIOjlN2DWO6bdz9YPBzD/AoJ/ffF71Yd5KG2ZWUalasiEQkm6DfCqwws+VmVkEqzE8ZPWNmrwLqgS1py+rNrDK43wBcCewcu20xiyX6NOJGRCI14agbdx80s7XAg0ApsN7dd5jZOqDF3UdC/2Zgg7und+ucB3zNzIZJ/VG5J320zkwQSyR1emIRidSEQQ/g7puATWOW3TXm8d0Ztvt34IIp1G/aa+9OcsGiuqirISIzmGbG5tHg0DAdPTrPjYhES0GfRx09/bhraKWIREtBn0exbo2hF5HoKejzqH3kEoLquhGRCCno82hkVqzOcyMiUVLQ51Es0YcZNGhWrIhESEGfR7FEknk1FZSXajeLSHSUQHkU6+6jUQdiRSRiCvo80qxYESkECvo8inXrWrEiEj0FfZ4MDzvxY0md0ExEIqegz5OOnn6Ghl1XlhKRyCno8ySW0LViRaQwKOjzZGSyVKNa9CISMQV9nsR1nhsRKRAK+jwZOc+NxtGLSNQU9HkSSySpqy6nqrw06qqIyAynoM+TWKJPB2JFpCAo6PMklkhqaKWIFAQFfZ5oVqyIFAoFfR64O/FEkkZ13YhIAcgq6M1stZm9YGatZnZnhvVfMLOng9suM+tKW3erme0ObrfmsvKFqqt3gP6hYRaq60ZECkDZRAXMrBS4F7gWaAO2mtlGd985Usbd70grfztwcXB/HvDXQDPgwLZg2yM5fRcFZmSylM5zIyKFIJsW/Sqg1d33uHs/sAG48TTlbwYeCO5fDzzk7p1BuD8ErJ5KhaeDkdMf6GCsiBSCbIJ+EXAg7XFbsOwUZrYUWA48PJltzew2M2sxs5Z4PJ5NvQtau2bFikgBySboLcMyH6fsGuCH7j40mW3d/T53b3b35sbGxiyqVNhGW/TquhGRApBN0LcBi9MeNwEHxym7hhPdNpPdtmjEupPUVpZRUzHhIRARkbzLJui3AivMbLmZVZAK841jC5nZq4B6YEva4geB68ys3szqgeuCZUVNQytFpJBM2OR090EzW0sqoEuB9e6+w8zWAS3uPhL6NwMb3N3Ttu00s0+T+mMBsM7dO3P7FgpPe3ef+udFpGBk1bfg7puATWOW3TXm8d3jbLseWP8K6zctxRJJLl4yN+pqiIgAmhmbc+5OLKEWvYgUDgV9jiWSg/QNDGsMvYgUDAV9jsW6NbRSRAqLgj7HYqOTpdSiF5HCoKDPMZ3nRkQKjYI+x06c50ZBLyKFQUGfY+3dSarLS5ldqVmxIlIYFPQ5FkskWTinErNMp/kREQmfgj7HYt19OhArIgVFQZ9jOs+NiBQaBX2O6Tw3IlJoFPQ51JMcpKd/iIVz1HUjIoVDQZ9Do2Po1aIXkQKioM+h0dMf6GCsiBQQBX0OtWtWrIgUIAV9Dp1o0SvoRaRwKOhzKJ5IUlFWQl11edRVEREZpaDPoVgiyYJazYoVkcKioM8hjaEXkUKkoM+hVIteI25EpLAo6HMo1t3HQo24EZECk1XQm9lqM3vBzFrN7M5xyrzHzHaa2Q4z+17a8iEzezq4bcxVxQtN38AQ3X2DLNCsWBEpMBOeNN3MSoF7gWuBNmCrmW10951pZVYAnwKudPcjZrYg7SmOu/tFOa53wRm5hGCj+uhFpMBk06JfBbS6+x537wc2ADeOKfOnwL3ufgTA3WO5rWbh05WlRKRQZRP0i4ADaY/bgmXpVgIrzew3Zva4ma1OW1dlZi3B8psyvYCZ3RaUaYnH45N6A4Vi5Dw3OqGZiBSabK53l2lQuGd4nhXA1UAT8Gsze627dwFL3P2gmZ0NPGxmz7r7iyc9mft9wH0Azc3NY597WtCsWBEpVNm06NuAxWmPm4CDGcr8zN0H3P0l4AVSwY+7Hwx+7gEeBS6eYp0LUnsiSVmJUV9TEXVVREROkk3QbwVWmNlyM6sA1gBjR8/8FPgDADNrINWVs8fM6s2sMm35lcBOilCsO0ljbSUlJZoVKyKFZcKuG3cfNLO1wINAKbDe3XeY2Tqgxd03BuuuM7OdwBDwF+7eYWavB75mZsOk/qjckz5ap5jEEn0aWikiBSmbPnrcfROwacyyu9LuO/Dx4JZe5t+BC6ZezcIXTyRZPK8m6mqIiJxCM2NzROe5EZFCpaDPgf7BYY70Dug8NyJSkBT0ORA/NjKGXi16ESk8CvocGB1Dr6AXkQKkoM+B9uA8N+q6EZFCpKDPgbjOcyMiBUxBnwOxRJISg/mzFfQiUniyGkc/HXT19vOHX90SyWvHEkkaZldSqlmxIlKAiiboS0qMFQtnR/LaKxbO5oqz50fy2iIiEymaoJ9TVc6X339p1NUQESk46qMXESlyCnoRkSKnoBcRKXIKehGRIqegFxEpcgp6EZEip6AXESlyCnoRkSJnqasAFg4ziwP7pvAUDcDhHFUnl1SvyVG9Jkf1mpxirNdSd2/MtKLggn6qzKzF3ZujrsdYqtfkqF6To3pNzkyrl7puRESKnIJeRKTIFWPQ3xd1Bcahek2O6jU5qtfkzKh6FV0fvYiInKwYW/QiIpJGQS8iUuSmZdCb2Woze8HMWs3szgzrK83s+8H635rZshDqtNjMHjGz581sh5n95wxlrjazo2b2dHC7K9/1SnvtvWb2bPC6LRnWm5n972CfbTezS0Ko06vS9sXTZtZtZh8bUyaUfWZm680sZmbPpS2bZ2YPmdnu4Gf9ONveGpTZbWa3hlCvz5nZ74LP6SdmNnecbU/7meehXneb2ctpn9UN42x72t/fPNTr+2l12mtmT4+zbT73V8Z8CO075u7T6gaUAi8CZwMVwDPA+WPK/Bnw1eD+GuD7IdTrTOCS4H4tsCtDva4GfhHRftsLNJxm/Q3AvwIGXA78NoLP9fekJn2Evs+Aq4BLgOfSln0WuDO4fyfw9xm2mwfsCX7WB/fr81yv64Cy4P7fZ6pXNp95Hup1N/DJLD7n0/7+5rpeY9b/T+CuCPZXxnwI6zs2HVv0q4BWd9/j7v3ABuDGMWVuBO4P7v8QuMbM8nrlbnc/5O5PBvcTwPPAony+Zo7dCHzLUx4H5prZmSG+/jXAi+4+lVnRr5i7bwY6xyxO/x7dD9yUYdPrgYfcvdPdjwAPAavzWS93/6W7DwYPHweacvV6U6lXlrL5/c1LvYIMeA/wQK5eL1unyYdQvmPTMegXAQfSHrdxaqCOlgl+IY4CoV29O+gquhj4bYbVV5jZM2b2r2b2mrDqBDjwSzPbZma3ZVifzX7NpzWM/wsY1T5b6O6HIPWLCizIUCbq/fYnpP4Ty2Sizzwf1gZdSuvH6YaIcn+9EWh3993jrA9lf43Jh1C+Y9Mx6DO1zMeOEc2mTF6Y2WzgR8DH3L17zOonSXVNXAh8EfhpGHUKXOnulwBvBT5qZleNWR/lPqsA3g78S4bVUe6zbES53/4SGAS+O06RiT7zXPsKcA5wEXCIVDfJWJHtL+BmTt+az/v+miAfxt0sw7JJ7bPpGPRtwOK0x03AwfHKmFkZUMcr+zdzUsysnNSH+F13//HY9e7e7e7HgvubgHIza8h3vYLXOxj8jAE/IfUvdLps9mu+vBV40t3bx66Icp8B7SPdV8HPWIYykey34IDc24D3e9CRO1YWn3lOuXu7uw+5+zDw9XFeL6r9VQa8E/j+eGXyvb/GyYdQvmPTMei3AivMbHnQElwDbBxTZiMwcmT63cDD4/0y5ErQ//dPwPPu/vlxypwxcqzAzFaR2v8d+axX8FqzzKx25D6pg3nPjSm2EfgjS7kcODryL2UIxm1pRbXPAunfo1uBn2Uo8yBwnZnVB10V1wXL8sbMVgP/FXi7u/eOUyabzzzX9Uo/pvOOcV4vm9/ffHgL8Dt3b8u0Mt/76zT5EM53LB9HmPN9IzVCZBepo/d/GSxbR+qLD1BFqhugFXgCODuEOr2B1L9T24Gng9sNwIeBDwdl1gI7SI00eBx4fUj76+zgNZ8JXn9kn6XXzYB7g336LNAcUt1qSAV3Xdqy0PcZqT80h4ABUi2oD5E6rvMrYHfwc15Qthn4Rtq2fxJ811qBPw6hXq2k+mxHvmcjI8zOAjad7jPPc72+HXx3tpMKsDPH1it4fMrvbz7rFSz/5sh3Kq1smPtrvHwI5TumU3WnQUEAAAA0SURBVCCIiBS56dh1IyIik6CgFxEpcgp6EZEip6AXESlyCnoRkSKnoBcRKXIKehGRIvf/AYifWf5EzIsVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(score)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reference\n",
    "\n",
    "[Gradient Boosting (Wikipedia)](https://en.wikipedia.org/wiki/Gradient_boosting)\n",
    "\n",
    "[Gradient Boost Part 3: Classification -- Youtube StatQuest](https://www.youtube.com/watch?v=jxuNLH5dXCs)\n",
    "\n",
    "[Gradient Boost Part 4: Classification Details -- Youtube StatQuest](https://www.youtube.com/watch?v=StWY5QWMXCw)\n",
    "\n",
    "[sklearn.tree.DecisionTreeRegressor -- scikit-learn 0.21.3 documentation](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html)\n",
    "\n",
    "[Understanding the decision tree structure -- scikit-learn 0.21.3 documentation](https://scikit-learn.org/stable/auto_examples/tree/plot_unveil_tree_structure.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}