Car Evaluation using Decision trees and Random Forests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Car Evaluation using Decision trees and Random Forests\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision tree learning\n",
    "\n",
    "Decision tree classifiers are attractive models of Machine Learning as they emphasize on interpretability.\n",
    "Like the name decision tree suggests, we can think of this model as breaking down our data by making decisions based on asking a series of questions. \n",
    "\n",
    "Let's consider the following example where we use a decision tree to decide upon an activity on a particular day:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src=\"images/decisiontrees.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[[Image source]]( https://github.com/rasbt/python-machine-learning-book/tree/master/images/image_gallery)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decision tree model learns a series of questions to infer the class labels of the samples according to the features in our training set. Although the preceding figure illustrated the concept of a decision tree based on categorical variables, the same concept applies to our features. Using the decision algorithm, we start at the tree root and split the data on the feature that results in the largest information gain (IG). In an iterative process, we can then repeat this splitting procedure at each child node until the leaves are pure. This means that the samples at each node all belong to the same class. In practice, this can result in a very deep tree with many nodes, which can easily lead to overfitting. Thus, we typically want to prune the tree by setting a limit for the maximal depth of the tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters in decision trees\n",
    "\n",
    "One of the most important features for a decision tree is the stopping criterion.\n",
    "As a tree is built, the final few decisions can often be somewhat random and rely\n",
    "on only a small number of samples to make their decision. Using such specific nodes\n",
    "can result in overfitting of the training data. Instead, a stopping\n",
    "criterion can be used to ensure that the decision tree does not reach this exactness.\n",
    "Instead of using a stopping criterion, the tree could be created in full and then\n",
    "pruned. This pruning process removes nodes that do not provide much\n",
    "information to the overall process. The decision tree implementation in scikit-learn provides a method to stop the\n",
    "building of a tree using the following options:\n",
    "    \n",
    "* min_samples_split: This specifies how many samples are needed in order to create a new node in the decision tree\n",
    "* min_samples_leaf: This specifies how many samples must be resulting from a node for it to stay\n",
    "\n",
    "The first dictates whether a decision node will be created, while the second dictates\n",
    "whether a decision node will be kept.\n",
    "Another parameter for decision tress is the criterion for creating a decision.\n",
    "Gini impurity and Information gain are two popular ones:\n",
    "\n",
    "* Gini impurity: This is a measure of how often a decision node would incorrectly predict a sample's class\n",
    "* Information gain: This uses information-theory-based entropy to indicate how much extra information is gained by the decision node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a decision tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision trees can build complex decision boundaries by dividing the feature space into rectangles. Generally Decision trees with lower heights are preferred over deeper trees, since the deeper the decision tree, the more complex the decision boundary becomes, which can easily result in overfitting. Using scikit-learn, we will now train a decision tree to evaluation the condition of a car."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "To implement our Decision Tree Classifier we will use The Car Evaluation Database. It contains examples with the structural information removed, i.e., it directly relates CAR to the six input attributes: buying, maint, doors, persons, lug_boot, safety. \n",
    "Basically, we have to build a classifier to classify a car as 'Unacceptable', 'Acceptable', 'Good' and 'Very Good' based on the attributes. \n",
    "\n",
    "The different attributes values are given as follows:\n",
    "\n",
    "1. **buying**: vhigh, high, med, low. \n",
    "2. **maint**: vhigh, high, med, low. \n",
    "3. **doors**: 2, 3, 4, 5, more. \n",
    "4. **persons**: 2, 4, more. \n",
    "5. **lug_boot**: small, med, big. \n",
    "6. **safety**: low, med, high. \n",
    "\n",
    "You can download the dataset from here : https://archive.ics.uci.edu/ml/datasets/Car+Evaluation\n",
    "\n",
    "Once downloaded, we can move on with the code.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, we manage our imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from sklearn.tree import DecisionTreeClassifier, export_graphviz\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import cross_validation, metrics\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from time import time\n",
    "from sklearn import preprocessing\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import roc_auc_score , classification_report\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.metrics import precision_score, recall_score, accuracy_score, classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we load our dataset into a pandas dataframe. We specify the column names before-hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read .csv from provided dataset\n",
    "csv_filename=\"car.data\"\n",
    "\n",
    "# df=pd.read_csv(csv_filename,index_col=0)\n",
    "df=pd.read_csv(csv_filename,\n",
    "              names=[\"Buying\", \"Maintenance\" , \"Doors\" , \"Persons\" , \"Lug-Boot\" , \"Safety\", \"Class\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Buying</th>\n",
       "      <th>Maintenance</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Persons</th>\n",
       "      <th>Lug-Boot</th>\n",
       "      <th>Safety</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>small</td>\n",
       "      <td>low</td>\n",
       "      <td>unacc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>small</td>\n",
       "      <td>med</td>\n",
       "      <td>unacc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>small</td>\n",
       "      <td>high</td>\n",
       "      <td>unacc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>med</td>\n",
       "      <td>low</td>\n",
       "      <td>unacc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>med</td>\n",
       "      <td>med</td>\n",
       "      <td>unacc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Buying Maintenance Doors Persons Lug-Boot Safety  Class\n",
       "0  vhigh       vhigh     2       2    small    low  unacc\n",
       "1  vhigh       vhigh     2       2    small    med  unacc\n",
       "2  vhigh       vhigh     2       2    small   high  unacc\n",
       "3  vhigh       vhigh     2       2      med    low  unacc\n",
       "4  vhigh       vhigh     2       2      med    med  unacc"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the target class labels are strings, we'll have to convert in a format that our classifier would understand. For this we would use the LabelEncoder class of Scikit-Learn module 'preprocessing'.  This converts our class labels into [1,2,3,4] where the integers would correspond to the respective class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Convert car-class labels to numbers\n",
    "le = preprocessing.LabelEncoder()\n",
    "df['Class'] = le.fit_transform(df.Class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 0, 3, 1], dtype=int64)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Class'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features = list(df.columns)\n",
    "features.remove('Class')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our features. Next we convert the dataset into binarized form. This makes the raw textual data which contains categorized values like vhigh etc and converts them into a dummy format better understandable to our classifier. We use the pd.get_dummies function to do it for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for f in features:\n",
    "    #Get binarized columns\n",
    "    df[f] = pd.get_dummies(df[f])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our data looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Buying</th>\n",
       "      <th>Maintenance</th>\n",
       "      <th>Doors</th>\n",
       "      <th>Persons</th>\n",
       "      <th>Lug-Boot</th>\n",
       "      <th>Safety</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Buying  Maintenance  Doors  Persons  Lug-Boot  Safety  Class\n",
       "0       0            0      1        1         0       0      2\n",
       "1       0            0      1        1         0       0      2\n",
       "2       0            0      1        1         0       1      2\n",
       "3       0            0      1        1         0       0      2\n",
       "4       0            0      1        1         0       0      2"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df[features]\n",
    "y = df['Class']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the dataset into 60% training and 40% testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# split dataset to 60% training and 40% testing\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "sc.fit(X_train)\n",
    "X_train_std = sc.transform(X_train)\n",
    "X_test_std = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the Feature importances with forests of trees\n",
    "This examples shows the use of forests of trees to evaluate the importance of features on an artificial classification task. The red bars are the feature importances of the forest, along with their inter-trees variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature ranking:\n",
      "1. feature 3 - Persons (0.536901) \n",
      "2. feature 5 - Safety (0.318936) \n",
      "3. feature 0 - Buying (0.048373) \n",
      "4. feature 4 - Lug-Boot (0.047934) \n",
      "5. feature 1 - Maintenance (0.027871) \n",
      "6. feature 2 - Doors (0.019986) \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABGwAAANDCAYAAAD4rYkxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3X+YlXWd//HXzIDyQwQUHExSQFegVJBRI1v8BRtGaV6u\nUnPJIrolNhkF1m5ZYePvth/TWiBqarayRK61ZVZkg7XrRkXCtqZI/kgIy9mmFDPEgDnfP/wy6wQI\ngwIf4fG4rnNdzOfc97nf98z5Q5/Xfe5TValUKgEAAACgGNW7egAAAAAAOhJsAAAAAAoj2AAAAAAU\nRrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAIBXvRUrVqS6ujpf\n/vKXd/UoAACvCMEGAF5lbr311lRXV2/2cckll+yw437nO99JY2PjDnv9l6uqqmpXj/CyXHfddbn1\n1lt39RgAQCG67OoBAIDOq6qqyuWXX55BgwZ1WD/iiCN22DG//e1vZ/bs2bn00kt32DG21yGHHJLn\nnnsuXbt23dWjbLfZs2enf//+Offcc3f1KABAAQQbAHiVOvXUUzNq1KiddrxKpbJDXnfNmjXp0aPH\ny36dvfba6xWYZud77rnn0r179109BgBQGB+JAoDd2G233ZZjjjkmPXr0yP7775/6+vqsWrWqwzb3\n3ntvJk6cmEMOOSTdunXLwQcfnBkzZmTt2rXt25x33nmZPXt2krR//KqmpiZJ8oMf/CDV1dX5j//4\njw6vu7n7ykyZMiW9evXKY489lgkTJmTffffNpEmT2p//yU9+klNPPTV9+vRJz549c9JJJ+VHP/rR\nVs/zpY7161//Om9729vSq1evDBw4sP087r///owdOzb77LNPBg0alHnz5nV4zY0fPfvP//zPTJ06\nNf369Uvv3r1z7rnn5umnn95khtmzZ+eII45It27dctBBB+Wiiy7K6tWrO2xz0kkn5aijjsqSJUty\nwgknpGfPnrnkkksyePDgPPDAA+2/y+rq6pxyyilJkqeeeiof/OAHc9RRR6VXr17p3bt3JkyYkP/5\nn//p8No//OEPU11dndtvvz1XXnllXvva16Z79+4ZN25cHn300U3m/clPfpIJEyZkv/32yz777JMR\nI0bk2muv7bDN8uXLc9ZZZ2X//fdP9+7dc+yxx+bOO+/ssM369evT2NiYww8/PN27d0+/fv0yZsyY\nNDc3b+3PBgC8BFfYAMCr1OrVq/P73/++w9r+++/f/u8rr7wyM2fOzDvf+c68+93vzu9+97tce+21\nOfHEE7N06dLsu+++SZLbb789zz33XBoaGrL//vvnpz/9aT7/+c/niSeeyPz585MkF154YX7zm9/k\n+9//fubOndvhapuqqqptvn9MVVVV1q9fn/Hjx2fMmDH5zGc+0351zcKFCzNhwoQcc8wx+cQnPpHq\n6urccsstOeWUU3LvvffmmGOO6dTvp6qqKm1tbXnLW96SE088MZ/61Kcyd+7cvO9970vPnj3z0Y9+\nNJMmTcrf/u3fZs6cOTn33HNz/PHH55BDDunwOhdddFH69u2bxsbGLF++PLNnz87KlStzzz33tG/z\niU98Ipdddlne/OY3p6GhoX27n/3sZ/mv//qv9rhVVVWV1tbWTJgwIe985zszefLk1NbW5uSTT85F\nF12UXr165WMf+1gqlUpqa2uTJI899li++c1v5uyzz87gwYPT0tKS66+/PieddFIefPDBDBgwoMO8\n11xzTWpqavKhD30oq1evzic/+clMmjQpixYtat/m7rvvzmmnnZbXvOY1+cAHPpABAwZk2bJlueuu\nuzJt2rQkyQMPPJC//uu/zsCBA/ORj3wkPXv2zFe/+tWcccYZ+drXvpa3v/3tSZJLL70011xzTS64\n4IIce+yxeeaZZ/Kzn/0sS5YsydixYzv1NwMAXqQCALyqfOlLX6pUVVVt8qiurm7fZsWKFZUuXbpU\nrrnmmg77PvDAA5WuXbtWrr766va1tWvXbnKMa665plJTU1P59a9/3b520UUXdTjGRj/4wQ8q1dXV\nlR/+8Icd1h9//PFKVVVV5dZbb21fmzJlSqW6urry0Y9+dJPXOfzwwysTJkzosLZ27drKkCFDKuPH\nj9/Sr2Orx/rkJz/Zvvb0009XevToUampqancfvvt7evLly+vVFVVVRobG9vXNv6ejzvuuMr69evb\n1z/1qU9VqqurK3feeWelUqlUfve731X23nvvylve8pYOM82aNatSXV1d+dKXvtS+dtJJJ1Wqq6sr\nN9544ybncMQRR1ROPvnkTdb//Oc/b7K2YsWKSrdu3SpXXHFF+9oPfvCDSlVVVeX1r399h3mvvfba\nSnV1deWBBx6oVCqVyoYNGyqDBw+uDBkypPLMM89s8tobjR07tjJy5MjKunXrOqy/6U1vqgwdOrT9\n55EjR1ZOO+20Lb4OALB9fCQKAF6Fqqqqct111+X73/9+++Puu+9uf/6OO+5IpVLJ2Wefnd///vft\njwMOOCB/9Vd/1eHqkL333rv932vWrMnvf//7vPGNb0xbW1uWLl26Q+a/8MILO/z83//933n44YdT\nX1/fYd4//vGPGTt27CYft+qMv//7v2//d+/evTN06ND07NkzZ511Vvv64Ycfnj59+uSxxx7bZP8L\nLrig/QqZJHnPe96TmpqafPvb307ywtUq69atywc+8IEO+7373e9Or169ctddd3VY33vvvTNlypRt\nnv/FN1Jua2vLH/7wh/To0SNDhw7NkiVLNtn+/PPP7zDvmDFjUqlU2s9t6dKlefzxx/OBD3wgvXr1\n2uwxn3rqqdxzzz05++yz26/k2vh485vfnIcffji//e1vkyR9+vTJAw88kEceeWSbzwkA2DofiQKA\nV6ljjz12izcdfuSRR9LW1pbDDjtsk+eqqqo63KD317/+dT7+8Y/nzjvvzFNPPdVhu7+8B8sroUuX\nLhk4cGCHtYcffjhJMnny5M3uU11dndWrV6d3796dOla3bt06fEwseSHa/OXxN66/+PyTF34Hf/k7\n7NmzZw488MA8/vjjSZKVK1cmeSH6vFjXrl0zZMiQrFixosP6QQcdlC5dtv0/wSqVSj73uc/luuuu\ny69+9ats2LChfbZ+/fptsv1rX/vaDj/37ds3SdrP7dFHH01VVVVe//rXb/GYjzzySCqVSj7+8Y/n\nYx/72CbPV1VV5X//939z4IEH5rLLLssZZ5yRww8/PEcccUROPfXU/N3f/V2OPPLIbT5HAGBTgg0A\n7Iba2tpSXV2d7373u6mu3vSC2n322ad9u3HjxuXpp5/ORz7ykfarT5544omce+65aWtr2+qxtnT/\nmo1h4S+9+IqeF8+bJJ/5zGcyYsSIze63cebOePGVJtuyXtlB34T1Yp39RqiN9yJ617velSuuuCL7\n7bdfqqur8/73v3+zf59X4tw2vu4HP/jBjB8/frPbbAxZY8aMyaOPPppvfOMb+d73vpebbropTU1N\nuf7663P++edv8zEBgI4EGwDYDR166KGpVCoZNGjQZq+y2ej+++/Pww8/nH/5l3/JOeec077+/e9/\nf5NttxRm+vbtm0qlssk3J228AmVb502SXr16tX87UgkqlUoefvjhnHjiie1rf/rTn/Lb3/42b33r\nW5Ok/SbFy5cvz6BBg9q3W7duXX71q1/lb/7mb7bpWFv6/d5xxx055ZRTcsMNN3RYf/rpp9O/f//O\nnE6S/3tv/OIXv9ji73rIkCFJXrhKaFv+Hn369Mm5556bc889N2vWrMmYMWPyiU98QrABgJfBPWwA\nYDd05plnprq6Oo2NjZt9/g9/+EOS/7sa4y+v1Pjc5z63SUDo2bNnkuSZZ57psH7IIYekpqZmk/vM\nzJ49e5u/Paquri6HHnpoPv3pT+dPf/rTJs+3trZu0+vsCDfccEPWr1/f/vPs2bOzYcOGTJgwIUky\nbty4dO3adZOvxP7iF7+YZ555Jm9729u26Tg9e/bc7NeF19TUbHJ1zO23354nnniis6eSJBk1alQG\nDx6cz33uc1v8yFv//v1z0kkn5frrr8+TTz65yfMv/ntsfC9t1KNHjxx22GF5/vnnt2s+AOAFrrAB\ngFehrX28ZciQIbniiityySWX5Fe/+lXOOOOM9OrVK4899lj+/d//PVOnTs2MGTMybNiwHHroobn4\n4ouzatWq7Lvvvrnjjjs2Gw7q6upSqVTyvve9L+PHj09NTU3e8Y53ZN99983ZZ5/dHiwOPfTQfOtb\n38rvfve7bT6fqqqqfPGLX8yECRPy+te/Puedd14OOuigPPHEE7nnnnvSu3fvfOMb3+jcL+kV8uc/\n/zljx47NxIkT89BDD+W6667LmDFj2kNMv3798pGPfCSXXXZZTj311Jx++unt2x133HEdrlx6KXV1\ndZkzZ06uvPLKHHbYYTnggANy8skn521ve1suv/zynH/++Tn++ONz//33Z+7cue1XJXXWxhtWn376\n6Rk5cmTOO++8HHjggXnooYfy4IMP5jvf+U6SZNasWRkzZkyOPPLIvPvd786QIUPS0tKSRYsW5Ykn\nnmi/IfXrXve6nHTSSamrq8t+++2XxYsX59/+7d/avx4cANg+gg0AvApty5Ur//iP/5ihQ4emqakp\nl112WZIXbki7MSokL9wA+Fvf+lamTZuWa665Jt26dcuZZ56Z9773vZvcS+bMM8/MtGnT8pWvfCVz\n585NpVLJO97xjiTJ5z//+axfvz7XX3999t5777zjHe/Ipz/96RxxxBHbPPuJJ56YRYsW5fLLL8+s\nWbPy7LPPZsCAAXnDG96QqVOnbtfvZEvH2tK2f7leVVWVL3zhC5k7d24uvfTSrFu3Luecc07++Z//\nucN2l156aQ444IB84QtfyIwZM7LffvvlwgsvzJVXXrnJPWW2NNPMmTOzcuXKfOpTn8of//jHnHji\niTn55JNzySWXZM2aNfnXf/3XfPWrX01dXV2+/e1v58Mf/vBm592W833zm9+ce+65J42NjfnsZz+b\ntra2HHroobngggvatxk+fHh+9rOfpbGxMbfeemv7t4wdffTRmTlzZvt273//+/PNb34zd999d55/\n/vkccsghueqqq/LBD35ws7MAANumqrIz7q4HAPAqc+utt+b888/P4sWLt/htXAAAO8p23cNm1qxZ\nGTx4cLp3757Ro0dn8eLFW9z25JNPTnV19SaP0047bbuHBgAAANiddTrYzJ8/PxdffHEaGxuzdOnS\njBgxIuPHj9/izQC//vWv58knn2x//OIXv0hNTU0mTpz4socHANiRXIgMAOwqnQ42TU1NmTp1aiZP\nnpxhw4Zlzpw56dGjR26++ebNbt+nT58ccMAB7Y/vfe976dmzZ84666yXPTwAwI60rd9yBQDwSuvU\nPWzWrVuXHj165I477mi/WWGSTJkyJatXr87Xv/71rb7GUUcdlTe96U257rrrtm9iAAAAgN1cp74l\nqrW1NRs2bEhtbW2H9dra2ixfvnyr+//0pz/NAw88kFtuuWWrx1mwYEEGDRqU7t27d2ZEAAAAgGI9\n99xzefzxxzN+/Pj069dvi9vt1K/1vummm3LkkUemrq7uJbdbsGBBJk2atJOmAgAAANi5brvttpxz\nzjlbfL5TwaZfv36pqalJS0tLh/WWlpYMGDDgJfdds2ZN5s+fnyuuuGKrxxk0aFCSF4YfPnx4Z0aE\nTps+fXqampp29RjsAbzX2Fm819hZvNfYWbzX2Fm819gZli1blkmTJrW3jy3pVLDp2rVr6urq0tzc\n3H4Pm0qlkubm5kybNu0l9/3qV7+aP//5zy9Zjzba+DGo4cOHZ9SoUZ0ZETqtd+/e3mfsFN5r7Cze\na+ws3mvsLN5r7Czea+xMW7sFTKe/JWrGjBm58cYb8+UvfzkPPfRQLrzwwqxZsyZTpkxJkkyePDmX\nXHLJJvvddNNNOeOMM9K3b9/OHhIAAABgj9Lpe9hMnDgxra2tmTlzZlpaWjJy5MgsWLAg/fv3T5Ks\nWrUqXbp0fNlf/vKX+dGPfpS77777lZkaAAAAYDe2XTcdbmhoSENDw2afW7hw4SZrhx9+eDZs2LA9\nhwIAAADY43T6I1Gwu6mvr9/VI7CH8F5jZ/FeY2fxXmNn8V5jZ/FeoyRVlUqlsquH+EtLlixJXV1d\n7rvvPjd8AgAAAHYb29o8XGEDAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAA\nCiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAw2xVsZs2alcGDB6d79+4ZPXp0Fi9e/JLbr169\nOu9973vzmte8Jt26dcuwYcPy3e9+d7sGBgAAANjddensDvPnz8/FF1+cG264Iccdd1yampoyfvz4\n/PKXv0y/fv022X7dunUZN25cBgwYkK997Wt5zWtekxUrVqRPnz6vyAkAAAAA7G46HWyampoyderU\nTJ48OUkyZ86c3HXXXbn55pvzD//wD5tsf9NNN+Xpp5/Oj3/849TU1CRJDj744Jc5NgAAAMDuq1Mf\niVq3bl3uu+++jB07tn2tqqoq48aNy6JFiza7z5133pk3vvGNaWhoyIABA3LkkUfm6quvTltb28ub\nHAAAAGA31akrbFpbW7Nhw4bU1tZ2WK+trc3y5cs3u89jjz2WhQsXZtKkSfnOd76TRx55JO95z3uy\nfv36fPzjH9/+yQEAAAB2U53+SFRntbW1pba2NjfccEOqqqpy9NFHZ9WqVfn0pz+91WAzffr09O7d\nu8NafX196uvrd+TI7ObmzXvhkSRr1yYrViSHHJJ06/bCWn39Cw8AAAB4OebNm5d5G/8H9P9bvXr1\nNu3bqWDTr1+/1NTUpKWlpcN6S0tLBgwYsNl9DjzwwOy1116pqqpqXxs+fHiefPLJrF+/Pl26bHmE\npqamjBo1qjMjwla9OMgsWZLU1b0QcLzVAAAAeCVt7qKTJUuWpK6ubqv7duoeNl27dk1dXV2am5vb\n1yqVSpqbm3P88cdvdp83velNeeSRRzqsLV++PAceeOBLxhoAAACAPVWngk2SzJgxIzfeeGO+/OUv\n56GHHsqFF16YNWvWZMqUKUmSyZMn55JLLmnf/j3veU/+8Ic/ZNq0aXn44Ydz11135eqrr85FF130\nip0EAAAAwO6k05e4TJw4Ma2trZk5c2ZaWloycuTILFiwIP3790+SrFq1qsOVMwMHDsyCBQsyffr0\njBgxIgcddFCmT5++2a8ABwAAAGA7bzrc0NCQhoaGzT63cOHCTdbe8IY35Ec/+tH2HAoAAABgj9Pp\nj0QBAAAAsGMJNgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABA\nYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwXXb1AJRj\n5cqVaW1t3dVj7FTLlnVPMjzLli1L8tyuHmen6tevXw4++OBdPQYAAACbIdiQ5IVYM3zo0KxZu3ZX\nj7KTHZ1kSSZNOifJ0l09zE7Vo1u3LFu+XLQBAAAokGBDkqS1tTVr1q7NbUmG7+phdqJlSSYle+Z5\nr12b1tZWwQYAAKBAgg0dDE8yalcPsQvsqecNAABAmdx0GAAAAKAwgg0AAABAYQQbAAAAgMIINgAA\nAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAA\nAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAA\nAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAA\nAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUJjtCjazZs3K4MGD071794wePTqLFy/e4ra3\n3nprqqurU1NTk+rq6lRXV6dHjx7bPTAAAADA7q7TwWb+/Pm5+OKL09jYmKVLl2bEiBEZP358Wltb\nt7hP79698+STT7Y/VqxY8bKGBgAAANiddTrYNDU1ZerUqZk8eXKGDRuWOXPmpEePHrn55pu3uE9V\nVVX69++fAw44IAcccED69+//soYGAAAA2J11KtisW7cu9913X8aOHdu+VlVVlXHjxmXRokVb3O/Z\nZ5/NoEGDcvDBB+eMM87Igw8+uP0TAwAAAOzmOhVsWltbs2HDhtTW1nZYr62tzZNPPrnZfYYOHZqb\nb7453/zmNzN37ty0tbXl+OOPz29+85vtnxoAAABgN9ZlRx9g9OjRGT16dPvPb3zjGzN8+PBcf/31\naWxs3NGHBwAAAHjV6VSw6devX2pqatLS0tJhvaWlJQMGDNi2A3bpkqOPPjqPPPLIVredPn16evfu\n3WGtvr4+9fX12z40AAAAwC4wb968zJs3r8Pa6tWrt2nfTgWbrl27pq6uLs3NzTn99NOTJJVKJc3N\nzZk2bdo2vUZbW1vuv//+vPWtb93qtk1NTRk1alRnRgQAAAAowuYuOlmyZEnq6uq2um+nPxI1Y8aM\nTJkyJXV1dTnuuOPS1NSUNWvWZMqUKUmSyZMnZ+DAgbnqqquSJJdffnlGjx6dww47LE8//XT+6Z/+\nKStXrsy73vWuzh4aAAAAYI/Q6WAzceLEtLa2ZubMmWlpacnIkSOzYMGC9q/qXrVqVbp0+b+Xfeqp\np3LBBRfkySefTN++fVNXV5dFixZl2LBhr9xZAAAAAOxGtuumww0NDWloaNjscwsXLuzw82c/+9l8\n9rOf3Z7DAAAAAOyROvW13gAAAADseIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAU\nRrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAU\nRrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAU\nRrABAAAAKIxgAwAAAFCYLrt6ANjZ5uWdmZf6JMna7J3DszwfztXplueTJPWZl/p8ZVeOCAAAwB5O\nsGGPU5+vCDIAAAAUzUeiAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEE\nGwAAAICiRiN9AAAgAElEQVTCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAUZruCzaxZszJ48OB07949o0eP\nzuLFi7dpv6985Suprq7OmWeeuT2HBQAAANgjdDrYzJ8/PxdffHEaGxuzdOnSjBgxIuPHj09ra+tL\n7vf444/nQx/6UE444YTtHhYAAABgT9DpYNPU1JSpU6dm8uTJGTZsWObMmZMePXrk5ptv3uI+bW1t\nmTRpUi677LIMHjz4ZQ0MAAAAsLvrVLBZt25d7rvvvowdO7Z9raqqKuPGjcuiRYu2uF9jY2Nqa2tz\n3nnnbf+kAAAAAHuILp3ZuLW1NRs2bEhtbW2H9dra2ixfvnyz+9x777255ZZb8vOf/3z7pwQAAADY\ng+zQb4l69tlnM3ny5Nx4443p27fvjjwUAAAAwG6jU1fY9OvXLzU1NWlpaemw3tLSkgEDBmyy/aOP\nPpoVK1bktNNOS6VSSfLC/WySZK+99sry5ctf8p4206dPT+/evTus1dfXp76+vjNjAwAAAOx08+bN\ny7x58zqsrV69epv27VSw6dq1a+rq6tLc3JzTTz89SVKpVNLc3Jxp06Ztsv3w4cNz//33d1j76Ec/\nmmeffTbXXnttXvva177k8ZqamjJq1KjOjAgAAABQhM1ddLJkyZLU1dVtdd9OBZskmTFjRqZMmZK6\nurocd9xxaWpqypo1azJlypQkyeTJkzNw4MBcddVV2WuvvfK6172uw/59+vRJVVVVhg8f3tlDAwAA\nAOwROh1sJk6cmNbW1sycOTMtLS0ZOXJkFixYkP79+ydJVq1alS5dOv2yAAAAAPx/21VWGhoa0tDQ\nsNnnFi5c+JL73nLLLdtzSAAAAIA9xg79ligAAAAAOk+wAQAAACiMYAMAAABQGMEGAAAAoDCCDQAA\nAEBhBBsAAACAwgg2AAAAAIURbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQmC67egCA3dW8eS88\nkmTt2mTFiuSQQ5Ju3V5Yq69/4QEAAPCXBBuAHeTFQWbJkqSu7oWAM2rUrp0LAAAon49EAQAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRG\nsAEAAAAojGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMJ02dUDAHuelStXprW1dVePsVMtW9Y9\nyfAsW7YsyXO7epydql+/fjn44IN39RgAAPCqItgAO9XKlSszfOjQrFm7dlePspMdnWRJJk06J8nS\nXT3MTtWjW7csW75ctAEAgE4QbICdqrW1NWvWrs1tSYbv6mF2omVJJiV75nmvXZvW1lbBBgAAOkGw\nAXaJ4UlG7eohdoE99bwBAIDOcdNhAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGAD\nAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAozHYF\nm1mzZmXw4MHp3r17Ro8encWLF29x269//es59thj07dv3+yzzz45+uijc9ttt233wAAAAAC7u04H\nm/nz5+fiiy9OY2Njli5dmhEjRmT8+PFpbW3d7Pb7779/Pvaxj+XHP/5x7r///px33nk577zzcvfd\nd7/s4QEAAAB2R50ONk1NTZk6dWomT56cYcOGZc6cOenRo0duvvnmzW5/wgkn5O1vf3uGDh2awYMH\nZ9q0aTnqqKNy7733vuzhAQAAAHZHnQo269aty3333ZexY8e2r1VVVWXcuHFZtGjRNr1Gc3NzfvnL\nX+bEE0/s3KQAAAAAe4gundm4tbU1GzZsSG1tbYf12traLF++fIv7PfPMMznooIPy/PPPp0uXLpk9\ne3ZOOeWU7ZsYAAAAYDfXqWCzvXr16pWf//znefbZZ9Pc3Jzp06dnyJAhOeGEE15yv+nTp6d3794d\n1urr61NfX78jxwUAAAB42ebNm5d58+Z1WFu9evU27dupYNOvX7/U1NSkpaWlw3pLS0sGDBiwxf2q\nqqoyZMiQJMlRRx2VBx98MFdfffVWg01TU1NGjRrVmREBAAAAirC5i06WLFmSurq6re7bqXvYdO3a\nNXV1dWlubm5fq1QqaW5uzvHHH7/Nr9PW1pbnn3++M4cGAAAA2GN0+iNRM2bMyJQpU1JXV5fjjjsu\nTU1NWbNmTaZMmZIkmTx5cgYOHJirrroqSXLNNdfkmGOOyaGHHprnn38+d911V2677bbMmTPnFT0R\nAAAAgN1Fp4PNxIkT09rampkzZ6alpSUjR47MggUL0r9//yTJqlWr0qXL/73sn/70p7z3ve/NqlWr\n0r179wwbNixz587NWWed9cqdBQAAAMBuZLtuOtzQ0JCGhobNPrdw4cIOP19++eW5/PLLt+cwAAAA\nAHukTt3DBgAAAIAdT7ABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFs\nAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFs\nAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFs\nAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFs\nAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFs\nAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYAAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhemy\nqwcA2F3NyzszL/VJkrXZO4dneT6cq9MtzydJ6jMv9fnKrhwRAAAolGADsIPU5yuCDAAAsF18JAoA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAAQGEEGwAAAIDCCDYAAAAAhRFsAAAAAAoj2AAAAAAURrABAAAAKIxgAwAAAFAYwQYA\nAACgMIINAAAA/L/27j9W67r+//jj8MM4YJ4NoYNDIHIJlPMAx+HRnJWQZ/6h1uZYx9iR/mg2Mqcw\n7bN0KLVMHOtMJ8qaUVvUAW0r/KPCiaVWWAjIcpxR9kNC5dTVlkvhGMPz/YNP1Pl+sDwK5/3yum63\n7T31xfu63s/r7KXz3Hdd7wsKI9gAAAAAFOYtBZu1a9dm5syZaW5uTkdHR7Zv3/6G5z7wwAO5+OKL\nM3HixEycODEf+9jH/uP5AAAAAI1u2MFm06ZNWbFiRVatWpVdu3alra0tnZ2dqdVqxz3/8ccfz9VX\nX52f/vSneeqppzJt2rRceumleemll9728AAAAAD1aNjBpqenJ9dee226u7sze/bsrFu3LuPHj8/6\n9euPe/63v/3tfPazn825556bs88+Ow888EBef/31bN269W0PDwAAAFCPhhVsDh8+nB07dmThwoXH\n1pqamrJo0aJs27btTT3Hq6++msOHD2fixInDmxQAAACgQQwr2NRqtRw5ciStra1D1ltbW3PgwIE3\n9Rxf+MIXMnXq1CxatGg4lwYAAABoGGNG8mJ33nlnHnzwwTz++OM55ZRT/uv5N954Y1paWoasdXV1\npaur62SNCAAAAHBC9Pb2pre3d8jayy+//KYeO6xgM2nSpIwePTr9/f1D1vv7+zNlypT/+Ng1a9bk\nrrvuytatW/PBD37wTV2vp6cn8+fPH86IAAAAAEU43ptOdu7cmfb29v/62GF9JGrs2LFpb28fcsPg\nwcHBbN26NRdeeOEbPu6uu+7KV77ylWzZsiXz5s0bziUBAAAAGs6wPxK1fPnyLF26NO3t7VmwYEF6\nenpy8ODBLF26NEnS3d2dM888M3fccUeSZPXq1bntttvS29ub6dOnH3t3zqmnnpoJEyacuFcCAAAA\nUCeGHWwWL16cWq2WlStXpr+/P3Pnzs2WLVsyefLkJMn+/fszZsy/nnbdunU5fPhwrrrqqiHPc9tt\nt2XlypVvc3wAAACA+vOWbjq8bNmyLFu27Lh/9thjjw355z/84Q9v5RIAAAAADWtY97ABAAAA4OQT\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFOYtBZu1a9dm5syZaW5uTkdHR7Zv3/6G5+7ZsydXXXVVZs6cmVGjRuWee+55\ny8MCAAAANIJhB5tNmzZlxYoVWbVqVXbt2pW2trZ0dnamVqsd9/yDBw/mrLPOyurVq3PGGWe87YEB\nAAAA6t2wg01PT0+uvfbadHd3Z/bs2Vm3bl3Gjx+f9evXH/f88847L6tXr87ixYtzyimnvO2BAQAA\nAOrdsILN4cOHs2PHjixcuPDYWlNTUxYtWpRt27ad8OEAAAAAGtGwgk2tVsuRI0fS2to6ZL21tTUH\nDhw4oYMBAAAANCrfEgUAAABQmDHDOXnSpEkZPXp0+vv7h6z39/dnypQpJ3SwJLnxxhvT0tIyZK2r\nqytdXV0n/FoAAAAAJ1Jvb296e3uHrL388stv6rHDCjZjx45Ne3t7tm7dmiuuuCJJMjg4mK1bt+b6\n668fzlO9KT09PZk/f/4Jf14AAACAk+14bzrZuXNn2tvb/+tjhxVskmT58uVZunRp2tvbs2DBgvT0\n9OTgwYNZunRpkqS7uztnnnlm7rjjjiRHb1S8Z8+eDA4O5h//+EdeeOGF7N69O6eeemrOOuus4V4e\nAAAAoO4NO9gsXrw4tVotK1euTH9/f+bOnZstW7Zk8uTJSZL9+/dnzJh/Pe2LL76YefPmpampKUmy\nZs2arFmzJh/+8Ifz2GOPnaCXAQAAAFA/hh1skmTZsmVZtmzZcf/s/48wM2bMyOuvv/5WLgMAAADQ\nkHxLFAAAAEBhBBsAAACAwgg2AAAAAIURbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAA\noDCCDQAAAEBhBBsAAACAwgg2AAAAAIURbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAA\noDCCDQAAAEBhBBsAAACAwgg2AAAAAIURbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAA\noDCCDQAAAEBhBBsAAACAwgg2AAAAAIUZU/UAAMDb09t79EiSgYHk+eeTGTOSceOOrnV1HT0AAHjn\nEGwA4B3u34PMzp1Je/vRgDN/frVzAQDw1gk2AAC8Kd7NBQAjR7ABAOBN8W4uABg5bjoMAAAAUBjB\nBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjB\nBgAAAKAwgg0AAABAYQQbAAAAgMIINgAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAojGADAAAAUBjB\nBgAAAKAwgg0AAABAYQQbAAAAgMKMqXoAADhZ9u3bl1qtVvUYI6qvrznJnPT19SU5VPU4I2rSpEmZ\nPn16Jde21+w1ADjRBBsA6tK+ffsyZ9asHBwYqHqUETYvyc4sWfKpJLuqHmZEjR83Ln179474L9L2\nmr0GACeDYANAXarVajk4MJANSeZUPcwI6kuyJGnM1z0wkFqtNuK/RNtrDfi6K9prADQWwQaAujYn\nyfyqh6hAo77uKjXqz7xRXzcAnGxuOgwAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2AAAAAIUR\nbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQmDFVDwAAAPDvenuPHkkyMJA8/3wyY0YybtzRta6u\nowdAPRNsAACAovx7kNm5M2lvPxpw5s+vdi6AkeQjUQAAAACFEWwAAAAACiPYAAAAABRGsAEAAAAo\njGADAAAAUBjBBgAAAKAwgg0AAABAYQQbAAAAgMKMqXoAAADgzdm3b19qtVrVY4yovr7mJHPS19eX\n5FDV44yoSZMmZfr06VWPAVREsAEAgHeAffv2Zc6sWTk4MFD1KCNsXpKdWbLkU0l2VT3MiBo/blz6\n9u4VbaBBCTYA8A7Xm0+mN11JkoG8K2dnb/4nX824vJYk6UpvurKxyhGpE/ZatWq1Wg4ODGRDkjlV\nDzOC+pIsSRrzdQ8MpFarCTbQoAQbAHiH68pGvyQzIuy1MsxJMr/qISrQqK8baFyCDQAAAA2pt/fo\nkSQDA8nzzyczZiTjxh1d6+o6ekAVBBsAAAAa0r8HmZ07k/b2owFnvrdzUQBf6w0AAABQGMEGAAAA\noDA+EgUAAMAQ+/btS61Wq3qMEdXX15xkTvr6+pIcqnqcETVp0iTfRlYgwQYAAIBj9u3blzmzZuXg\nwEDVo4yweUl2ZsmSTyXZVfUwI2r8uHHp27tXtCmMYAMAAMAxtVotBwcGsiFHv069UfQlWZI05use\nGEitVhNsCiPYAAAARenNJ9Obo1/dM5B35ezszf/kqxmX15IkXelNVzZWOWJDmJOkEb8sqVFfN+UR\nbAAAgKJ0ZaMgAzQ83xIFAAAAUBjBBgAAAKAwPhIFAABAQ3K/JEom2AAAANCQ3C+JkvlIFAAAAEBh\nBBsAAACAwgg2AAAAAIURbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQ2vt+oB\naBj2GiPFXmOk2GuMFHuNkWKvUZK3FGzWrl2bmTNnprm5OR0dHdm+fft/PP+hhx7KnDlz0tzcnLa2\ntvzoRz96S8PCyeA/yowUe42RYq8xUuw1Roq9xkix1yjJsIPNpk2bsmLFiqxatSq7du1KW1tbOjs7\nU6vVjnv+L37xi1x99dX5zGc+k2eeeSZXXnllPv7xj2fPnj1ve3gAAACAejTsYNPT05Nrr7023d3d\nmT17dtatW5fx48dn/fr1xz3/nnvuyWWXXZbly5dn1qxZ+dKXvpT58+fn3nvvfdvDAwAAANSjYQWb\nw4cPZ8eOHVm4cOGxtaampixatCjbtm077mO2bduWRYsWDVnr7Ox8w/MBAAAAGt2Y4Zxcq9Vy5MiR\ntLa2DllvbW3N3r17j/uYAwcOHPf8AwcOvOF1Dh06lCTp6+sbzni8Df/8Wf8wSaP91Pcn+U7VQ4yw\nP/zvX6v4d8xeayz2WjXstZFlrzUWe60a9trIstcaS5V7rVH982f9z/bxhgaH4cUXXxxsamoafOqp\np4as33zzzYMdHR3Hfcwpp5wyuHHjxiFr99133+CUKVPe8DobNmwYTOJwOBwOh8PhcDgcDofDUZfH\nhg0b/mODGdY7bCZNmpTRo0env79/yHp/f3+mTJly3MdMmTJlWOcnRz8ytWHDhrz3ve9Nc3PzcEYE\nAAAAKNahQ4fyxz/+MZ2dnf/xvKbBwcHB4TxxR0dHzj///Nx9991JksHBwUyfPj3XX399brrppv9z\n/ic/+ckcOnQomzdvPrb2oQ99KG1tbbnvvvuGc2kAAACAhjD69ttvv304DzjttNOycuXKTJs2Le96\n17ty6623Zvfu3XnggQcyYcKEdHd3Z/v27cduTDx16tTceuutmTBhQiZOnJh77703Dz30UL7xjW9k\n8uTJJ+M1AQAAALyjDesjUUmyePHi1Gq1rFy5Mv39/Zk7d262bNlyLL7s378/Y8b862kvuOCCfPe7\n380tt9ySW265Je9///uzefPmfOADHzhxrwIAAACgjgz7I1EAAAAAnFyjqh4AAAAAgKEEGxrOunXr\n0tbWlpaWlrS0tOTCCy/Mj3/846rHog6tWrUqo0aNGnL4OCgn09q1azNz5sw0Nzeno6Mj27dvr3ok\n6tydd96ZUaNGZfny5VWPQh168sknc8UVV2Tq1KkZNWpUHn744apHog599atfzYIFC3LaaaeltbU1\nn/jEJ/Kb3/ym6rEgiWBDA5o2bVpWr16dnTt3ZseOHbnkkkty5ZVXpq+vr+rRqEPnnHNO+vv7c+DA\ngRw4cCA/+9nPqh6JOrVp06asWLEiq1atyq5du9LW1pbOzs7UarWqR6NObd++PV//+tfT1tZW9SjU\nqVdffTVz587Nfffdl6ampqrHoU49+eST+fznP59f/vKXefTRR3P48OFceumlOXToUNWjgXvYQJKc\nfvrpWbNmTT796U9XPQp1ZNWqVdm8eXN27txZ9Sg0gI6Ojpx//vm5++67kySDg4OZNm1arr/++tx8\n880VT0e9eeWVV9Le3p77778/X/7ylzNv3rx87Wtfq3os6tioUaPygx/8IFdccUXVo1DnarVa3vOe\n9+SJJ57IRRddVPU4NDjvsKGhvf7669m4cWMOHjyYCy64oOpxqEO//e1vM3Xq1Jx11llZsmRJ/vSn\nP1U9EnXo8OHD2bFjRxYuXHhsrampKYsWLcq2bdsqnIx69bnPfS6XX355LrnkkqpHATih/va3v6Wp\nqSkTJ06sehQY/td6Qz149tlnc8EFF2RgYCDvfve78/3vfz+zZ8+ueizqTEdHR771rW9l1qxZeeml\nl3L77bfn4osvzrPPPpsJEyZUPR51pFar5ciRI2ltbR2y3tramr1791Y0FfVq48aNeeaZZ/L0009X\nPQrACTU4OJgbbrghF110kfsOUgTBhoY0e/bs7N69Oy+//HK+973vpbu7O0888YRowwnV2dl57O/P\nOeecLFiwIDNmzMiDDz7o43fAO9L+/ftzww035NFHH83YsWOrHgfghFq2bFn27NmTn//851WPAkkE\nGxrUmDFj8r73vS9JMm/evPzqV7/K3Xffnfvvv7/iyahnLS0tOfvss/Pcc89VPQp1ZtKkSRk9enT6\n+/uHrPf392fKlCkVTUU92rFjR/7yl79k/vz5+edtEI8cOZInnngi9957b1577TU3hwXeka677rr8\n8Ic/zJNPPpkzzjij6nEgiXvYQJKj97J57bXXqh6DOvfKK6/kd7/7nf8J4IQbO3Zs2tvbs3Xr1mNr\ng4OD2bp1ay688MIKJ6PeLFq0KL/+9a/zzDPPZPfu3dm9e3fOO++8LFmyJLt37xZrgHek6667Lps3\nb85PfvKTTJ8+vepx4BjvsKHhfPGLX8xll12W6dOn5+9//3u+853v5PHHH88jjzxS9WjUmZtuuimX\nX5I0R8QAAAGNSURBVH55ZsyYkRdeeCG33XZbxowZk66urqpHow4tX748S5cuTXt7exYsWJCenp4c\nPHgwS5curXo06siECRP+z30dJkyYkNNPPz1z5sypaCrq1auvvprnnnvu2Lu5fv/732f37t2ZOHFi\npk2bVvF01Itly5alt7c3Dz/8cCZMmHDs3aotLS0ZN25cxdPR6AQbGs6f//znXHPNNXnppZfS0tKS\nc889N4888ohvuuCE279/f66++ur89a9/zeTJk3PRRRflqaeeyumnn171aNShxYsXp1arZeXKlenv\n78/cuXOzZcuWTJ48uerRqHPeVcPJ8vTTT+ejH/1ompqa0tTUlBUrViRJrrnmmqxfv77i6agX69at\nS1NTUz7ykY8MWf/mN7+Z7u7uaoaC/9U0+M9kDQAAAEAR3MMGAAAAoDCCDQAAAEBhBBsAAACAwgg2\nAAAAAIURbAAAAAAKI9gAAAAAFEawAQAAACiMYAMAAABQGMEGAAAAoDCCDQAAAEBhBBsAAACAwgg2\nAAAAAIX5f0WcCv6Eux6KAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa0e0f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "# Build a classification task using 3 informative features\n",
    "\n",
    "# Build a forest and compute the feature importances\n",
    "forest = ExtraTreesClassifier(n_estimators=250,\n",
    "                              random_state=0)\n",
    "\n",
    "forest.fit(X, y)\n",
    "importances = forest.feature_importances_\n",
    "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
    "             axis=0)\n",
    "indices = np.argsort(importances)[::-1]\n",
    "\n",
    "# Print the feature ranking\n",
    "print(\"Feature ranking:\")\n",
    "\n",
    "for f in range(X.shape[1]):\n",
    "    print(\"%d. feature %d - %s (%f) \" % (f + 1, indices[f], features[indices[f]], importances[indices[f]]))\n",
    "\n",
    "# Plot the feature importances of the forest\n",
    "plt.figure(num=None, figsize=(14, 10), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(X.shape[1]), importances[indices],\n",
    "       color=\"r\", yerr=std[indices], align=\"center\")\n",
    "plt.xticks(range(X.shape[1]), indices)\n",
    "plt.xlim([-1, X.shape[1]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. feature 3 - Persons (0.536901)\n",
      "2. feature 5 - Safety (0.318936)\n",
      "3. feature 0 - Buying (0.048373)\n",
      "4. feature 4 - Lug-Boot (0.047934)\n",
      "5. feature 1 - Maintenance (0.027871)\n"
     ]
    }
   ],
   "source": [
    "for f in range(5):\n",
    "    print(\"%d. feature %d - %s (%f)\" % (f + 1, indices[f], features[indices[f]] ,importances[indices[f]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "best_features = []\n",
    "for i in indices[:5]:\n",
    "    best_features.append(features[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the top 5 feature importances of the forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApsAAAINCAYAAABxrbQ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3XtclGX+//H3DKAcRNRQPCtaKmkemHK11iMk6qq1pRTl\nEroaRq2burXbySRNbWuX1s1DJ8vSH7mdc6u1wnSzrEyUXCU8JajF1FjSFqII1+8Pv842AcqgV4C+\nno/H/Siuua77+szcDrznPsztMMYYAQAAABY4a7sAAAAAnL0ImwAAALCGsAkAAABrCJsAAACwhrAJ\nAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAUA35+flyOp165plnarsUAKhXCJsAKli2\nbJmcTmely5133mlt3jfffFPp6enW1n+6HA5HbZdwWhYvXqxly5bVdhkAzjGBtV0AgLrJ4XBo9uzZ\n6tixo097jx49rM35xhtvaNGiRbr33nutzVFTHTp00OHDhxUUFFTbpdTYokWL1Lx5c91www21XQqA\ncwhhE0CVhg8frtjY2J9tPmOMlfUWFxcrNDT0tNfToEGDM1DNz+/w4cMKCQmp7TIAnKM4jA7gtCxf\nvlwXX3yxQkNDdd555ykpKUn79+/36bN+/XolJiaqQ4cOCg4OVvv27TV9+nSVlJR4+0yYMEGLFi2S\nJO8h+4CAAEnS2rVr5XQ69e9//9tnvZWdR5mSkqLw8HDt2bNHI0eOVOPGjTV+/Hjv4x999JGGDx+u\nJk2aKCwsTIMHD9YHH3xwyud5srn27dunUaNGKTw8XG3btvU+j61btyouLk6NGjVSx44dlZmZ6bPO\nE6crvPfee0pNTVVkZKQiIiJ0ww036NChQxVqWLRokXr06KHg4GC1adNGt9xyi4qKinz6DB48WD17\n9lR2drYGDhyosLAw3XnnnYqOjta2bdu8r6XT6dTQoUMlSd9++63+8Ic/qGfPngoPD1dERIRGjhyp\nTz/91Gfd69atk9Pp1PPPP6/7779f7dq1U0hIiOLj47V79+4K9X700UcaOXKkmjVrpkaNGqlXr15a\nsGCBT5+8vDyNHTtW5513nkJCQnTJJZdo1apVPn2OHTum9PR0denSRSEhIYqMjNSAAQOUlZV1qs0G\noA5gzyaAKhUVFengwYM+beedd573/++//37NnDlT1157rSZPnqyvv/5aCxYs0KBBg7R582Y1btxY\nkvT888/r8OHDSktL03nnnaePP/5Yf//733XgwAGtXLlSkjRlyhR98cUXeuedd7RixQqfvZwOh6Pa\n50s6HA4dO3ZMCQkJGjBggP7yl79492quWbNGI0eO1MUXX6xZs2bJ6XTqqaee0tChQ7V+/XpdfPHF\nfr0+DodD5eXlGjFihAYNGqQHH3xQK1as0O9+9zuFhYXprrvu0vjx43X11VdryZIluuGGG3TppZeq\nQ4cOPuu55ZZb1LRpU6WnpysvL0+LFi1SQUGB3n33XW+fWbNm6b777tOwYcOUlpbm7ffJJ5/o/fff\n9wZzh8Mhj8ejkSNH6tprr1VycrKioqI0ZMgQ3XLLLQoPD9fdd98tY4yioqIkSXv27NFrr72mcePG\nKTo6Wm63W48++qgGDx6s7du3q2XLlj71zp8/XwEBAbrttttUVFSkBx54QOPHj9eGDRu8fd5++22N\nHj1arVu31q233qqWLVsqNzdXr7/+uqZOnSpJ2rZtm375y1+qbdu2uuOOOxQWFqZ//OMfuvLKK/XS\nSy/piiuukCTde++9mj9/vm688UZdcskl+u677/TJJ58oOztbcXFxfm0zALXAAMBPPP3008bhcFRY\nnE6nt09+fr4JDAw08+fP9xm7bds2ExQUZObNm+dtKykpqTDH/PnzTUBAgNm3b5+37ZZbbvGZ44S1\na9cap9Np1q1b59O+d+9e43A4zLJly7xtKSkpxul0mrvuuqvCerp06WJGjhzp01ZSUmI6depkEhIS\nqno5TjnXAw884G07dOiQCQ0NNQEBAeb555/3tufl5RmHw2HS09O9bSde5759+5pjx4552x988EHj\ndDrNqlWrjDHGfP3116Zhw4ZmxIgRPjUtXLjQOJ1O8/TTT3vbBg8ebJxOp3n88ccrPIcePXqYIUOG\nVGg/evRohbb8/HwTHBxs5syZ421bu3atcTgcpnv37j71LliwwDidTrNt2zZjjDFlZWUmOjradOrU\nyXz33XcV1n1CXFyc6d27tyktLfVpv+yyy0zXrl29P/fu3duMHj26yvUAqNs4jA6gUg6HQ4sXL9Y7\n77zjXd5++23v4y+++KKMMRo3bpwOHjzoXVq0aKELLrjAZ69cw4YNvf9fXFysgwcPqn///iovL9fm\nzZut1D9lyhSfn7ds2aKdO3cqKSnJp97//ve/iouLq3CI3h+//e1vvf8fERGhrl27KiwsTGPHjvW2\nd+nSRU2aNNGePXsqjL/xxhu9eyYl6aabblJAQIDeeOMNScf3EpaWlurWW2/1GTd58mSFh4fr9ddf\n92lv2LChUlJSql3/jy96Ki8v1zfffKPQ0FB17dpV2dnZFfpPnDjRp94BAwbIGON9bps3b9bevXt1\n6623Kjw8vNI5v/32W7377rsaN26cdw/6iWXYsGHauXOnvvzyS0lSkyZNtG3bNu3atavazwlA3cFh\ndABVuuSSS6q8QGjXrl0qLy/X+eefX+Exh8PhczHNvn37dM8992jVqlX69ttvffr99JzDMyEwMFBt\n27b1adu5c6ckKTk5udIxTqdTRUVFioiI8Guu4OBgn1MLpOOB86fzn2j/8fOXjr8GP30Nw8LC1KpV\nK+3du1eSVFBQIOl4YP2xoKAgderUSfn5+T7tbdq0UWBg9X+9G2P08MMPa/Hixfr8889VVlbmrS0y\nMrJC/3bt2vn83LRpU0nyPrfdu3fL4XCoe/fuVc65a9cuGWN0zz336O67767wuMPh0FdffaVWrVrp\nvvvu05VXXqkuXbqoR48eGj58uH7zm9/ooosuqvZzBFB7CJsAaqS8vFxOp1P/+te/5HRWPEjSqFEj\nb7/4+HgdOnRId9xxh3ev34EDB3TDDTeovLz8lHNVdb7miVD0Uz/ek/rjeiXpL3/5i3r16lXpuBM1\n++PHe/iq024sXXH/Y/5eeX7i3NtJkyZpzpw5atasmZxOp37/+99Xun3OxHM7sd4//OEPSkhIqLTP\niRA+YMAA7d69W6+++qreeustPfnkk8rIyNCjjz6qiRMnVntOALWDsAmgRjp37ixjjDp27Fjp3s0T\ntm7dqp07d+rZZ5/V9ddf721/5513KvStKlQ2bdpUxpgKV2if2PNX3XolKTw83HsVdl1gjNHOnTs1\naNAgb9sPP/ygL7/8Ur/61a8kyXtBUV5ens/3npaWlurzzz/X5ZdfXq25qnp9X3zxRQ0dOlSPPfaY\nT/uhQ4fUvHlzf56OpP/92/jPf/5T5WvdqVMnScf3zlZnezRp0kQ33HCDbrjhBhUXF2vAgAGaNWsW\nYROoBzhnE0CNXHXVVXI6nVXe8eebb76R9L+9YD/dQ/bwww9XCD9hYWGSpO+++86nvUOHDgoICKhw\nXuWiRYuqfZW6y+VS586d9dBDD+mHH36o8LjH46nWemx47LHHdOzYMe/PixYtUllZmUaOHClJio+P\nV1BQUIWvDXriiSf03XffadSoUdWaJywsrNKvVAoICKiwV/L555/XgQMH/H0qkqTY2FhFR0fr4Ycf\nrvI0iebNm2vw4MF69NFHVVhYWOHxH2+PE/+WTggNDdX555+vI0eO1Kg+AD8v9mwCqNSpDol26tRJ\nc+bM0Z133qnPP/9cV155pff7LV955RWlpqZq+vTp6tatmzp37qwZM2Zo//79aty4sV588cVKQ4/L\n5ZIxRr/73e+UkJCggIAAXXPNNWrcuLHGjRvnDVudO3fWP//5T3399dfVfj4Oh0NPPPGERo4cqe7d\nu2vChAlq06aNDhw4oHfffVcRERF69dVX/XuRzpCjR48qLi5OiYmJ+uyzz7R48WINGDDAGyIjIyN1\nxx136L777tPw4cM1ZswYb7++ffv67DE+GZfLpSVLluj+++/X+eefrxYtWmjIkCEaNWqUZs+erYkT\nJ+rSSy/V1q1btWLFCu/eYH+duLhszJgx6t27tyZMmKBWrVrps88+0/bt2/Xmm29KkhYuXKgBAwbo\noosu0uTJk9WpUye53W5t2LBBBw4c8F48duGFF2rw4MFyuVxq1qyZNm7cqBdeeMH7FUoA6rhauQYe\nQJ329NNPG6fTaTZt2nTKvi+//LIZOHCgCQ8PN+Hh4ebCCy80U6dONTt37vT2+eyzz8ywYcNM48aN\nTYsWLcyUKVPM1q1bjdPp9PkqobKyMvP73//eREVFmYCAAJ+vQfJ4PGbcuHGmUaNG5rzzzjNpaWlm\n+/btFdaRkpJiGjduXGW9OTk5ZuzYsaZ58+YmJCTEREdHm2uvvda8++67J32ee/furfZcgwcPNj17\n9qzQHh0dbcaMGeP9+cTr/N5775kpU6aY8847zzRu3NgkJyebb7/9tsL4RYsWmQsvvNA0bNjQtGrV\nytxyyy2mqKioWnMbY4zb7TajR482ERERxul0er8G6ciRI+a2224zbdq0MWFhYWbgwIHmo48+MkOG\nDDFDhw71jj/xFVQvvvjiKV8bY4z54IMPTEJCgomIiDDh4eGmd+/eZtGiRT59Pv/8c5OSkmJat25t\nGjZsaNq1a2fGjBljXnrpJW+fuXPnmn79+plmzZqZsLAwc+GFF5r58+f7fP0SgLrLYczPcLY6AKCC\nZcuWaeLEidq4cePPeltQAPg51eiczYULFyo6OlohISHq16+fNm7cWGXfIUOGeG+N9uNl9OjRNS4a\nAAAA9YPfYXPlypWaMWOG0tPTtXnzZvXq1UsJCQlVnlz/8ssvq7Cw0Lv85z//UUBAgBITE0+7eACo\n7zi4BOBs53fYzMjIUGpqqpKTk9WtWzctWbJEoaGhWrp0aaX9mzRpohYtWniXt956q8KdNQDgXFXd\nq+kBoL7y65zN0tJShYaG6sUXX9SYMWO87SkpKSoqKtLLL798ynX07NlTl112mRYvXlyzigEAAFBv\n+PXVRx6PR2VlZYqKivJpj4qKUl5e3inHf/zxx9q2bZueeuqpU86zevVqdezY0e87YQAAAMC+w4cP\na+/evUpISKj01rYn/Kzfs/nkk0/qoosuksvlOmm/1atXa/z48T9TVQAAAKip5cuXn/T7fv0Km5GR\nkQoICJDb7fZpd7vdatmy5UnHFhcXa+XKlZozZ84p5zlxO7bly5crJibGnxLxM5g2bZoyMjJquwxU\ngm1Tt7F96i62Td3Ftqm7cnNzNX78eJ/b6FbGr7AZFBQkl8ulrKws7zmbxhhlZWWd8k4O//jHP3T0\n6NFq3enixKHzmJgYvnuuDoqIiGC71FFsm7qN7VN3sW3qLrZN3XeqUx79vhp9+vTpevzxx/XMM8/o\ns88+05QpU1RcXKyUlBRJUnJysu68884K45588kldeeWVatq0qb9TAgAAoJ7y+5zNxMREeTwezZw5\nU263W71799bq1avVvHlzSdL+/fsVGOi72h07duiDDz7Q22+/fWaqBgAAQL1QowuE0tLSlJaWVulj\na9asqdDWpUsXlZWV1WQqAAAA1GM1ul0lzm1JSUm1XQKqwLap29g+dRfbpu5i29R/fn2p+88lOztb\nLpdLmzZt4qRgAACAOqi6eY09mwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsI\nmwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACw\nhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAA\nAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsA\nAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIaw\nCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABr\nCJsAAACwhrAJAAAAawibAAAAsKZGYXPhwoWKjo5WSEiI+vXrp40bN560f1FRkW6++Wa1bt1awcHB\n6tatm/71r3/VqGAAAADUH4H+Dli5cqVmzJihxx57TH379lVGRoYSEhK0Y8cORUZGVuhfWlqq+Ph4\ntWzZUi+99JJat26t/Px8NWnS5Iw8AQAAANRdfofNjIwMpaamKjk5WZK0ZMkSvf7661q6dKluv/32\nCv2ffPJJHTp0SB9++KECAgIkSe3btz/NsgEAAFAf+HUYvbS0VJs2bVJcXJy3zeFwKD4+Xhs2bKh0\nzKpVq9S/f3+lpaWpZcuWuuiiizRv3jyVl5efXuUAAACo8/zas+nxeFRWVqaoqCif9qioKOXl5VU6\nZs+ePVqzZo3Gjx+vN998U7t27dJNN92kY8eO6Z577ql55QAAAKjz/D6M7q/y8nJFRUXpsccek8Ph\nUJ8+fbR//3499NBDhE0AAICznF9hMzIyUgEBAXK73T7tbrdbLVu2rHRMq1at1KBBAzkcDm9bTEyM\nCgsLdezYMQUGVl3CtGnTFBER4dOWlJSkpKQkf8rGacrMPL5IUkmJlJ8vdeggBQcfb0tKOr4AAICz\nU2ZmpjJPhIH/U1RUVK2xfoXNoKAguVwuZWVlacyYMZIkY4yysrI0derUSsdcdtllFYrLy8tTq1at\nTho0peMXI8XGxvpTIiz4cZjMzpZcruPhk00DAMC5obKdfdnZ2XK5XKcc6/f3bE6fPl2PP/64nnnm\nGX322WeaMmWKiouLlZKSIklKTk7WnXfe6e1/00036ZtvvtHUqVO1c+dOvf7665o3b55uueUWf6cG\nAABAPeP3OZuJiYnyeDyaOXOm3G63evfurdWrV6t58+aSpP379/vssWzbtq1Wr16tadOmqVevXmrT\npo2mTZtW6dckAQAA4OxSowuE0tLSlJaWVulja9asqdD2i1/8Qh988EFNpgIAAEA9xr3RAQAAYA1h\nEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADW\nEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAA\nYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMA\nAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1gTW\ndgHwVVBQII/HU9tlVCk3N0RSjHJzcyUdru1yqhQZGan27dvXdhkAAJzzCJt1SEFBgWK6dlVxSUlt\nl3ISfSRla/z46yVtru1iqhQaHKzcvDwCJwAAtYywWYd4PB4Vl5RouaSY2i6mCrmSxkt1v8aSEnk8\nHsImAAC1jLBZB8VIiq3tIk6hPtQIAABqHxcIAQAAwBrCJgAAAKwhbAIAAMAawiYAAACsIWwCAADA\nGsImAAAArCFsAgAAwBrCJgAAAKwhbAIAAMAawiYAAACsIWwCAADAGsImAAAArCFsAgAAwBrCJgAA\nAKwhbAIAAMAawiYAAACsqVHYXLhwoaKjoxUSEqJ+/fpp48aNVfZdtmyZnE6nAgIC5HQ65XQ6FRoa\nWuOCAQAAUH/4HTZXrlypGTNmKD09XZs3b1avXr2UkJAgj8dT5ZiIiAgVFhZ6l/z8/NMqGgAAAPWD\n32EzIyNDqampSk5OVrdu3bRkyRKFhoZq6dKlVY5xOBxq3ry5WrRooRYtWqh58+anVTQAAADqB7/C\nZmlpqTZt2qS4uDhvm8PhUHx8vDZs2FDluO+//14dO3ZU+/btdeWVV2r79u01rxgAAAD1hl9h0+Px\nqKysTFFRUT7tUVFRKiwsrHRM165dtXTpUr322mtasWKFysvLdemll+qLL76oedUAAACoFwJtT9Cv\nXz/169fP+3P//v0VExOjRx99VOnp6ScdO23aNEVERPi0JSUlKSkpyUqtAAAAqCgzM1OZmZk+bUVF\nRdUa61fYjIyMVEBAgNxut0+72+1Wy5YtqzdhYKD69OmjXbt2nbJvRkaGYmNj/SkRAAAAZ1hlO/uy\ns7PlcrlOOdavw+hBQUFyuVzKysrythljlJWVpUsvvbRa6ygvL9fWrVvVqlUrf6YGAABAPeT3YfTp\n06crJSVFLpdLffv2VUZGhoqLi5WSkiJJSk5OVtu2bTV37lxJ0uzZs9WvXz+df/75OnTokP785z+r\noKBAkyZNOqNPBAAAAHWP32EzMTFRHo9HM2fOlNvtVu/evbV69Wrv1xnt379fgYH/W+23336rG2+8\nUYWFhWratKlcLpc2bNigbt26nblnAQAAgDqpRhcIpaWlKS0trdLH1qxZ4/PzX//6V/31r3+tyTQA\nAACo57g3OgAAAKwhbAIAAMAawiYAAACsIWwCAADAGsImAAAArCFsAgAAwBrCJgAAAKwhbAIAAMAa\nwiYAAACsIWwCAADAGsImAAAArCFsAgAAwJrA2i4AdV+mrlWmkiRJJWqoLsrTnzRPwToiSUpSppL0\nXG2WCAAA6ijCJk4pSc8RJgEAQI1wGB0AAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEA\nAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWET\nAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQ\nNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABg\nDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAA\nANbUKGwuXLhQ0dHRCgkJUb9+/bRx48ZqjXvuuefkdDp11VVX1WRaAAAA1DN+h82VK1dqxowZSk9P\n1+bNm9WrVy8lJCTI4/GcdNzevXt12223aeDAgTUuFgAAAPWL32EzIyNDqampSk5OVrdu3bRkyRKF\nhoZq6dKlVY4pLy/X+PHjdd999yk6Ovq0CgYAAED94VfYLC0t1aZNmxQXF+dtczgcio+P14YNG6oc\nl56erqioKE2YMKHmlQIAAKDeCfSns8fjUVlZmaKionzao6KilJeXV+mY9evX66mnnlJOTk7NqwQA\nAEC9ZPVq9O+//17Jycl6/PHH1bRpU5tTAQAAoA7ya89mZGSkAgIC5Ha7fdrdbrdatmxZof/u3buV\nn5+v0aNHyxgj6fj5m5LUoEED5eXlnfQczmnTpikiIsKnLSkpSUlJSf6UDQAAgNOQmZmpzMxMn7ai\noqJqjfUrbAYFBcnlcikrK0tjxoyRJBljlJWVpalTp1boHxMTo61bt/q03XXXXfr++++1YMECtWvX\n7qTzZWRkKDY21p8SAQAAcIZVtrMvOztbLpfrlGP9CpuSNH36dKWkpMjlcqlv377KyMhQcXGxUlJS\nJEnJyclq27at5s6dqwYNGujCCy/0Gd+kSRM5HA7FxMT4OzUAAADqGb/DZmJiojwej2bOnCm3263e\nvXtr9erVat68uSRp//79Cgz0e7UAAAA4C9UoFaalpSktLa3Sx9asWXPSsU899VRNpgQAAEA9xL3R\nAQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDd9RBNRjmZnHF0kqKZHy86UOHaTg4ONtSUnH\nFwAAagthE6jHfhwms7Mll+t4+OTGWwCAuoLD6AAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAA\nsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkA\nAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawib\nAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCG\nsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAA\nawibAAAAsCawtgsA6ouCggJ5PJ7aLqNKubkhkmKUm5sr6XBtl3NSkZGRat++fW2XAQD4GRA2gWoo\nKChQTNeuKi4pqe1STqKPpGyNH3+9pM21XcxJhQYHKzcvj8AJAOcAwiZQDR6PR8UlJVouKaa2i6lC\nrqTxUp2uUfq/OktK5PF4CJsAcA4gbAJ+iJEUW9tFnEJ9qBEAcO7gAiEAAABYQ9gEAACANYRNAAAA\nWEPYBAAAgDU1CpsLFy5UdHS0QkJC1K9fP23cuLHKvi+//LIuueQSNW3aVI0aNVKfPn20fPnyGhcM\nAACA+sPvsLly5UrNmDFD6enp2rx5s3r16qWEhIQqv+z6vPPO0913360PP/xQW7du1YQJEzRhwgS9\n/fbbp108AAAA6ja/w2ZGRoZSU1OVnJysbt26acmSJQoNDdXSpUsr7T9w4EBdccUV6tq1q6KjozV1\n6lT17NlT69evP+3iAQAAULf5FTZLS0u1adMmxcXFedscDofi4+O1YcOGaq0jKytLO3bs0KBBg/yr\nFAAAAPWOX1/q7vF4VFZWpqioKJ/2qKgo5eXlVTnuu+++U5s2bXTkyBEFBgZq0aJFGjp0aM0qBgAA\nQL3xs9xBKDw8XDk5Ofr++++VlZWladOmqVOnTho4cOBJx02bNk0RERE+bUlJSUpKSrJZLgAAAH4k\nMzNTmZk6P9+YAAAgAElEQVSZPm1FRUXVGutX2IyMjFRAQIDcbrdPu9vtVsuWLasc53A41KlTJ0lS\nz549tX37ds2bN++UYTMjI0Oxsdx4DwAAoDZVtrMvOztbLpfrlGP9OmczKChILpdLWVlZ3jZjjLKy\nsnTppZdWez3l5eU6cuSIP1MDAACgHvL7MPr06dOVkpIil8ulvn37KiMjQ8XFxUpJSZEkJScnq23b\ntpo7d64kaf78+br44ovVuXNnHTlyRK+//rqWL1+uJUuWnNEnAgAAgLrH77CZmJgoj8ejmTNnyu12\nq3fv3lq9erWaN28uSdq/f78CA/+32h9++EE333yz9u/fr5CQEHXr1k0rVqzQ2LFjz9yzAAAAQJ1U\nowuE0tLSlJaWVulja9as8fl59uzZmj17dk2mAQAAQD3HvdEBAABgDWETAAAA1hA2AQAAYA1hEwAA\nANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYB\nAABgDWETAAAA1hA2AQAAYA1hEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1hA2AQAAYA1h\nEwAAANYQNgEAAGANYRMAAADWEDYBAABgDWETAAAA1gTWdgEAai5T1ypTSZKkEjVUF+XpT5qnYB2R\nJCUpU0l6rjZLBACc4wibQD2WpOcIkwCAOo3D6AAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAA\nsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkA\nAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawib\nAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwpkZhc+HChYqOjlZISIj69eunjRs3Vtn3iSee\n0MCBA9WsWTM1a9ZMl19++Un7AwAA4Ozhd9hcuXKlZsyYofT0dG3evFm9evVSQkKCPB5Ppf3XrVun\n6667TmvXrtWHH36odu3aadiwYfryyy9Pu3gAAADUbX6HzYyMDKWmpio5OVndunXTkiVLFBoaqqVL\nl1ba/9lnn9WUKVPUs2dPdenSRU888YTKy8uVlZV12sUDAACgbvMrbJaWlmrTpk2Ki4vztjkcDsXH\nx2vDhg3VWscPP/yg0tJSNWvWzL9KAQAAUO/4FTY9Ho/KysoUFRXl0x4VFaXCwsJqreOPf/yj2rRp\no/j4eH+mBgAAQD0U+HNONn/+fP3jH//QunXr1KBBg59zagAAANQCv8JmZGSkAgIC5Ha7fdrdbrda\ntmx50rEPPfSQ/vznPysrK0vdu3ev1nzTpk1TRESET1tSUpKSkpL8KRsAAACnITMzU5mZmT5tRUVF\n1RrrV9gMCgqSy+VSVlaWxowZI0kyxigrK0tTp06tctyf//xnzZs3T2+99Zb69OlT7fkyMjIUGxvr\nT4kAAAA4wyrb2ZednS2Xy3XKsX4fRp8+fbpSUlLkcrnUt29fZWRkqLi4WCkpKZKk5ORktW3bVnPn\nzpUkPfDAA7r33nuVmZmp9u3be/eKNmrUSGFhYf5ODwAAgHrE77CZmJgoj8ejmTNnyu12q3fv3lq9\nerWaN28uSdq/f78CA/+32iVLlqi0tFRjx471Wc+9996rmTNnnmb5AAAAqMtqdIFQWlqa0tLSKn1s\nzZo1Pj9//vnnNZkCAAAAZwHujQ4AAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCG\nsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAA\nawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAA\nALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJ\nAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsI\nmwAAALCGsAkAAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACw\nhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABrCJsAAACwpkZhc+HChYqOjlZISIj69eun\njRs3Vtl3+/btGjt2rKKjo+V0OrVgwYIaFwsAAID6xe+wuXLlSs2YMUPp6enavHmzevXqpYSEBHk8\nnkr7FxcXq3PnznrggQfUqlWr0y4YAAAA9YffYTMjI0OpqalKTk5Wt27dtGTJEoWGhmrp0qWV9r/4\n4ov1wAMPKDExUQ0aNDjtggEAAFB/+BU2S0tLtWnTJsXFxXnbHA6H4uPjtWHDhjNeHAAAAOo3v8Km\nx+NRWVmZoqKifNqjoqJUWFh4RgsDAABA/RdY2wWczLRp0xQREeHTlpSUpKSkpFqqCAAA4NyTmZmp\nzMxMn7aioqJqjfUrbEZGRiogIEBut9un3e12q2XLlv6sqloyMjIUGxt7xtcLAACA6qtsZ192drZc\nLtcpx/p1GD0oKEgul0tZWVneNmOMsrKydOmll/qzKgAAAJwD/D6MPn36dKWkpMjlcqlv377KyMhQ\ncXGxUlJSJEnJyclq27at5s6dK+n4RUXbt2+XMUZHjx7VgQMHlJOTo0aNGqlz585n9MkAAACgbvE7\nbCYmJsrj8WjmzJlyu93q3bu3Vq9erebNm0uS9u/fr8DA/632iy++UJ8+feRwOCRJDz30kB566CEN\nGjRIa9asOUNPAwAAAHVRjS4QSktLU1paWqWP/TRAdujQQeXl5TWZBgAAAPUc90YHAACANYRNAAAA\nWEPYBAAAgDWETQAAAFhD2AQAAIA1hE0AAABYQ9gEAACANYRNAAAAWEPYBAAAgDWETQAAAFhD2AQA\nAIA1hE0AAABYQ9gEAACANYRNAAAAWBNY2wUAwNkqM/P4IkklJVJ+vtShgxQcfLwtKen4AgBnM8Im\nAFjy4zCZnS25XMfDZ2xs7dYFPggAPyfCJgDgnMMHAeDnwzmbAAAAsIawCQAAAGsImwAAALCGsAkA\nAABrCJsAAACwhrAJAAAAawibAAAAsIawCQAAAGsImwAAALCGsAkAAABruF0lgHqvoKBAHo+ntss4\nqdzcEEkxys3NlXS4tsupUmRkpNq3b1/bZQA4ixA2AdRrBQUFiunaVcUlJbVdyin0kZSt8eOvl7S5\ntoupUmhwsHLz8s5o4KzrHwb4IADYRdgEUK95PB4Vl5RouaSY2i7mJHIljZfqdJ25ksaXlMjj8Zyx\nUFM/Pgycux8EgJ8DYRPAWSFGUmxtF1EN9aXOM6U+fBg4Vz8IAD8XwiYAwLr6ELLrQ41AfcTV6AAA\nALCGsAkAAABrCJsAAACwhrAJAAAAa7hACAAA1BmZmccXSSopkfLzpQ4dpODg421JSccX1B+ETQAA\nUGf8OExmZ0su1/HwGctXBdRbHEYHAACANYRNAAAAWEPYBAAAgDWcswkAwDmqoKBAHo+ntsuoUm5u\niKQY5ebmSjpc2+WcVGRkJLcSrQJhEwCAc1BBQYFiunZVcUlJbZdyEn0kZWv8+Oslba7tYk4qNDhY\nuXl5BM5KEDYBwJJMXatMHb+stkQN1UV5+pPmKVhHJElJylSSnqvNEs9ZbBvJ4/GouKREy3X8vvB1\nUa6k8VKdrlH6vzpLSuTxeAiblSBsAoAlSXrurA8s9RXb5n9iJNX1bxWqDzWialwgBAAAAGsImwAA\nALCGsAkAAABrOGcTAADUGVy8dfYhbAIAgDqDi7fOPhxGBwAAgDWETQAAAFhD2AQAAIA1hE0AAABY\nQ9iE3zJruwBUiW1Tt7F96i62Td3Ftqn/ahQ2Fy5cqOjoaIWEhKhfv37auHHjSfs///zziomJUUhI\niHr16qU333yzRsWibuCNX3exbeo2tk/dxbapu9g29Z/fYXPlypWaMWOG0tPTtXnzZvXq1UsJCQny\neDyV9v/ggw903XXXafLkydqyZYuuuOIKXXnlldq+fftpFw8AAIC6ze+wmZGRodTUVCUnJ6tbt25a\nsmSJQkNDtXTp0kr7L1iwQCNGjND06dPVtWtX3XfffYqNjdUjjzxy2sUDAACgbvMrbJaWlmrTpk2K\ni4vztjkcDsXHx2vDhg2VjtmwYYPi4+N92hISEqrsDwAAgLOHX3cQ8ng8KisrU1RUlE97VFSU8vLy\nKh1TWFhYaf/CwsIq5zl8+LAkKTc315/y6r0Tz/cNSXX5me+XtKK2iziJz//vv2fy3w/b5sw509un\nvmwbqe5vH947dRfbpm6zsX3qgxPP90Ruq5LxwxdffGEcDof58MMPfdpvv/12069fv0rHNGjQwDz3\n3HM+bYsWLTItW7ascp7ly5cbSSwsLCwsLCwsLHV8Wb58+Unzo197NiMjIxUQECC32+3T7na71bJl\ny0rHtGzZ0q/+0vHD7MuXL1fHjh0VEhLiT4kAAAD4GRw+fFh79+5VQkLCSfs5jDHGnxX369dPv/jF\nL/S3v/1NkmSMUfv27TV16lTddtttFfpfe+21Onz4sF599VVv22WXXaZevXpp0aJF/kwNAACAeiZg\n1qxZs/wZ0LhxY82cOVPt2rVTw4YNdffddysnJ0dPPPGEwsLClJycrI0bN3ovImrTpo3uvvtuhYWF\nqVmzZnrkkUf0/PPP68knn1Tz5s1tPCcAAADUEX4dRpekxMREeTwezZw5U263W71799bq1au9wXH/\n/v0KDPzfavv376//9//+n+666y7ddddduuCCC/Tqq6/qwgsvPHPPAgAAAHWS34fRAQAAgOri3ugA\nAKDG8vPz5XQ69emnn9Z2KaijCJtnkQkTJsjpdCogIEANGzbUBRdcoNmzZ6u8vLy2S4NFr7zyii64\n4AIFBQVp+vTptV0OqoE/zqhtKSkpcjqdSktLq/DYzTffLKfTqYkTJ1ZrXe3bt1dhYaF69OhR7fnT\n09PVp0+favdH/UbYPMuMGDFChYWF2rVrl/7whz9o1qxZevDBB2u0rtLS0jNcHX7K4/HopptuUocO\nHRQcHKxWrVppxIgRft1ha8qUKUpMTNT+/fs1e/bsao2Jjo7WggULalr2We3Eh7YTS2RkpEaMGKGt\nW7eesTlq8sf5XDFhwgRdddVVP+ucy5Yt835QdzqdCg8P18UXX6yXX375jM9VV957DodD7du313PP\nPacjR454248cOaLMzEx16NDBr3W1aNFCTqd/kcLhcPjVH/UXYfMs07BhQzVv3lzt2rVTamqq4uPj\n9dprr0mS1q9fr4EDByo0NFQdOnTQ73//exUXF3vHRkdHa86cObrhhhsUERGh1NRUlZaW6pZbblHr\n1q0VEhKi6OhoPfDAA94x+/bt0xVXXKHw8HBFRETommuu0VdffeV9/MSn1+XLlys6OlpNmjRRUlKS\nfvjhB2+fF154QT179lRoaKgiIyM1bNiwU9+N4Cxx1VVXKScnR88++6x27typVatWafDgwTp48GC1\nxn///ff66quvNGzYMEVFRSksLMxyxeeGESNGyO12q7CwUGvWrFFgYKBGjx59xtZf0z/OsCciIkKF\nhYUqLCzUli1blJCQoMTERO3cubO2S7OmT58+ateunV566SVv20svvaQOHTr47HVcvXq1BgwYoKZN\nmyoyMlKjR4/Wnj17vI//dE/9unXr5HQ6tWbNGl1yySUKCwvTZZdd5n0tly1bpvT0dOXk5HhD/jPP\nPCNJKioq0qRJk9SiRQtFREQoPj7e5whAdf6mVLfel19+WUOHDlVYWJh69+6tDz/80Of1ef/99zVk\nyBDvt+mMGDFCRUVFko5/7eO8efPUqVMnhYaGqk+fPnrxxRdPe5ucrfhNd5YLCQnR0aNHtWfPHo0Y\nMULjxo3Tf/7zH61cuVLvv/++fve73/n0/8tf/qLevXtry5Ytuueee7RgwQL985//1AsvvKAdO3Zo\nxYoV6tixo6Tjb7YxY8bo0KFDeu+99/TOO+9oz549uvbaa33WuXv3br366qt644039Prrr2vdunWa\nP3++pOO3M73uuus0adIkffbZZ1q3bp2uuuoqnQvXrRUVFWn9+vV64IEHNHDgQLVr104XX3yx/vjH\nP2rUqFGSpIyMDPXs2VONGjVS+/btdfPNN3s/IKxbt06NGzeWw+HQkCFDFBAQoH//+9+SKv9gcSLA\nDxkyRPn5+Zo2bZr3F31xcbEiIiJ8/uhIxw/RN2rUyOcX+bngxIe2Fi1aqGfPnvrTn/6kffv26eDB\ng1q7dq2cTqe+++47b/8TfzQLCgqq9Vr6+8f5hDlz5igqKkoRERGaPHmy7rjjjnPmUGRlpx4UFRXJ\n6XR6/91L0muvvaYuXbooJCREcXFxeuaZZypsr8o4HA7vNu/cubPmzJlTYb5Dhw4pOTlZzZo1U1hY\nmEaOHKldu3b5rOfFF19Ujx49FBwcrOjoaP31r3/1PlbZe682ORwOTZw4UUuXLvW2LV26VBMmTPD5\nHfzDDz9oxowZys7O1po1axQQEKBf//rXFdb1U3fffbcyMjK0adMmBQYGeg/LX3PNNZoxY4a6d+8u\nt9utL7/8Utdcc40kaezYsTp48KBWr16t7OxsxcbGKj4+XocOHfKu92R/U6pb74n6br/9duXk5KhL\nly667rrrvKedbdmyRfHx8erRo4c+/PBDvf/++xo9erTKysokSXPnztXy5cv12GOPafv27Zo2bZp+\n85vf6L333vN7O5wTTn6DStQnKSkp5te//rX357ffftsEBweb22+/3UyaNMlMmTLFp/97771nAgIC\nzJEjR4wxxnTs2NFcffXVPn2mTp1q4uPjK53vrbfeMkFBQebAgQPetu3btxuHw2E++eQTY4wxs2bN\nMo0aNTI//PCDt8/tt99u+vfvb4wxJjs72zidTlNQUHAaz7x+OnbsmAkPDzfTp0/3boOf+tvf/mbW\nrl1r8vPzzbvvvmtiYmLMzTffbIwxprS01OzYscM4HA7zyiuvGLfbbUpLS82uXbtMo0aNzIIFC8zu\n3bvNhg0bjMvlMhMnTjTGGPPNN9+Ydu3amfvvv9+43W7jdruNMcbceOONZtSoUT7zX3HFFWbChAkW\nX4W656fvo//+978mNTXVdO3a1RhjzNq1a43T6TRFRUXePlu2bDFOp9Pk5+cbY079Wu7du9c4nU6T\nk5PjXafD4TD9+/c37733nsnNzTUDBw40v/zlL73jly9fbkJCQsyyZcvMzp07zX333WciIiJMnz59\n7LwQteSnr/8JP33NjDHm0KFDxuFwmHXr1hljjNmzZ49p0KCB+eMf/2h27NhhVq5cadq2bVthe/3U\n008/bZo2ber9uayszCxdutQ0bNjQ7Nmzx9s+ZswY0717d/P++++bTz/91AwfPtx06dLFHDt2zBhj\nzCeffGICAgLM/fffb3bu3GmWLVtmQkNDzbJly4wxVb/3asOJ1/nrr782wcHBpqCgwOzdu9eEhoaa\ngwcPmiuvvLLK9/7XX39tHA6H2bZtmzHm+LZxOBw+/56dTqd59913vWPeeOMN43Q6vb/rZs2aVeHf\n7vr1602TJk3M0aNHfdrPP/988/jjj3vHnexvij/1PvXUU94+27dvN06n0+Tl5RljjLnuuuvMgAED\nKl3fkSNHTFhYWIVbd0+aNMlcf/31VdZxLvP7ezZRt61atUrh4eEqLS2VMUbXX3+9Zs2apUGDBmnr\n1q1avny5t6/5v0+un3/+ubp27SpJcrlcPutLSUnR5Zdfrq5du2r48OEaNWqULr/8cknSZ599pnbt\n2ql169be/jExMWrSpIlyc3O96+rYsaNCQ0O9fVq1auU91N6rVy/FxcWpR48eSkhI0LBhwzR27Fg1\nadLEwqtTtwQEBGjZsmWaPHmyFi9erNjYWA0aNEjXXnutLrroIknS1KlTvf3bt2+v2bNn66abbtIj\njzyiwMBAtWjRQpLUtGlT7//Pnz9f48eP9+617tSpkx5++GENHjxYixcvVtOmTRUQEKBGjRp5x0jS\npEmTdNlll8ntdisqKkpff/213njjDa1Zs+bneknqjBPvI+n4XpLWrVvrn//8Z7XHV+e1ND/Ze+9w\nODR37lz98pe/lCT96U9/0qhRo3T06FE1aNBAjzzyiCZPnqzk5GRJ0j333KO33nrrnNrr/NPX7Kce\nffRRdevWzbuX64ILLtDWrVs1d+7cU6770KFDaty4sYwxOnz4sBo0aKDHHntM0dHRkqRdu3Zp1apV\n2rBhg37xi19IklasWKF27drplVde0dVXX62MjAzFx8frzjvvlCSdf/752rZtmx588EElJydX+d6r\nTZGRkRo1apSeeuopGWP0q1/9Ss2aNfPps3PnTt1777366KOP5PF4VF5eLofDoYKCgpN+Z/aJ32PS\n8d/7kvTVV1+pbdu2lfbPycnRf//73wrzl5SUaPfu3d6fT/Y3RTq+rWbOnHnKen9anzFGX331lbp0\n6aItW7YoMTGx0jp37dql4uJiXX755T7/JktLS8+ZIw3+4jD6WWbo0KH69NNPtWvXLh0+fFhLly5V\nSEiIvv/+e6WmpurTTz9VTk6OcnJy9Omnn2rHjh3q3Lmzd/xPz/nr06eP9u7dqzlz5qikpESJiYlV\nvgGrEhQU5POzw+HwHqpwOp1666239K9//Uvdu3fX3//+d3Xr1k35+fk1fAXql1//+tf64osvtGrV\nKo0YMULr1q1TbGys9/yld955R/Hx8Wrbtq0aN26s3/zmNzp48KBKSkqqXGdOTo6efvpphYeHe5fh\nw4dLOv7BoiqXXHKJLrzwQi1btkyS9Oyzz6pjx47e8HMuOfE+ysnJ0caNG5WQkKDhw4dr37591Rpf\n09eyqj/OkpSXl6dLLrnEp3/fvn2r/ZzOBTt27Djla3TiPdG4cWOfK7EbN27s/d24ZcsWzZ07V6mp\nqXr99dclSbm5uQoKCvJZX7NmzdS1a1fl5uZ6+1x22WU+8504HeJUQbk2TZgwQU8//bSeeeYZ/fa3\nv63w+OjRo/Xtt9/qiSee0Mcff6yPP/5YxhgdPXr0pOv98e/+E4fZT/btKN9//71at27t83cqJydH\neXl5PrfDPtnfFEkaNWpUteo9WX0hISEnrVOS3njjDZ86t2/frhdeeKHKcecy9myeZcLCwryfxH8s\nNjZW27dvr/SxU2nUqJHGjRuncePG6eqrr9aIESN06NAhxcTEaN++fTpw4IDatGkjSdq+fbsOHTqk\n7t27+zVH//791b9/f91zzz3q0KGDXn75Zd16661+11ofNWjQQHFxcYqLi9Ndd92lyZMn695779Wg\nQYM0evRo3Xzzzf+/vbsPaaqL4wD+3QapOafLsjcMieWaaZlkloFihS8Jtui/EoxQCmJILMsiU0hz\nilj06kuBaZEZGhkOLBUylfKlMSWabvkWWYIO/xFzRef5I3Yfr1pbPq169PeBgbvn3ruzM+/u75x7\nfne4cOEClixZgufPnyMxMREWiwXOzs6z7s/asUhOTp5xgluzZs0P65KYmIjr16/j5MmTKCkpsfvW\nJ/PN9OOouLgY7u7uKC4u5kb2p49oTDeXtvzZk/NCYk2mstXutuj1eu5viUTC2//Uz9zf3x+1tbXI\nyclBbGzsXKr8vxEdHQ2LxQKRSITIyEhemdlsRk9PD27dusUF0k1NTf/5NRctWsTNf7QKCgrCx48f\nIRKJbH5XfY+99bWVCb9x40bU19cjPT19Rpmfnx+cnJwwMDCwIDvjc0EjmwvEqVOn0NLSApVKBb1e\nD5PJhEePHs1IEJru4sWLKC8vR3d3N3p6elBRUYEVK1bAw8ODmzx98OBB6HQ6tLa2IiEhAREREXZf\nSmhtbUV2djY6Ojrw7t07VFZWYmRkZEH/nKmfnx/Gx8fR0dEBxhjy8vKwdetWyGQyvH//3ub2UzsW\na9eu5T2sPyU72xc9AMTHx2NgYABXrlzBmzdvuEu25FswMjExgWXLloExhg8fPnBlOp1uxvq/ui3l\ncjna2tp4y6Y/n8+sP4k8vd2nBg1yuRzt7e287VpbW3nPpx4PS5cu/eFrikQiLrFOoVDgy5cvePny\nJVc+OjqK7u5urnOtUCjQ3NzM20dTUxN8fX25en7v2PuThEIhDAYDXr9+PSMIk0ql8PT0RFFREd6+\nfYuGhgao1WqbwdpsI7lTl/n4+KCvrw96vR6jo6OwWCzYvXs3tm/fDqVSiadPn2JgYAAtLS04e/Ys\nXr16Zdd7sbe+tkaaT58+jba2Nhw7dgxdXV0wGAwoKCiA2WyGWCzGiRMncPz4cZSWlqK3txc6nQ5X\nr15FWVmZXfVcaCjYXCACAgLw7NkzGI1GhIWFISgoCBkZGdyIJDB7T8/NzQ25ubkIDg5GSEgIBgcH\nodVqufLq6mpIpVKEh4cjMjISMpkM5eXldtdLIpGgsbERsbGxkMvlOHfuHPLz82f0rucjs9mMXbt2\n4e7du+jq6kJ/fz8ePHiA3NxcKJVKyGQyfP78GZcvX0ZfXx/KyspQWFhoc7/2dCx8fHzQ2NiIoaEh\n3m2WPDw8sG/fPqSkpCAqKoo3H3chmZycxPDwMIaHh2EwGKBSqTA+Po64uDjIZDJ4e3sjIyMDJpMJ\nNTU1vIxjq59tS1snZ5VKhZs3b6K0tBQmkwmZmZno7Oycl/cqHBsb412e1Ov1GBkZwbZt26DRaLg7\nV6SlpfG2O3LkCAwGA1JTU2E0GlFRUcFNZbAnOLJ+5v39/SgqKkJtbS2USiWAb/Mv4+LikJSUhObm\nZuj1esTHx8Pb2xtxcXEAALVajfr6emRmZsJoNOL27du4du0a7xLw9469P00sFkMsFs9YLhAIUF5e\njh8fbPoAAALqSURBVI6ODgQEBECtViMvL2/W9X70fPqy/fv3Izo6GhEREfDy8uLOG1qtFmFhYTh8\n+DDkcjkOHDiAwcFBLF++3K73IRAIcP/+/Z+u7/Rl69atw5MnT9DZ2YmQkBDs2LED1dXVXIf9/Pnz\nSEtLg0ajgZ+fH2JiYqDVaud09XBB+J3ZSISQf01OTrIzZ86wLVu2MKlUysRiMVMoFCw9PZ19+vSJ\nMcbYpUuX2OrVq5mrqyuLiYlhd+7c4WXWjo2NMaFQyGXjWrW3t7OoqCgmkUiYm5sbCwwMZNnZ2Vz5\nixcvWGBgIHN2dmZCoZC3bUNDAxMIBKyystLBLfB3OnToEBMKhdzD3d2dhYSEsIcPH3LrtLS0sE2b\nNrHFixez8PBwVllZyctGt/peW86WjW4rw50xxjIzM5mXlxeTSCQsMTGRJScns9DQUEc0wx8zvf2t\nj6SkJGYwGFhoaChzdXVlQUFBrK6ubsb//+PHj5mvry9zcXFhO3fuZIWFhbws6NmUlJTwXsvFxYWt\nX7+eaTQa9vXrV269sbExlpCQwKRSKXN1dWV79uxhJpOJt6+qqirm7+/PnJycmI+PD8vPz+eV/+jY\nI2S+EjD2F89aJoT8dmVlZVCr1RgaGuJ68WRuHN2WkZGRWLlyJTd6R2bKyspCUVHRgkk6JORvRGcS\nQggAYGJiAr29vcjJycHRo0cp0PwPHNGWExMTKCgoQFRUFIRCIe7du4f6+nrU1dX9ghrPHzdu3EBw\ncDA8PT3R1NSEvLw83i3ECCG/H83ZJIQAAHJzc7F582asWrUKqampf7o6/2uOaEuBQACtVovw8HAE\nBwejpqYGVVVViIiI+CX7ny+MRiP27t2LDRs2ICsrCykpKbNmFBNCfh+6jE4IIYQQQhyGRjYJIYQQ\nQojDULBJCCGEEEIchoJNQgghhBDiMBRsEkIIIYQQh6FgkxBCCCGEOAwFm4QQQgghxGEo2CSEEEII\nIQ5DwSYhhBBCCHGYfwBKgvitX56nTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa104a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(num=None, figsize=(8, 6), dpi=80, facecolor='w', edgecolor='k')\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(5), importances[indices][:5], \n",
    "       color=\"r\",  yerr=std[indices][:5], align=\"center\")\n",
    "plt.xticks(range(5), best_features)\n",
    "plt.xlim([-1, 5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we implement our DecisionTreeClassifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTree\n",
      "Acurracy:  0.75289017341\n",
      "time elapsed:  0.0060002803802490234\n"
     ]
    }
   ],
   "source": [
    "t0=time()\n",
    "print (\"DecisionTree\")\n",
    "\n",
    "dt = DecisionTreeClassifier(min_samples_split=20,random_state=99)\n",
    "# dt = DecisionTreeClassifier(min_samples_split=20,max_depth=5,random_state=99)\n",
    "\n",
    "clf_dt=dt.fit(X_train_std,y_train)\n",
    "\n",
    "print (\"Acurracy: \", clf_dt.score(X_test_std,y_test))\n",
    "t1=time()\n",
    "print (\"time elapsed: \", t1-t0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we achieved an accuracy of about 75% on the test set.\n",
    "\n",
    "### Cross validation for Decision Trees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.67071057  0.68923611  0.75826087]\n",
      "0.7060691842\n",
      "time elapsed:  0.026001453399658203\n"
     ]
    }
   ],
   "source": [
    "tt0=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(dt, X, y, cv=3)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt1=time()\n",
    "print (\"time elapsed: \", tt1-tt0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus the overall accuracy of 70% helps us to better understand our classifier. It is comparatively low, but still far better than random guessing at 50%. One reason that the accuracy is a bit on the low sized might be the  small size of the dataset. Several other factors might affect the accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A nice feature in scikit-learn is that it allows us to export the decision tree as a .dot file after training, which we can visualize using the GraphViz program. This program is freely available at http://www.graphviz.org and supported by Linux, Windows, and Mac OS X."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import export_graphviz\n",
    "\n",
    "export_graphviz(tree, \n",
    "                out_file='tree.dot', \n",
    "                feature_names=['Persons', 'Safety'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have installed GraphViz on our computer, we can convert the tree.dot file into a PNG file by executing the following command from the command line in the location where we saved the tree.dot file:\n",
    "\n",
    "> `dot -Tpng tree.dot -o tree.png`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src=\"images/tree.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining weak to strong learners via random forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random forests have gained huge popularity in applications of machine learning during the last decade due to their good classification performance, scalability, and ease of use. Intuitively, a random forest can be considered as an ensemble of decision trees. The idea behind ensemble learning is to combine weak learners to build a more robust model, a strong learner, that has a better generalization error and is less susceptible to overfitting. The random forest algorithm can be summarized in four simple steps:\n",
    "\n",
    "1. Draw a random bootstrap sample of size n (randomly choose n samples from the training set with replacement).\n",
    "2. Grow a decision tree from the bootstrap sample. At each node:\n",
    "    * Randomly select d features without replacement.\n",
    "    * Split the node using the feature that provides the best split according to the objective function, for instance, by maximizing the information gain.\n",
    "    \n",
    "3. Repeat the steps 1 to 2 k times.\n",
    "4. Aggregate the prediction by each tree to assign the class label by majority vote. \n",
    "\n",
    "There is a slight modification in step 2 when we are training the individual decision trees: instead of evaluating all features to determine the best split at each node, we only consider a random subset of those.\n",
    "\n",
    "Although random forests don't offer the same level of interpretability as decision trees, a big advantage of random forests is that we don't have to worry so much about choosing good hyperparameter values. We typically don't need to prune the random forest since the ensemble model is quite robust to noise from the individual decision trees. The only parameter that we really need to care about in practice is the number of trees k (step 3) that we choose for the random forest. Typically, the larger the number of trees, the better the performance of the random forest classifier at the expense of an increased computational cost.\n",
    "\n",
    "Via the sample size n of the bootstrap sample, we control the bias-variance tradeoff of the random forest. By choosing a larger value for n, we decrease the randomness and thus the forest is more likely to overfit. On the other hand, we can reduce the degree of overfitting by choosing smaller values for n at the expense of the model performance. In most implementations, including the RandomForestClassifier implementation in\n",
    "scikit-learn, the sample size of the bootstrap sample is chosen to be equal to the number of samples in the original training set, which usually provides a good\n",
    "bias-variance tradeoff. For the number of features d at each split, we want to choose a value that is smaller than the total number of features in the training set. A reasonable default that is used in scikit-learn and other implementations is d = sqroot(m), where m is the number of features in the training set.\n",
    "\n",
    "#### Parameters in Random forests\n",
    "\n",
    "The Random forest implementation in scikit-learn is called\n",
    "RandomForestClassifier, and it has a number of parameters. As Random forests\n",
    "use many instances of DecisionTreeClassifier, they share many of the same\n",
    "parameters such as the criterion (Gini Impurity or Entropy/Information Gain),\n",
    "max_features, and min_samples_split.\n",
    "Also, there are some new parameters that are used in the ensemble process:\n",
    "\n",
    "* n_estimators: This dictates how many decision trees should be built. A higher value will take longer to run, but will (probably) result in a higher accuracy.\n",
    "* oob_score: If true, the method is tested using samples that aren't in the random subsamples chosen for training the decision trees.\n",
    "* n_jobs: This specifies the number of cores to use when training the decision trees in parallel.\n",
    "\n",
    "The scikit-learn package uses a library called Joblib for in-built parallelization.\n",
    "This parameter dictates how many cores to use. By default, only a single core is\n",
    "used—if you have more cores, you can increase this, or set it to -1 to use all cores.\n",
    "\n",
    "### Applying Random forests\n",
    "\n",
    "Random forests in scikit-learn use the estimator interface, allowing us to use almost\n",
    "the exact same code as before to do cross fold validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomForest\n",
      "Acurracy:  0.75289017341\n",
      "time elapsed:  0.5240299701690674\n"
     ]
    }
   ],
   "source": [
    "t2=time()\n",
    "print (\"RandomForest\")\n",
    "rf = RandomForestClassifier(n_estimators=100,n_jobs=-1)\n",
    "clf_rf = rf.fit(X_train,y_train)\n",
    "print (\"Acurracy: \", clf_rf.score(X_test,y_test))\n",
    "t3=time()\n",
    "print (\"time elapsed: \", t3-t2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross result========\n",
      "[ 0.67764298  0.67013889  0.74608696]\n",
      "0.697956275449\n",
      "time elapsed:  1.5300874710083008\n"
     ]
    }
   ],
   "source": [
    "tt2=time()\n",
    "print (\"cross result========\")\n",
    "scores = cross_validation.cross_val_score(rf, X, y, cv=3)\n",
    "print (scores)\n",
    "print (scores.mean())\n",
    "tt3=time()\n",
    "print (\"time elapsed: \", tt3-tt2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the preceding code, we trained a random forest from 100 decision trees via the n_estimators parameter and used the entropy criterion as an impurity measure to split the nodes. Although we are growing a very small random forest from a very small training dataset, we used the n_jobs parameter for demonstration purposes, which allows us to parallelize the model training using multiple cores of our computer (here, -1 which means all cores will be used)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter, we learned about ensemble methods such as random forests don't require much parameter tuning and don't overfit so easily as decision trees, which makes it an attractive model for many practical problem domains. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}