pytorch_ipynb/tricks/cnn-alexnet-cifar10-batchincrease.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UEBilEjLj5wY"
   },
   "source": [
    "Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.\n",
    "- Author: Sebastian Raschka\n",
    "- GitHub Repository: https://github.com/rasbt/deeplearning-models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 536,
     "status": "ok",
     "timestamp": 1524974472601,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "GOzuY8Yvj5wb",
    "outputId": "c19362ce-f87a-4cc2-84cc-8d7b4b9e6007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sebastian Raschka \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.6.1\n",
      "\n",
      "torch 1.1.0\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -a 'Sebastian Raschka' -v -p torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rH4XmErYj5wm"
   },
   "source": [
    "# Increase the Batch Size (AlexNet CIFAR-10 Classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a notebook experimenting with increasing the batch size during training, which is inspired by the paper\n",
    "\n",
    "- Smith, S. L., Kindermans, P. J., Ying, C., & Le, Q. V. (2017). Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489.\n",
    "\n",
    "To summarize the main points of the paper:\n",
    "\n",
    "- Stochastic gradient descent adds noise to the optimization problem; during the early training epochs, this noise helps with exploring the loss landscape, and in general, it helps with escaping sharp minima which are known to be bad for generalization.\n",
    "\n",
    "- However, during the course of the training process, one wants to decay the learning rate gradually (like simulated annealing) for fine-tuning, i.e., to help with convergence\n",
    "\n",
    "- Due to the relationship between learning rate, batch size, and momentum, one can also just increase the batch size instead of decreasing the learning rate to reduce the noise. This way, more training examples can be used in each update and fewer steps (parameter updates) overall may be required to converge.\n",
    "\n",
    "The relationship between learning rate and batch size is as follows:\n",
    "\n",
    "\n",
    "$$g=\\epsilon\\left(\\frac{N}{B}-1\\right),$$\n",
    "\n",
    "where $\\epsilon$ is the learning rate, $B$ is the batch size, and $N$ is the number of training examples\n",
    "\n",
    "Or, with added momentum term, this becomes:\n",
    "\n",
    "$$\\begin{aligned} g &=\\frac{\\epsilon}{1-m}\\left(\\frac{N}{B}-1\\right) \\\\ & \\approx \\frac{\\epsilon N}{B(1-m)} \\end{aligned}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, the CIFAR-10 dataset is used for training a classic AlexNet network [1] for classification:\n",
    "    \n",
    "- [1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. \"[Imagenet classification with deep convolutional neural networks.](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)\" In Advances in Neural Information Processing Systems, pp. 1097-1105. 2012.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MkoGLH_Tj5wn"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "ORj09gnrj5wp"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import Subset\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I6hghKPxj5w0"
   },
   "source": [
    "## Model Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23936,
     "status": "ok",
     "timestamp": 1524974497505,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "NnT0sZIwj5wu",
    "outputId": "55aed925-d17e-4c6a-8c71-0d9b3bde5637"
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Hyperparameters\n",
    "RANDOM_SEED = 1\n",
    "LEARNING_RATE = 0.0001\n",
    "BATCH_SIZE = 256\n",
    "NUM_EPOCHS = 40\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "\n",
    "# Other\n",
    "DEVICE = \"cuda:0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "train_indices = torch.arange(0, 48000)\n",
    "valid_indices = torch.arange(48000, 50000)\n",
    "\n",
    "\n",
    "train_transform = transforms.Compose([transforms.Resize((70, 70)),\n",
    "                                      transforms.RandomCrop((64, 64)),\n",
    "                                      transforms.ToTensor()])\n",
    "\n",
    "test_transform = transforms.Compose([transforms.Resize((70, 70)),\n",
    "                                     transforms.CenterCrop((64, 64)),\n",
    "                                     transforms.ToTensor()])\n",
    "\n",
    "train_and_valid = datasets.CIFAR10(root='data', \n",
    "                                   train=True, \n",
    "                                   transform=train_transform,\n",
    "                                   download=True)\n",
    "\n",
    "train_dataset = Subset(train_and_valid, train_indices)\n",
    "valid_dataset = Subset(train_and_valid, valid_indices)\n",
    "test_dataset = datasets.CIFAR10(root='data', \n",
    "                                train=False, \n",
    "                                transform=test_transform,\n",
    "                                download=False)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=True)\n",
    "\n",
    "valid_loader = DataLoader(dataset=valid_dataset, \n",
    "                          batch_size=BATCH_SIZE,\n",
    "                          num_workers=4,\n",
    "                          shuffle=False)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=4,\n",
    "                         shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set:\n",
      "\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "\n",
      "Validation Set:\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n",
      "\n",
      "Testing Set:\n",
      "Image batch dimensions: torch.Size([256, 3, 64, 64])\n",
      "Image label dimensions: torch.Size([256])\n"
     ]
    }
   ],
   "source": [
    "# Checking the dataset\n",
    "print('Training Set:\\n')\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    break\n",
    "    \n",
    "# Checking the dataset\n",
    "print('\\nValidation Set:')\n",
    "for images, labels in valid_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    break\n",
    "\n",
    "# Checking the dataset\n",
    "print('\\nTesting Set:')\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.size())\n",
    "    print('Image label dimensions:', labels.size())\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "class AlexNet(nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes):\n",
    "        super(AlexNet, self).__init__()\n",
    "        self.features = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            nn.Conv2d(64, 192, kernel_size=5, padding=2),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "            nn.Conv2d(192, 384, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(384, 256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(256, 256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=3, stride=2),\n",
    "        )\n",
    "        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(256 * 6 * 6, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Dropout(0.5),\n",
    "            nn.Linear(4096, 4096),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(4096, num_classes)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.features(x)\n",
    "        x = self.avgpool(x)\n",
    "        x = x.view(x.size(0), 256 * 6 * 6)\n",
    "        logits = self.classifier(x)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits, probas\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_acc(model, data_loader, device):\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    model.eval()\n",
    "    for i, (features, targets) in enumerate(data_loader):\n",
    "            \n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        assert predicted_labels.size() == targets.size()\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RAodboScj5w6"
   },
   "source": [
    "# Training 1: Constant Batch Size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "_lza9t_uj5w1"
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "\n",
    "model = AlexNet(NUM_CLASSES)\n",
    "model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 1547
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2384585,
     "status": "ok",
     "timestamp": 1524976888520,
     "user": {
      "displayName": "Sebastian Raschka",
      "photoUrl": "//lh6.googleusercontent.com/-cxK6yOSQ6uE/AAAAAAAAAAI/AAAAAAAAIfw/P9ar_CHsKOQ/s50-c-k-no/photo.jpg",
      "userId": "118404394130788869227"
     },
     "user_tz": 240
    },
    "id": "Dzh3ROmRj5w7",
    "outputId": "5f8fd8c9-b076-403a-b0b7-fd2d498b48d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/040 | Batch 000/188 | Cost: 2.3029\n",
      "Epoch: 001/040 | Batch 150/188 | Cost: 1.7122\n",
      "Epoch: 001/040\n",
      "Train ACC: 32.64 | Validation ACC: 31.45\n",
      "Time elapsed: 0.21 min\n",
      "Epoch: 002/040 | Batch 000/188 | Cost: 1.7477\n",
      "Epoch: 002/040 | Batch 150/188 | Cost: 1.6831\n",
      "Epoch: 002/040\n",
      "Train ACC: 44.09 | Validation ACC: 43.40\n",
      "Time elapsed: 0.42 min\n",
      "Epoch: 003/040 | Batch 000/188 | Cost: 1.5064\n",
      "Epoch: 003/040 | Batch 150/188 | Cost: 1.4504\n",
      "Epoch: 003/040\n",
      "Train ACC: 51.37 | Validation ACC: 51.00\n",
      "Time elapsed: 0.63 min\n",
      "Epoch: 004/040 | Batch 000/188 | Cost: 1.4089\n",
      "Epoch: 004/040 | Batch 150/188 | Cost: 1.2423\n",
      "Epoch: 004/040\n",
      "Train ACC: 58.56 | Validation ACC: 57.75\n",
      "Time elapsed: 0.84 min\n",
      "Epoch: 005/040 | Batch 000/188 | Cost: 1.0506\n",
      "Epoch: 005/040 | Batch 150/188 | Cost: 1.1601\n",
      "Epoch: 005/040\n",
      "Train ACC: 59.14 | Validation ACC: 58.55\n",
      "Time elapsed: 1.06 min\n",
      "Epoch: 006/040 | Batch 000/188 | Cost: 1.0774\n",
      "Epoch: 006/040 | Batch 150/188 | Cost: 1.1084\n",
      "Epoch: 006/040\n",
      "Train ACC: 62.25 | Validation ACC: 60.95\n",
      "Time elapsed: 1.27 min\n",
      "Epoch: 007/040 | Batch 000/188 | Cost: 1.0387\n",
      "Epoch: 007/040 | Batch 150/188 | Cost: 1.0570\n",
      "Epoch: 007/040\n",
      "Train ACC: 65.41 | Validation ACC: 63.20\n",
      "Time elapsed: 1.49 min\n",
      "Epoch: 008/040 | Batch 000/188 | Cost: 1.0650\n",
      "Epoch: 008/040 | Batch 150/188 | Cost: 0.9280\n",
      "Epoch: 008/040\n",
      "Train ACC: 64.37 | Validation ACC: 63.95\n",
      "Time elapsed: 1.70 min\n",
      "Epoch: 009/040 | Batch 000/188 | Cost: 1.0195\n",
      "Epoch: 009/040 | Batch 150/188 | Cost: 0.7793\n",
      "Epoch: 009/040\n",
      "Train ACC: 69.71 | Validation ACC: 67.30\n",
      "Time elapsed: 1.91 min\n",
      "Epoch: 010/040 | Batch 000/188 | Cost: 0.7986\n",
      "Epoch: 010/040 | Batch 150/188 | Cost: 0.7988\n",
      "Epoch: 010/040\n",
      "Train ACC: 69.41 | Validation ACC: 65.45\n",
      "Time elapsed: 2.12 min\n",
      "Epoch: 011/040 | Batch 000/188 | Cost: 0.8688\n",
      "Epoch: 011/040 | Batch 150/188 | Cost: 0.7943\n",
      "Epoch: 011/040\n",
      "Train ACC: 70.95 | Validation ACC: 67.35\n",
      "Time elapsed: 2.34 min\n",
      "Epoch: 012/040 | Batch 000/188 | Cost: 0.7696\n",
      "Epoch: 012/040 | Batch 150/188 | Cost: 0.8943\n",
      "Epoch: 012/040\n",
      "Train ACC: 75.26 | Validation ACC: 67.95\n",
      "Time elapsed: 2.55 min\n",
      "Epoch: 013/040 | Batch 000/188 | Cost: 0.6622\n",
      "Epoch: 013/040 | Batch 150/188 | Cost: 0.7226\n",
      "Epoch: 013/040\n",
      "Train ACC: 77.99 | Validation ACC: 72.45\n",
      "Time elapsed: 2.76 min\n",
      "Epoch: 014/040 | Batch 000/188 | Cost: 0.6180\n",
      "Epoch: 014/040 | Batch 150/188 | Cost: 0.6502\n",
      "Epoch: 014/040\n",
      "Train ACC: 77.82 | Validation ACC: 70.85\n",
      "Time elapsed: 2.97 min\n",
      "Epoch: 015/040 | Batch 000/188 | Cost: 0.6359\n",
      "Epoch: 015/040 | Batch 150/188 | Cost: 0.8206\n",
      "Epoch: 015/040\n",
      "Train ACC: 79.41 | Validation ACC: 71.35\n",
      "Time elapsed: 3.18 min\n",
      "Epoch: 016/040 | Batch 000/188 | Cost: 0.6694\n",
      "Epoch: 016/040 | Batch 150/188 | Cost: 0.5700\n",
      "Epoch: 016/040\n",
      "Train ACC: 79.59 | Validation ACC: 70.75\n",
      "Time elapsed: 3.39 min\n",
      "Epoch: 017/040 | Batch 000/188 | Cost: 0.6395\n",
      "Epoch: 017/040 | Batch 150/188 | Cost: 0.5564\n",
      "Epoch: 017/040\n",
      "Train ACC: 82.24 | Validation ACC: 72.75\n",
      "Time elapsed: 3.61 min\n",
      "Epoch: 018/040 | Batch 000/188 | Cost: 0.5724\n",
      "Epoch: 018/040 | Batch 150/188 | Cost: 0.4650\n",
      "Epoch: 018/040\n",
      "Train ACC: 83.02 | Validation ACC: 71.55\n",
      "Time elapsed: 3.82 min\n",
      "Epoch: 019/040 | Batch 000/188 | Cost: 0.4790\n",
      "Epoch: 019/040 | Batch 150/188 | Cost: 0.4548\n",
      "Epoch: 019/040\n",
      "Train ACC: 84.87 | Validation ACC: 73.35\n",
      "Time elapsed: 4.03 min\n",
      "Epoch: 020/040 | Batch 000/188 | Cost: 0.4254\n",
      "Epoch: 020/040 | Batch 150/188 | Cost: 0.4183\n",
      "Epoch: 020/040\n",
      "Train ACC: 85.73 | Validation ACC: 72.55\n",
      "Time elapsed: 4.25 min\n",
      "Epoch: 021/040 | Batch 000/188 | Cost: 0.5254\n",
      "Epoch: 021/040 | Batch 150/188 | Cost: 0.4328\n",
      "Epoch: 021/040\n",
      "Train ACC: 85.22 | Validation ACC: 72.25\n",
      "Time elapsed: 4.46 min\n",
      "Epoch: 022/040 | Batch 000/188 | Cost: 0.4798\n",
      "Epoch: 022/040 | Batch 150/188 | Cost: 0.4075\n",
      "Epoch: 022/040\n",
      "Train ACC: 88.92 | Validation ACC: 73.90\n",
      "Time elapsed: 4.68 min\n",
      "Epoch: 023/040 | Batch 000/188 | Cost: 0.2946\n",
      "Epoch: 023/040 | Batch 150/188 | Cost: 0.3808\n",
      "Epoch: 023/040\n",
      "Train ACC: 89.33 | Validation ACC: 73.80\n",
      "Time elapsed: 4.88 min\n",
      "Epoch: 024/040 | Batch 000/188 | Cost: 0.2511\n",
      "Epoch: 024/040 | Batch 150/188 | Cost: 0.3758\n",
      "Epoch: 024/040\n",
      "Train ACC: 89.94 | Validation ACC: 74.20\n",
      "Time elapsed: 5.10 min\n",
      "Epoch: 025/040 | Batch 000/188 | Cost: 0.2348\n",
      "Epoch: 025/040 | Batch 150/188 | Cost: 0.4043\n",
      "Epoch: 025/040\n",
      "Train ACC: 90.37 | Validation ACC: 74.10\n",
      "Time elapsed: 5.31 min\n",
      "Epoch: 026/040 | Batch 000/188 | Cost: 0.2663\n",
      "Epoch: 026/040 | Batch 150/188 | Cost: 0.2651\n",
      "Epoch: 026/040\n",
      "Train ACC: 91.69 | Validation ACC: 72.55\n",
      "Time elapsed: 5.52 min\n",
      "Epoch: 027/040 | Batch 000/188 | Cost: 0.2907\n",
      "Epoch: 027/040 | Batch 150/188 | Cost: 0.2981\n",
      "Epoch: 027/040\n",
      "Train ACC: 92.33 | Validation ACC: 73.10\n",
      "Time elapsed: 5.74 min\n",
      "Epoch: 028/040 | Batch 000/188 | Cost: 0.2318\n",
      "Epoch: 028/040 | Batch 150/188 | Cost: 0.2904\n",
      "Epoch: 028/040\n",
      "Train ACC: 91.91 | Validation ACC: 72.10\n",
      "Time elapsed: 5.95 min\n",
      "Epoch: 029/040 | Batch 000/188 | Cost: 0.1949\n",
      "Epoch: 029/040 | Batch 150/188 | Cost: 0.1721\n",
      "Epoch: 029/040\n",
      "Train ACC: 93.64 | Validation ACC: 73.15\n",
      "Time elapsed: 6.16 min\n",
      "Epoch: 030/040 | Batch 000/188 | Cost: 0.1504\n",
      "Epoch: 030/040 | Batch 150/188 | Cost: 0.2986\n",
      "Epoch: 030/040\n",
      "Train ACC: 94.12 | Validation ACC: 73.50\n",
      "Time elapsed: 6.37 min\n",
      "Epoch: 031/040 | Batch 000/188 | Cost: 0.1666\n",
      "Epoch: 031/040 | Batch 150/188 | Cost: 0.1380\n",
      "Epoch: 031/040\n",
      "Train ACC: 92.82 | Validation ACC: 72.90\n",
      "Time elapsed: 6.59 min\n",
      "Epoch: 032/040 | Batch 000/188 | Cost: 0.2123\n",
      "Epoch: 032/040 | Batch 150/188 | Cost: 0.2601\n",
      "Epoch: 032/040\n",
      "Train ACC: 94.51 | Validation ACC: 72.80\n",
      "Time elapsed: 6.80 min\n",
      "Epoch: 033/040 | Batch 000/188 | Cost: 0.1769\n",
      "Epoch: 033/040 | Batch 150/188 | Cost: 0.1912\n",
      "Epoch: 033/040\n",
      "Train ACC: 94.81 | Validation ACC: 72.15\n",
      "Time elapsed: 7.01 min\n",
      "Epoch: 034/040 | Batch 000/188 | Cost: 0.2098\n",
      "Epoch: 034/040 | Batch 150/188 | Cost: 0.2454\n",
      "Epoch: 034/040\n",
      "Train ACC: 95.87 | Validation ACC: 73.25\n",
      "Time elapsed: 7.22 min\n",
      "Epoch: 035/040 | Batch 000/188 | Cost: 0.1446\n",
      "Epoch: 035/040 | Batch 150/188 | Cost: 0.1103\n",
      "Epoch: 035/040\n",
      "Train ACC: 94.59 | Validation ACC: 72.05\n",
      "Time elapsed: 7.43 min\n",
      "Epoch: 036/040 | Batch 000/188 | Cost: 0.1118\n",
      "Epoch: 036/040 | Batch 150/188 | Cost: 0.1148\n",
      "Epoch: 036/040\n",
      "Train ACC: 96.36 | Validation ACC: 74.30\n",
      "Time elapsed: 7.65 min\n",
      "Epoch: 037/040 | Batch 000/188 | Cost: 0.1138\n",
      "Epoch: 037/040 | Batch 150/188 | Cost: 0.2091\n",
      "Epoch: 037/040\n",
      "Train ACC: 95.63 | Validation ACC: 73.35\n",
      "Time elapsed: 7.85 min\n",
      "Epoch: 038/040 | Batch 000/188 | Cost: 0.1720\n",
      "Epoch: 038/040 | Batch 150/188 | Cost: 0.0837\n",
      "Epoch: 038/040\n",
      "Train ACC: 95.77 | Validation ACC: 74.10\n",
      "Time elapsed: 8.07 min\n",
      "Epoch: 039/040 | Batch 000/188 | Cost: 0.1058\n",
      "Epoch: 039/040 | Batch 150/188 | Cost: 0.0731\n",
      "Epoch: 039/040\n",
      "Train ACC: 97.03 | Validation ACC: 73.55\n",
      "Time elapsed: 8.28 min\n",
      "Epoch: 040/040 | Batch 000/188 | Cost: 0.1014\n",
      "Epoch: 040/040 | Batch 150/188 | Cost: 0.1611\n",
      "Epoch: 040/040\n",
      "Train ACC: 96.68 | Validation ACC: 72.30\n",
      "Time elapsed: 8.49 min\n",
      "Total Training Time: 8.49 min\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "cost_list = []\n",
    "train_acc_list, valid_acc_list = [], []\n",
    "\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "        \n",
    "        features = features.to(DEVICE)\n",
    "        targets = targets.to(DEVICE)\n",
    "            \n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        cost = F.cross_entropy(logits, targets)\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        #################################################\n",
    "        ### CODE ONLY FOR LOGGING BEYOND THIS POINT\n",
    "        ################################################\n",
    "        cost_list.append(cost.item())\n",
    "        if not batch_idx % 150:\n",
    "            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '\n",
    "                   f'Batch {batch_idx:03d}/{len(train_loader):03d} |' \n",
    "                   f' Cost: {cost:.4f}')\n",
    "\n",
    "        \n",
    "\n",
    "    model.eval()\n",
    "    with torch.set_grad_enabled(False): # save memory during inference\n",
    "        \n",
    "        train_acc = compute_acc(model, train_loader, device=DEVICE)\n",
    "        valid_acc = compute_acc(model, valid_loader, device=DEVICE)\n",
    "        \n",
    "        print(f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d}\\n'\n",
    "              f'Train ACC: {train_acc:.2f} | Validation ACC: {valid_acc:.2f}')\n",
    "        \n",
    "        train_acc_list.append(train_acc)\n",
    "        valid_acc_list.append(valid_acc)\n",
    "        \n",
    "    elapsed = (time.time() - start_time)/60\n",
    "    print(f'Time elapsed: {elapsed:.2f} min')\n",
    "  \n",
    "elapsed = (time.time() - start_time)/60\n",
    "print(f'Total Training Time: {elapsed:.2f} min')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdd3xT1fvA8c9J0nRA2WWPsjctUDYyZAoIKg4QUQQFHD8UReXrFhcqLlRAEEFREReKguy9KXtPC5TZMlpW04zz+6Nt6EjadIQW8rxfr75I7j333qeIeXLvOec5SmuNEEII32XI7wCEEELkL0kEQgjh4yQRCCGEj5NEIIQQPk4SgRBC+DhTfgeQXaVKldKhoaH5HYYQQtxUNm/eHKu1DnG176ZLBKGhoURGRuZ3GEIIcVNRSh11t08eDQkhhI+TRCCEED5OEoEQQvi4m66PQAiRc1arlejoaBISEvI7FOElAQEBVKxYET8/P4+PkUQghA+Jjo4mODiY0NBQlFL5HY7IY1przp07R3R0NFWrVvX4OHk0JIQPSUhIoGTJkpIEblFKKUqWLJntOz5JBEL4GEkCt7ac/Pf1qUSweM8ZTly8lt9hCCFEgeIziWD78Ys89n0kbcYuze9QhPBpSikGDhzofG+z2QgJCaFXr14AzJkzh7Fjx2Z6jpMnT3LvvfcCMH36dJ5++ulsxfDee+9l2WbQoEH89ttv2TpvTmzbto158+Z5/TqZ8ZlEsO7IufwOQQgBFCpUiF27dnHtWtLd+aJFi6hQoYJzf+/evRk9enSm5yhfvnyuPqQ9SQQ3iiSCG2h4++r5HYIQItkdd9zB3LlzAZg5cyb9+/d37kv9DX/QoEGMGDGC1q1bU61aNeeHf1RUFA0aNHAec/z4cbp3707t2rV56623nNvvuusumjZtSv369Zk8eTIAo0eP5tq1a4SHhzNgwAAAvv/+exo1akRYWFiau5WVK1dmuHZ6ro49evQonTp1olGjRnTq1Iljx44B8Ouvv9KgQQPCwsJo164diYmJvP7668yaNYvw8HBmzZqVu7/YHPKp4aPd65dl/u7T+R2GEAXCW3/vZs/J+Dw9Z73yRXjjzvpZtuvXrx9jxoyhV69e7Nixg8GDB7Nq1SqXbU+dOsXq1avZt28fvXv3dj4SSm3jxo3s2rWLoKAgmjVrRs+ePYmIiODbb7+lRIkSXLt2jWbNmtG3b1/Gjh3Ll19+ybZt2wDYvXs37777LmvWrKFUqVKcP3/e42u7O/bpp5/m4Ycf5pFHHuHbb79lxIgR/Pnnn4wZM4YFCxZQoUIFLl68iNlsZsyYMURGRvLll196/Pec13zmjgCgTrlgAOwOWadZiPzUqFEjoqKimDlzJj169Mi07V133YXBYKBevXqcOXPGZZsuXbpQsmRJAgMDueeee1i9ejUA48ePJywsjJYtW3L8+HEOHjyY4dilS5dy7733UqpUKQBKlCjh8bXdHbtu3ToefPBBAAYOHOiMp02bNgwaNIgpU6Zgt9sz/b1vJJ+6IwjwMwJgsdkJMvvUry5EBp58c/em3r17M2rUKJYvX865c+778Pz9/Z2vtXb9JS79kEmlFMuXL2fx4sWsW7eOoKAgOnTo4HJ8vdba7ZDLrK6d2bGu4ps0aRIbNmxg7ty5hIeHO+9K8ptP3RH4m5J+XYvVkc+RCCEGDx7M66+/TsOGDXN9rkWLFnH+/HmuXbvGn3/+SZs2bYiLi6N48eIEBQWxb98+1q9f72zv5+eH1WoFoFOnTvzyyy/OZJT60VBW3B3bunVrfv75ZwB+/PFH2rZtC8Dhw4dp0aIFY8aMoVSpUhw/fpzg4GAuXbqU67+D3PCpRJByR5BgKzi3ZEL4qooVK/LMM8/kybnatm3LwIEDCQ8Pp2/fvkRERNC9e3dsNhuNGjXitddeo2XLls72Q4cOpVGjRgwYMID69evzyiuv0L59e8LCwnjuuec8vq67Y8ePH8+0adNo1KgRM2bM4PPPPwfghRdeoGHDhjRo0IB27doRFhZGx44d2bNnT752Fit3t1oFVUREhM7pwjR/bInmuV+2s2xUB6qWKpTHkQlR8O3du5e6devmdxjCy1z9d1ZKbdZaR7hq71N3BObkR0OJNnk0JIQQKXwqEfibkh4NXbPKoyEhhEjhU4kg5Y7grq/W5HMkQghRcPhWIjD61K8rhBAe8alPxpQ7AiGEENf51Cejf6pEcC1R+gmEEAJ8LBGkviMY9dv2fIxECN9lNBoJDw+nQYMG3HnnnVy8eNEr12ndurVXznsr8qlE4Jeqj2DXibh8jEQI3xUYGMi2bdvYtWsXJUqU4KuvvvLKddauXeuV8+aFglRnCHwsEZgMskSfEAVJq1atOHHiBADLly93Lk4DSRU8p0+fDkBoaChvvPEGTZo0oWHDhuzbtw+AN998k8GDB9OhQweqVavG+PHjnccXLlzYed4OHTpw7733UqdOHQYMGOCsGzRv3jzq1KlD27ZtGTFiRJrrp4iKiuK2226jSZMmNGnSxJlgHnjggTTrCAwaNIjff/8du93OCy+8QLNmzWjUqBFff/21M46OHTvy4IMPOstquCqTDTB16lRq1apFhw4dePzxx51luWNiYujbty/NmjWjWbNmrFmTNyMgfaryWuo7AkkJwuf9OxpO78zbc5ZtCHdkvrpYCrvdzpIlSxgyZIhH7UuVKsWWLVuYMGEC48aN45tvvgFg3759LFu2jEuXLlG7dm2eeOIJ/Pz80hy7detWdu/eTfny5WnTpg1r1qwhIiKCYcOGsXLlSqpWrZpmTYTUSpcuzaJFiwgICODgwYP079+fyMhI+vXrx6xZs+jRoweJiYksWbKEiRMnMnXqVIoWLcqmTZuwWCy0adOGrl27AtfLZVetWhXAZZlsi8XC22+/zZYtWwgODub2228nLCwMgGeeeYaRI0fStm1bjh07Rrdu3di7d69Hf3+Z8alEYDJe//iXBbyFyB8pi8JERUXRtGlTunTp4tFx99xzDwBNmzbljz/+cG7v2bMn/v7++Pv7U7p0ac6cOUPFihXTHNu8eXPntpRrFy5cmGrVqjk/lPv375/mW3kKq9XK008/zbZt2zAajRw4cABIWlxnxIgRWCwW5s+fT7t27QgMDGThwoXs2LHDuZBNXFwcBw8exGw207x5c+f1IKkm0ezZswGcZbJPnz5N+/btnSWt77vvPuc1Fy9ezJ49e5zHx8fHc+nSJYKDgz36O3THpxJBiSBzfocgRMHh4Tf3vJbSRxAXF0evXr346quvGDFiBCaTCYfjevmX9CWjU0pCG41GbDZbhu2u9mXWxtM6a59++illypRh+/btOBwOAgICAAgICKBDhw4sWLCAWbNmOe8otNZ88cUXdOvWLc15li9fTqFChdK8d1UmO7O4HA4H69atIzAw0KPYPeVTfQSGVH0E/8VeycdIhBBFixZl/PjxjBs3DqvVSpUqVdizZw8Wi4W4uDiWLFni1evXqVOHI0eOEBUVBeC28mdcXBzlypXDYDAwY8aMNB29/fr1Y9q0aaxatcr5wd+tWzcmTpzoLHN94MABrlzJ+Hnjrkx28+bNWbFiBRcuXMBms/H77787j+natWualczyaj0Dn0oEQoiCpXHjxoSFhfHzzz9TqVIl7r//fmd56MaNG3v12oGBgUyYMIHu3bvTtm1bypQpQ9GiRTO0e/LJJ/nuu+9o2bIlBw4cSPOtvmvXrqxcuZLOnTtjNic9cXjssceoV68eTZo0oUGDBgwbNszlXYq7MtkVKlTg5ZdfpkWLFnTu3Jl69eo54xo/fjyRkZE0atSIevXqMWnSpDz5u/CpMtQAoaPnOl9Hje2ZFyEJcdOQMtRpXb58mcKFC6O15qmnnqJmzZqMHDkyv8NyxmWz2bj77rsZPHgwd999t8fHF5gy1EqpSkqpZUqpvUqp3UqpDCtQqCTjlVKHlFI7lFJNvBWPEEKkN2XKFMLDw6lfvz5xcXEMGzYsv0MCkobFpky6q1q1KnfddZdXr+fNzmIb8LzWeotSKhjYrJRapLXek6rNHUDN5J8WwMTkP2+IyKjzRISWyLqhEOKWNHLkyAJxB5DeuHHjbuj1vHZHoLU+pbXekvz6ErAXqJCuWR/ge51kPVBMKVXOWzGld++kdTfqUkIUGDfb42CRPTn573tDOouVUqFAY2BDul0VgOOp3keTMVmglBqqlIpUSkXGxMR4K0whbnkBAQGcO3dOksEtSmvNuXPnnENcPeX1eQRKqcLA78CzWuv49LtdHJLhX6jWejIwGZI6i3MTT98mFfl9S3RuTiHETatixYpER0cjX6huXQEBARkm1GXFq4lAKeVHUhL4UWv9h4sm0UClVO8rAie9GVOlEnk7EUOIm4mfn1+ama1CgHdHDSlgKrBXa/2Jm2ZzgIeTRw+1BOK01qe8FRNAkNnozdMLIcRNx5t3BG2AgcBOpVTK9LeXgcoAWutJwDygB3AIuAo86sV4hBBCuOC1RKC1Xk0WRT51Uo/VU96KwfU1b+TVhBCi4PP5EhObj57P7xCEECJf+VYicDioWzZtuda+E9ex/si5fApICCHyn+8kgj1z4P2KtCt9LcOu03EJLg4QQgjf4DuJoHBpsF6BmH1UK1UozS6DLGEphPBhvpMIQmon/Xl2b5qVygCMslqZEMKH+U4iCCwOhctCzH4GtgpNs+tUXMbHRUII4St8JxEAlK4DMXtpWrl4ms3vzN0ryUAI4bN8KxGE1IWY/dQtWyjDrlbvL82HgIQQIv/5ViIoXQesV1Fxx7NuK4QQPsK3EkFInaQ/Y/YztF21/I1FCCEKCN9MBGf3UqVkUP7GIoQQBYRvJYLAYhBcDmL24XC4Ljp0JOYye0+lXzZBCCFuXV5fmKbACakDZ/eSGJIxEUS8s4jYy4kARI3teaMjE0KIfOFbdwQApetC7AFsNluGXSlJQAghfInvJYJyYWC9ysDqV/M7EiGEKBB8LxFUaQ1A0MkN+RyIEEIUDL6XCIpVhqKV4Ojq/I5ECCEKBN9LBJB0V3B0LZC95cocDs0bf+3iv9gr3olLCCHygW8mgqrt4UoM95ZzvyDNygMx2NMNMT1w9hLfrTvK8BmbvR2hEELcML6ZCGp1AxTjws+4bfLwtxuZvPJImm2y3rEQ4lbkm4mgUKmk0UOHl2Xa7Nh514+AZPkCIcStxDcTAUD12yF6I4VxP4zU7tD8suk4787dcwMDE0KIG8u3E4HDRiuD+w/5XyKjefH3HUxZ9R8gj4aEELcm300ElVqAfxG6GzfldyRCCJGvfDcRmMxQ/y7u9t9MEAn5HY0QQuQb300EAGEPYrBd5Q7Dxiybho6ey1t/774BQQkhxI3l24mgcksoXpW+xpUeNd/w33kAlAwbEkLcQnw7ESgFjQfQ2riHOupYfkcjhBD5wrcTAUCzx4jXgYww/ZHfkQghRL6QRBBYnGn2O+hh3EhDdSTr9kDKg6EZ66LYf/qS10ITQogbQRIBMNV2BzG6CC+YZmXruNf+2k23zzzrXxBCiIJKEgHw7fDOTLd1p51xJ9XViSzbS1+xEOJWIokAaFy5ODPtt2PRfgwxzsv28QlWuxeiEkKIGyPLRKCUMt6IQPLbeYrwm70dDxiXUw735ald+XqFZ30LQghREHlyR3BIKfWRUqqe16PJZ9/Ye2BUmqdMf2babvfJ+DTvLTa5IxBC3Lw8SQSNgAPAN0qp9UqpoUqpIlkdpJT6Vil1Vim1y83+DkqpOKXUtuSf17MZe54xKChZyMywu7syy9aBe4yrKcJlj4//a9tJL0YnhBDelWUi0Fpf0lpP0Vq3Bl4E3gBOKaW+U0rVyOTQ6UD3LE6/SmsdnvwzxuOo85hSis2vdaFf88p8Z+9KkLJwn3FFpsek7hc4cfGat0MUQgiv8aiPQCnVWyk1G/gc+BioBvwNuO1Z1VqvBM7nVaA3yumgWmxw1OER40IMONy2q/Pa/DTv465avR2aEEJ4hSePhg4CfYCPtNaNtdafaK3PaK1/A+ZncWxWWimltiul/lVK1XfXKPlxVKRSKjImJiaXl8ycAqbbulHZEMPthq0eH/fevL3eC0oIIbzIoz4CrfUQrfXa9Du01iNyce0tQBWtdRjwBeC2h1ZrPVlrHaG1jggJCcnFJbOmFCx0RHBSl+AR4wKPj5sVedyLUQkhhPd4kghKK6X+VkrFJnf+/qWUqpbbC2ut47XWl5NfzwP8lFKlcnve3Brarhp2jPxg68Jtxl1EqH35HZIQQniVJ4ngJ+AXoCxQHvgVmJnbCyulyqrkes5KqebJsWRvAL8XDG1XnaixPZlp7wjAMNNcj4/VspalEOIm5EkiUFrrGVprW/LPD0CWn3hKqZnAOqC2UipaKTVEKTVcKTU8ucm9wC6l1HZgPNBPF6BP0gsU4SPr/XQxbiZMHfLomKr/m8eFK4lejkwIIfKWJ4lgmVJqtFIqVClVRSn1IjBXKVVCKVXC3UFa6/5a63Jaaz+tdUWt9VSt9SSt9aTk/V9qretrrcO01i1d9UHkp1d71mW6vRvndDDPm371+Lhpa6O8F5QQQniByYM2DyT/OSzd9sEk3Rnkur+gIKpYPIgrBDLR1ptX/X6kuW0vG3XdLI8zJBeku3AlkeKFzF6OUgghcs+TCWVVM/m5JZMAXK8w+oO9M2d0MZ73+xUPnoihUGw+ep7Gby9i3s5T3g1SCCHygCcTyvyUUiOUUr8l/zytlPK7EcHlp5RK0wn486XtLloY9tHW4LJaRhqrD8Xw+PebAVh3ON/7voUQIkue9BFMBJoCE5J/miZvu6WlXqB+Y7FeROtSjDLNIqu7gk1RFzif3GE8Y/1RQkd7PupICCHygyeJoJnW+hGt9dLkn0eBZt4OLL+1rVGKBhWK8Haf+ix4oQvTjX0JNxzJ1mxjIYS4GXiSCOxKqeopb5Ink93ydZcDzUb++b/bGNgqFIA5dOS0Ls5jOVi45rLFRrN3F7PhiDwqEkIUPJ4kghdIGkK6XCm1AlgKPO/dsAoeizYyxdaD1sY9hHs4ryDFrhNxxFyy8PHCA9gdmoFTN0j/gRCiwMg0ESilDMA1oCYwIvmnttZ62Q2IrUAZ06c+C/y7E6+DeMtvOkWzsV7B2UsWAGKvWDh32cKqg7H830x5xCSEKBgyTQRaawfwsdbaorXeobXerrW23KDYCpQ+4RVY/XpvRlmHUUcdY5zfJDwZTgowIvlD/0jMFXadjAMg9rKFQ2cveStcIYTwmCePhhYqpfqq1MNofNhCRzM+sPWni3ELDxqXZvv41KuZdf5kJfN3nWbz0Qt5GaIQQmSLJ4ngOZIKzVmUUvFKqUtKqfisDrqVTbN3Y6W9Ia+ZZlBNZW+ZyvTLWg7/YTN9J67l2LmreRmiEEJ4zJOZxcFaa4PW2qy1LpL8Pss1i29lGgOjrMOx4McM8/v4Ycv1OY/Eet7nIIQQecmTmcVLPNnma85SnHG2+6mgzjEkB0NK0xs0bVMeRJXWD+uPMn+XlLkQQmTObSJQSgUkVxctpZQqnlJtVCkVStK6BD5pSNuqztc/2DuzyN6EkabfqajO5mNUrr365y6G/7Alv8MQQhRwmd0RDAM2A3WS/0z5+Qv4yvuhFUwju9RK9U7xmvVRrBh52zQNT0cRCSFEQeI2EWitP9daVwVGaa2rpao4Gqa1/vIGxligpF875zQl+cR2Hx2N2+luyPvHO0II4W2edBZ/oZRqrZR6UCn1cMrPjQiuIHK4+NL/nb0rux1VmGT+jN6GNXlyndjLFmasP5on5xJCiMx40lk8AxgHtCWp2FwzIMLLcRVYrlbTtGPkWetTXNYBjDd/xUjTbzk6t9XuYO2hWK5YbIyYuZXX/twlk86EEF7nyQplEUC9grSecH4K8DMC0KNhWebtPO3cflBXpLllAov8X+AZ0x8ssjdhVzbX7an5yr8ZtllsjtwFLIQQWfBkQtkuoKy3A7lZBPgZ2f9Od17ukXHZyqsE0MfyDid0SX4yv0tLw55cX0/SrxDC2zxJBKWAPUqpBUqpOSk/3g6sIPM3GSlZyB+APuFpR9LGUpR7LW9yVhfnZ/M73G7I3fDN2MsWriYmTVjTWvPt6v/YfPQCb87Zzem4hFydWwghwLNHQ296O4ibUaDZSNTYngDsP32JfaevP8s/RUn6J77CpoCn+NY8jl9s7fnIdj8xFM/2dQZN20SpwmYiX+3CrhPxjPnn+l3G9uiLzH6yDTuiL1KvXBFMRk/yuhBCpJXZhLI6AFrrFcB6rfWKlB/AJyuQulPIP2M+jaE44Qlfs9FRm/tNK1jk/yJ11LEcnT/2ctLSl4n2tP0FlxNs7D99id5fruHDBfsBmLXpGKsPxuboOkII35TZV8ifUr1el27fBC/EctNyV5b1IsHcn/gGXS0fkICZ78xjqZ3DZLAp6jyuJqzFXk7KyTujk8pbv/T7Th6auiFH1xBC+KbMEoFy89rVe5/2RIfqad73alSOAL/rf7UHdCUeSvwfAL+Z3yIsmyucAdw3aR19J6bNx5rr/yHWHTnHrhNxbo+ftekYk1YczvZ1hRC3vswSgXbz2tV7n9apbhlqlSnsfP/J/eHULhOcps0hXZF7LG9xXgczwzyWdobtub6u1jpNSu71xWq3bV/6fSdj/92X62sKIW49mSWCikqp8UqpL1K9Tnlf4QbFd1MymwwUDTJn2H6CEAZZX8KCH1P8PqaGis7VdZLuCLK+OTsSIyWuhRDuZZYIXiCpyFxkqtcp71/0fmg3l0/uD6dTndIcfPcOAOqWC3bZ7j9djnsS38QPO4v9X2SQcT45vcE6EnMF7eZYW6qOZav9ehuZqSyESE/dbBOGIyIidGRkZH6HkaUrFhv131jgdn9TtZ+fze/gp+ysd9Slf+IraI+mdXimTtlg55DWBc+2o9tnKwGYNbQlLaqVzLPrCCFuDkqpzVprl+WBZOC5lxTyN7HihQ5u92/Wtalp+Z7f7O1oadjLa6Yf8Ccxz66fel7D6D92uGyTYLU7Rx0JIXyXJAIvqlAsMIsWilHWYfxg68Rg03z2BwziOdMveR7H1mMXna8Px1xxvn546kYi3lkMwL87T7Foz5k8v7YQouCTROBFJqOB7W90zaKV4lXbYIYkPs8RR1lGmP5ktvl1PvabSBHyvpP35dk7na83Rp13vn7ixy08/n3Bf+QmhMh7npSh/lApVUQp5aeUWqKUilVKPXQjgrsVFA3086CVYomjKXckjmW7oxqNDYfoa1zFKv9nKU58nscUc0keBwkhrvPkjqCr1joe6AVEA7VIGkUkPDRxQBOP2lkw0yfxHSISJvKKdTCBWJhk/gwz1jyNp9m7i3njr13O93tP5X2yEULcPDxJBClfaXsAM7XW5zNrnEIp9a1S6qxSapeb/Sp5XsIhpdQOpZRnn5Y3oTsalstW+1iK8qO9M89bn6CFYR9/mN/AQN6uS/Dduuurn93x+SqPjlm2/ywfLdjHlmMX8jQWIUT+8iQR/K2U2kfSAjVLlFIhgCf1j6cD3TPZfwdQM/lnKDDRg3Pe1AqZjYzpUx+AN++sl2X7vx2t+dD6AA0MUUz0+yzPk4Erp+KucejsJa4l2rmUkPZO5NFpm/hq2WHumbDW63EIIW6cLMtQa61HK6U+AOK11nal1BWgjwfHrVRKhWbSpA/wffLKZ+uVUsWUUuW01qc8jP2mMumhptQvX4RKJYJ4uFUo248njeRpVLEoO6Ld1wiaYO9NOXWOgabFvK2n8Y29B5d0ELEU9Uqcrd5fCkBIsD8xlyzOUttCiFuXJ53F9wG25CTwKvADUD6LwzxRATie6n00bkpXKKWGKqUilVKRMTExeXDpG697g7JUKhGUgyMVr9ke5UdbJwaYlrDM/3kiA55gtOknvFnyKasO5WPnrjoXzBFC3Nw8eTT0mtb6klKqLdAN+I68eYzjqkiOy082rfVkrXWE1joiJCQkDy5dsMwY0jyLFopXbEPoaXmPL2192OGoynDTP6z3f5rRpp9oZdh9Q+JMrd1Hy6j3uvuZ00KIm4cnicCe/GdPYKLW+i8gY0W17IsGKqV6XxE4mQfnvSnUKF2YQmYjz3WpxW01Q1j8XLsMbdIvg7lbhzLO9gB3J47hJ1tHinOZ4aZ/mGl+l6iAB3nCOCfPh5vaHZovlhwkPiFvRy4JIQoOTxLBCaXU18D9wDyllL+Hx2VlDvBw8uihlkDcrdo/4EohfxO7x3SnQ+3SANQoHczh93qkaWMyuP5rtmPkZdvjtLaMp1HCZJbYGwPwkt/PLPUfRQgXXR6XE9+tjeLjRQfoP3m9R+1/3xyd6boIQoiCx5MP9PuBBUB3rfVFoAQezCNQSs0kaWWz2kqpaKXUEKXUcKXU8OQm84AjwCFgCvBkTn6BW4nRoFg+qgP1yxdxbrszzH13zDmKEk9hhlhfoGrCDzya+ALF1WU2BTxJdXUiT2JKWSN590nP7jSe/3U7vb5YzVfLsr/4jhAif2SZCLTWV4HDQDel1NNAaa31Qg+O66+1Lqe19tNaV9RaT9VaT9JaT0rer7XWT2mtq2utG2qtpb4BEFqqEINahzrff3xfmEfHaQwsczTm2cSkfLrE/wWaqAPeCNEjHyWvoSyEKPg8GTX0DPAjUDr55wel1P95OzABGo3ZlL2ncH862tLL8g4XdGH+8H+TgcYsc3auRMVeYf6u05m22RR1njfn7Gbg1A04HDdX2XMhfEGW8wiAIUALrfUVgOQ5BeuAL7wZmC9TKnlAlU55D9lZNmKXrkZXywdsCniKt/2mU0GdY6k9nI26Dnm93PTtHy/HoaF3WHkGtKjsss19k66vtZxgsxNkTvvP7lKCFZtdU7xQXoxBEEJklydfNxXXRw6R/FoWr/ei9H+5EVWKA/Bqz7qEBPsztF21LM8RQ3E6Wz5ks6Mmw01/84v/2ywzP0dZzuVprClf8OdsP8kD6TqU95/OuBrahiNJFUrOX0kkdPRcft54jFbvL6Xx24vyNC4hhOc8SQTTgA1KqTeVUm8C64GpXo3Kx5VI/mZctmgAcH1d4oYVirLplc683KMur/asm+V5DumK9E18i9ssnwJQ1XCG9QH/R5XfVagAACAASURBVFvDziyOzBuu1kp+dPomDp65RPSFqwD8uOEYly0yMU2I/ORJZ/EnwKPAeeAC8KjW+jNvB+bLOtQO4asHm/Bs51pptqd+OvTYbVnfFaQ4rssQmvATDya+DMAP5vd5zDgXb85MzsyFq1YMyY+/bNJnIES+yzQRKKUMSqldWustWuvxWuvPtdZbb1RwvkopRc9G5ZwdxW6mE2TbWkcDwhO+5qijNK/6/UhUwAD+NL+KCe98I3e34plDa+djI+k8FiL/ZfoRo7V2ANuVUq57AcUNMe6+MB5uVYVmoSXSbH8gopKbI9y7SDC9E99hqT0cgHDDET73+5JAjwrKZs8/O13PD+w3eT3P/7odAHuqXvBNUec5dPYSibaMVVYtNjuho+fy3dqoPI9TCF/nyXfNcsDu5NXJ5qT8eDswcV3F4kGM6dMAoyFtN/JbySWtsyuOwgy2vkhowk98au1LT+NG9gYMpqthU16E62RUWY8pSH1HcN+kdXT+ZCVj/slYOyn+WtJdyxdLD+ZdgEIIwLPho295PQqRIwF+Rl7tWZd35u7N8Tk+t9/DZQJ4ze9HJps/5bCjHNt1dTRw2FGeyfZe2Dz6Z5KRwYOxZQ4X42I3/Zd24Rub3cGV5A7l7AyjFUJ4xu3/4UqpGkAZrfWKdNvbAXlTv0Dk2mO3VaNs0QB+3nicHx5rwcWriYSPyc5QTMVUe0/+tLflGdMfPGxaRHWSH+kYYZTpV76292KKrSfnKZL5qdK5kmjPupEL564klcD+culBZm48ToMKRViw+0zyvsQcnVMI4V5mj4Y+AzIOBIeryftEAdGrUXl+eKwFAMWCzAxuU9W5L/3jJHfOUZTXbY8SmvATNRO+JzThR36zt8OgNE+Y/mZLwHC+9fuQiip760GsPRybrfYAsZeTPuzHLTzAiYvXnElACOEdmSWCUK31jvQbk2sChXotIpFrqR/N1ykb7HzdpV4Zj463YgIUo6zDaZYwga9tPTnkKM/txm2s9n+G0aafMONZWeoHp2zIdH/UuasenceVu75aQ+jouWz8z6NltIUQbmSWCAIy2ReY14GIvHN3Y5cLvdGrUblsnyuGYrxvG0DnxHEMSxxJvA5kuOkfFptHEUzOP8SzsvZQ1ncS25KX+7z/63VZtBRCZCazRLBJKfV4+o1KqSHAZu+FJHKrQYWirP9fJwDa17q+oluf8Ar8OrxVjs+7wNGMMMsUfrB1orIhhjf9vsNbk9K2Z7KOsytxV60kWHPWJyGEr1PazTAMpVQZYDaQyPUP/giSVie7W2udeclJL4mIiNCRkVKx2hOn4q5ROjiA6i/PA3AuRH/i4jXajF2aq3OPNP3KM6bZ7HVUZkjiKE5SKtfxeirl9wgdPTfN9lplCrNwZPsbFocQNxOl1GatdYSrfW5HDWmtzwCtlVIdgQbJm+dqrXP3CSJumHJFXT/Bq1Ds+vZ+zSrx86bj2T73p7Z7KcZlHjEtYpb5bX6zt+M/XY5VjgZcyOboouzSWl+v0JrKgTOXsdjsGJXCZMyj6dhC+ABPag0t01p/kfwjSeAWExLsn8MjFW/YHuV+y2sUU5cZ6fc7481fEun/BPPM/6MIV/I0ztSq/m8esZctLvfVfnU+D03NvIM6tUNnL/HZ4vxbwEeIgkC+NvmAF7vXZvaTrV3us6ea2du8agmXbTKzUdcl3DKZZglfMco6jDWOBtQzHGWF/0g6GzajyFguIi9sPnrB7b71R66PIlp7OJbQ0XO54Gb+QedPVvLZ4oOEjp5L6Oi5/LThWJ7HKkRBJ4nABzzZoQaNKxd3uW9Y++rO1/c2rZij89sxEkNxfrO352HraL609cGGgW/MH7PT/zG+8xtLa8OuHJ3bnWEzsh6voLV2Dl/dHn3Ro/O+PPvGlOgWoiCRROCjvhvcnG8ejqBooB///F9bXulRl16NyhFWsajbY3o2LEfPhlkNQVWMsz1AR8sn/GDrxAZHXaobTjLD730eN/7DjSx9/W6q0htXE+38EnmchbvzZYyDEAVazorIiJte6mGlDSoUpUGFpATw19Nt6fPVGrYfz/gNemSXmhQPMjPXTVXR1C4TxKu2IQAEkcA4v0m84vcT9xlXMNL6JLt11SzOkHv/plpL+ckftzhfb3u9C8WCPF8Wc+uxC9w9YS2Ln2tPjdKF8zRGIQoCuSMQGbgbUlyjdDCBZiMAAX6e/9O5SgBPWp/hZ1sHahlOMNf/FV41zaCvYSUhuH/Wn1snLl5zuf21vzJWN83M3RPWAjBy1rZcxyREQSSJQGSQ0oH811Nt6B1WPs2+ILOJ9f/rxM43u2XzrIrRtqG0SficLY4aPGb6l4/Nk9gU8BTvmqYy0vSrx2UrPJF+jkFqf28/yW0fLs32BDTt5rGW1ppdJ7I3AU6IgkQSgcggZc3kILOR57rUyrC/bNEA/NKN08+67yDJCUK4J3EMEQkT+dqWNDFsgGkJz5hmcyDgETb5D6eaOpnL3yBrx89fY8ux7N2NuCuB/ee2E/T6YjXzd2X9yEyIgkgSgcjgswfCeffuBtQsE0z55MlnfsbMq5g+0Cx7q6XFUpT3bQMITfiJ+glTecE6lB2OqoSoeJb6j6KNwfujd9wVxPtxw9E0w2pT2OzXt607fI5JKw4DcOjs5TR/CnGzkc5ikUHJwv4MaFEFuF7J1JsLwlwhkF/tHfjV3oEm6gCf+E3ke7+xnKIk2xw1WGiPYL6jGYn4eS+IVF6ZvQurzcFdjSswPdXSmPvPXK/K3n/KegCGt6+OIukvSZZfFjcrSQQiUyn3Aa5WEitV2OxcOyCvbNG1uDPxXcb6TaancSMVjbH0Mq7nkg5kj67CsMSRXCQ46xPl0ul4i8cL/BhuQLIUwpvk0ZDIlNGgqBZSiE/uD8+wL/LVLgAUDfQjq+WJb6tZinJFM6tsft0lgnjK+iyhCT9SO2E6AxNHs95RlxaGfazxH8EG/yf5P+MftDTsyfbv46nl+8+63O5w8bU/pe6Rq2SZYubGY1LKQhRYckcgMqWUYunzHdzuX/lCR4IDTOw6eX3UTJ/w8vy1LW2Hb/cGZRnQogpbj13gssXGwKkbPbk6FsyscjRilaMRd9rX0sG4jR6GjTzv9xsOrfjL0ZqPrA/kefXTfaddLc4HsyKPp1nvwWKzE30haZjqpBWHGemicx3gf38k9Xk829n1fiHyk9wRiFypXDKI4oXSTs76vF9j4ProI4BOdZJWR2tcuTjVQ3I2KetvR2uetz5JXcs0GiVMYaEjgruNa1gbMIKXTT/m8DfInv/9sZNE+/X6Sc//sp3ft0QDYLE5iLuad0NghbhR5I5A5ImUDtM2NUoCsPW1LgT4GZ0T0FIrXyy3C9wp4inEcOuz9LRv4E2/7xhqmksnwxb26iq8bB1MPN6bAZz6CdA/O9IOGR06I5JZw3K++I8Q+UHuCIRXFC9kdpkEXAkyG3NYukEx19GSFpav+NPeGn9lpZdxPZv9n+At0zTuMy73SjnsyCj3ayRvPeZZcTshChK5IxD5Ys+YbtR7fQEA217vyqUEK03fWUyXemXo16wSQ77zfBU6BwaetT4NaHob1vGu31QeMSWN+Bltmslse1v+srehiLrCVR3AVl0zV7HvPhnvdl/qx0bnryTS5O1FBPp5lhCFyC+SCESe8nQIZZDZRJki/pyJt2A2GShZ2J/fhreibrkiFPLP6T9LxRxHa+ZYWlOOczxkWkR3wyYeM/3LY6Z/na0+tt7LJHtvrDn85//JIs9G/zR5OykZXctmKQuHQ3P8wlWqlCyU7diEyAmvPhpSSnVXSu1XSh1SSo12sX+QUipGKbUt+ecxb8YjvCer4aOu/P10W2YNbel8HxFawm0SSFmn2FOnKMlHtn50ShxHe8snvGIdzBvWRziri/G832/8YX6dJso7wzm/WHKQ+IScdxpPXnWE9h8tZ++peGasP8qfW0/kYXRpzdp0jJNuivMJ3+G1OwKllBH4CugCRAOblFJztNbpB3/P0lo/7a04xI1Rq0zSJK+UGcmeKF0kgNJFXM8tKFHIzHk3q4plj+KoLstRe1kAvrN3pZdhPW/4fccf/m8SrwOZZ2/BQkcEGxx1uUJuO7Lh40UH+DiLu4aFu08zbuF+Dpy5zL63u/Ptmv/oE16BCsUCiYxKqoF09NxVXvszaUGfu1INWc2K3aE5HZ+QZm1qV+KuWXnp951UCymU6RBhcevz5h1Bc+CQ1vqI1joR+Bno48XriXwUEuxP1Nie9GzkWfG5rLSsdn3ZzNrJSWbVix3z4MyKfxytaG/5jHetD7LBUZd+puV8ax7HOv//43bDlqxPkQeGztjMgTNJtYle/mMnH87fz2PJ/SIpM5VHzNzqbJ/ZcpvpjVu4nzZjl2b5TT+l3Pi5PJ4dLm4+3kwEFYDjqd5HJ29Lr69SaodS6jelVPYql4lb1kf3hjlfVy4ZBEClEkE81jbjgjZj+tTn/26vka3zXyWAKfZePG4dRSfLRzyY+DLndTAT/T7jA9NkuhgiuduwivoqKle/R3quZib/kfzo51Ly4yRD8nO21B3PAL9EHscTqw7GABB72ZJpOymJIVJ4s7PY1VPj9P/0/gZmaq0tSqnhwHfA7RlOpNRQYChA5cqV8zpOUQCl7iswpPqX1K5WCN+s/g+Aw+/14NDZy9Qum3TH0KF2CH0nrsv2tQ7rChzWFXjE+hLvmqbygGk5D7DcuT/SUYsvbX3Y4qhFPLnrwE20O1i484zLfdEXrnH+SiLz3SynaUuXROwOjUPrDCXBU+Z0ePpBn75/51KCFZPB4PHwX3Hz82YiiAZSf8OvCKSpO6C1Ppfq7RTgA1cn0lpPBiYDREREyPcYH6NSfadIqefTrlYIRoNyJgGAplVKZDg2O47qsjxkfYUQ60XCDYe4qAtzu3ErDxkXM938ERbtx2/2dlzFnyO6HH/Z23AVz+onpZb6kU96KSONPDF4+iZWHIjJ0JGekjj7fLXGo0729Amj4ZsLKR7kx9bXu3oci7i5efPR0CagplKqqlLKDPQD5qRuoJRK/UC5N7AXITLRpkYp+jevxNh7Grrcnxd9FDEUY5Ejgk26Dh/Y+nO75WNGWYex0tGQfsalDDH+y/t+U/nH/DLN1D788fwZe24fx/SduJbJK5PWQVhxIOkRUOv3l6RZlnN7tGerpWU20uuClMrwKV5LBFprG/A0sICkD/hftNa7lVJjlFK9k5uNUErtVkptB0YAg7wVj7g1+BkNvH9PI7dlKjrXLe322K2vdcnRNWMoxm/29jxuHUU9yzRqWGYwOHEUxdVlfvUfww7/xxhoXIjCkeW56r4+P0cxpNh89ALvzdvH8fNXndtOxiXwq5v+A7tD8/vmaGf/gxCueHUegdZ6nta6lta6utb63eRtr2ut5yS//p/Wur7WOkxr3VFrvc+b8YibU3aGTt7duCJ7xnSjeWjax0QbX+6UoTheTlgw48DAUkcTOlo+5i3rQLbpGrztN51/zf8jgMw7aHPjowX7na9v+3BZmn2fLT7o8pjhP2zm+V+3M+rX7c5tv0Qe5+mftrD5aNIw1bhrkiR8ncwsFgXWr8NbsWjPGbo3KJut44LMJh67rSobU9UESpmvUKVkEEfPXU3T/qXudZx9D6k/bLNykWCm2e9gmr07Txr/4kW/X/jD/CZTbD2Y7bgtWzHnhR3RF9mTrvxFyvsFu693UL/42w4ASgdf79/YfPQ8pQr7U8bNvA5xa5NEIAqsZqElaBaasw7grvXLEjW2J6Gj56bZvuKFjtgdmuovz3Nue6JDdQDWHIrNYaSKCfa7OKrL8ozpdz41T+QJxxw+s/VllaMRlwjK4Xmzp/eXazJsO3spwW371AvppIy2aly5WI6v3+PzVTzSugoPNJORfTcbqT4qbnnpi74ZDa57SVtXL5nmfVjFotm6zlxHS7onfsA3tjuoZTjBBPN4tvs/znumKTRRBzzqQ8hrVrv73mm7izkNuameuudUPC/9vjPHx4v8I3cE4pb2y7BWVCqRsWO5SICJ+AQbvz9xfe0AlW4YzV9Pt8Vqd9Bv8nrn8/SsODDwjm0gY239aWnYyz3GVTxgXM6DpmWstdfjV3t7ljgae3W9BHdsdgemVHMOLuZh30Ci7cYnOZF3JBGIW1rzqq4fLUW+2gWNxt+U+aQpP6OBmY+35JvVR/hw/vX+g/7NK7HyQGyaYZup2TCx2tGQ1Y6GvM1D9Dcu5WnTX7Q27sGuFQmYWeNoQGl1kZO6JEd0Ob603UUC/jn/ZbNw4uK1NBVN/95+MpPW2XPuivc6yYX3SSIQPslsyvyp6KDWoWnaPtmhRppEMKBFFV7uUZeO41bwTOeaHDt3hSmr/nN5rgsUYYL9Libae9NQ/Udn42YqqRjuMGwkQFkJJ2lewNOmv1hsb8wVApls68VuHeryfPlhxMytlChk5s3e9V3ud/GUKVOxly0s3x/DvU0r5kF0IrckEQjhgqsPPLPR4Kz/oxQEB/gR+WpnAI6fv+o2EaTQGNihq7PDltQ5/TxPUEWdIUqXobnaz0DTIloZdlNSXaKPcS3L7GF8ZuvLdp29OkruzN91mqqlPC+R8cH8fXSrX5bwSsWYk3z38GL32qw9dI4APyNta5ZytnVVQykzw2ZsZvPRC7SpUZJyRXNf8dUTp+MSeP7XbUx4sClFg/xuyDVvFpIIhPCQPZNpwZVKBDGiU03GL3E9nt8VBwb+00kzoTfoumyw1gWghormPuMK+hpXMdv8BjPsnXnPNgALuZsH8f6/2ZumM3H5YSYuP8ycp9s4tz3y7UY2JZfJfrJDdSqXCKJf88oZZkxrrXl59i4SbQ4GtqpCeKW0o5HOxCeNZrJl0pmd175adog1h87x1/YTPNwq9IZd92YgiUCIVKY92owAN/0GNUIKs//MJcB1qYgn2lfPkAiKBvple8LWIV2R920D+MJ2N2P8pvOIaRGPmBZxTgezzlGPFY4w6qsoWht2c1qXYKuuwWx7W6J03pQATy/1sNSUJAAwYXnSI60HmlVKMxQV4Ey8hZkbjwHw+5ZoZ82jLccu8OCU9c4Kq6JgkOGjQqTSsXZpWqUbRprix8dbZHps6mqdKeWym4UW56XudXIUy2WCeM76BM8nDmeBPQILfvQybuAjv8ncb1zBBYKpoGJ5xjSbheYXeck0k+K4X0/ZW3aeiMtQVrjrpytctp20/DAJVgdXEz1fvlNrzdrDsc71E0Tek0QghIdKFfZnTJ+kvgN3tY5SNK1SHACTwcATHaoTkfw++xS/O9oxzPocrS1f0tbyGQMS/0eYZQoPJL5Op8SPaZPwOfMcLRhm/IetAcNZZX6GB4zLyFj13Tt6f7kmzR3BnV+sJj7B5rKtq66E4+evEpdJkbt/d53mwSkb+GHDsVzHCrIOgyuSCITIhodbhRI1ticl3NQtalcrhFFda3F73dLcH1GRt5ITx4whLXioZe5n3Ebr0qxxNMSa6qnuCUJ41vo0PRLf5wNrPxIx8YHfFL72+5QG6ki2qqPm1IYj18t57DyRsfrpvztPsXjPGZff6m/7cBmdPlkOgMVmJ3T0XMYt2O+c8HbiQtIQ3cNnL2d5V/Db5mhCR8/lnglruJburkOeRrknfQRC5KHvBzd3vv4w1SprgWYj3eqX5Yf1ab/VVigWyJrRt2cohZET+3Rl9tkr8629O0OM//Ks6Te6+UcSrwOZa29JJXWWIGXhvA7mvC7CYkcTYnVRtuiauF5HynMfzM+8I/qJH5OWAO1YOyTN9kvJdw6xyctlprz/ctkhLlxN5N27G/LuvKTq9NPXRlG6iD/9m1VGg8tknFJcb8uxi3y98jDPdq7l3Ld031kg4wI/Qu4IhLhh0neQLn6uHYuea5fpMcVzMMzRgpkJ9j50ThzH17ae7NVVeMC4nFqGEwRgpYo6S1/jSiabP+UP/zd50/Rdrqumetohvmx/TJr3Pcavcr6OvWzh/knXV5j7ccOxDGUwPpy/n8ZvL3K5gE98ulLb1nRLfUYn31nM23nKo1hTfLc2itDRcznv4ZrR3rL12AUSrJ73rWSH3BEIcYO0qlaS57rU4pNFBwCoUTo4iyOgXvkirDl0Lst2rhzTZXjfNsDlviASqKWiGWSazyDTQgaZFrLXUYnnrU+wJ58mskW8szjDtrf/2ePRsTuj47jzy9Uu9+0+GcfKA9cLCm4+eoGTF6+57Oc5dPYSh2Ou0K3+9Yq3P29KWuvh5MVrbh8Jrj0cS9FAP+qXz159Kk9FX7jK3RPW0rdJRT6+PyzrA7JJEoEQN4jBoBjRqaYzEeSnqwSwTdfgWetT/GlvQ2PDIfoZlzHP/2WW2sNZ7WjIfl2RA46KBKpEruoAYilKEa4QTxC5fZTkqelrozxqN21Nxsl8VrtGa03P8RkTxLOztvHLsFYZtnf+ZCWAR0t8pvbglA05Os5T8deSHpntPunZ6nPZJYlAiAIg5QNk67EL3D1hbZp90wY149HpmzIcM/3RZtxWMyRNSe3sUyx3NGa5ozEzbF0ZaFrIQ8bF3G7clqHlXkdl6hqOcUKX5B97S6baenCWnI6Gyr1TcddItDkI8DPyx9YTGfY7HJovlh5yeWzKTOjB0zexdN9Z/vm/tlxMNXIp0ebAbDJwOOYye08lDcm95uKxzPxdp9h23LMPZ601luR4U9z/9To2/nee3W91o9/k9Xx4byPqlisCwOqDsfy44SgTBjRxdnR7a8STJAIhbrCW1Upw2eJ6eGXjytc/WKuWKsQL3epkmJWbokNt98ty5kQsRfnUdh9f2O6mooqhhjpJWXUeKyZ6GtbTxHCQ2fY2VFCxDDPNZYjxX9Y56jHb3pZVjkZYMXKRrB935ZVW7y/NdP/hmMsZ+iRSpHywpnQg9/oi7V3DK7N38tF9YXT6+Pp8iPsmrcvwjX/4D1tcnj/mkoW+E9cy/dFmVAtJqjQ7dfV/vDN3Lxtf7uRcKGnjf0mjreq/sQBI6nSf/mjSgINHpm3E7tDYHNrrI54kEQhxg/08NOMjCVeWjergfL36pY60/eD68pT1yxdxvl75QkeuWe2cjLvGo9My3jlklw0TUbpcmpnKs+wd07SppM4wwLiE+43Luc24y7l9uT2Mbbo6Vm0iERObHbXYrqtjJ/Mqr97gLgkAJFgzL5v96+ZoProv+8/i7Q7N7pNxbD12kWPnrzJtTRRv39UAgHfmJo1+OhWX4EwE6aX+xp8yVHbJ3jPOhKO9NDdEEoEQBcyMIc2pXCLtqmYVi19/n/5baeWSSftCSwXRuHIxmoeWYPWhWF7vVY8/t51g5saMC9uXKmzm0TZVs7U0Z2rHdRnG2h5krK0/4eowzQz7KKXi6G9cSge1PU3byzqAixTmkg7kgK7EBV0Yhea8LsISR2OuEkAwV/HHymFdnnMUwdt9EDtPxDFyVsbHX6kdjrmc7fOmPKarUTrpLsDVN/mU0WOxlzOO1Lp49frIpJSP/J9S/ffz1shXSQRCFDC31QzJupEL/iYjs59sk2Zbi2olMySCv55qQ4MKRTkTn5BpIvh2UASDp0dmcVXFNl2DbfakCqkf2+4nEAsJmCnMNVoY9tLMsJ8i6irFuEwLw14KkzSMs7BKYCS/ZzhjgvZjk6M2/zhaEaOLssFRlyvkfYXS2S76FVJzlSg2/neewv4m1h6OpXpp94sLHTqblES+X3eUJXvPMuXhCOe+O79czdbXurgcJbU9OmN/w6Hk+laA18psSCIQwod8N7g5Ycl9DuWLBbJsVAc6jlsOwBt31qNltZIEmY0cib1CiaDsVztNxI9EkuY+WDAzz9GSeY6WLttWUaepo44TrK5yURcmVJ3GgYEKKpZexnV8YJwCJCWGv+2tmONozVZHDYpyhXLqHPt15TTrQRtw0MKwl7O6GId1hWzHnt4OFx/K93+9zkXLzJ24eC3NfAmAP7dlnoTg+mOik3Hu153OK5IIhLhJLB/VIVdVOw++ewd+xrRzSAslF8orZDbyaJuqzu1VShYi5lLmk8xaVC1BYX8TS5I7XLPrqC7LUV3W5b73bA9SUcVQXp3jTsM67jSu4z7TyjRtrmp/ljrCCVFx1FXH8MeKv0oa+bPI3oRv7Xfg0AbiCeKELkU8nq/F4G1v/Z35/Iizl1x/+B87f9Ub4UgiEOJmEZqNRWVcSZ8E4Hq5hSKBGWcwhwT78+dTbbjrqzUZ9lUqEcisYa24lmin7uvzcxWXK3aMzkSxzlGft20P0cKwj4bqCH7KxiFHBe40rqOh+o8LBLPUEU6sLsp+XYlK6izDjX/TxXh9RI9dK7br6kQ6arPY3oSTlCJGF8VCyu9dcAoRxV2z0vzdJS73Wb20foMkAiFucWP61He7tnIhc9JHQOe6ZVzuTz06KUXqzurUpbe96RoBLHeEs5xw57Y5jjZu2/9mb08FFYtCU4o4GhmOEGY4zKPG+Qw1pa3rdEEXJkYX5ZwuynxHM2bb2xCP++f/3hb21sIbfk11s9X4joiI0JGRWXVgCSE8dSruGiGF/TG5umOwO6jxyr/O9z0bluOrAU3StPkv9godxy2naZXibD56If0pCpSynKOO4Rhl1QVKEI+/slKecxRW16iloqluOEWC9mOJozHHdRkW2CPYqmumOYcZKxVVDCWIJ1AlYtF+XCCYg7oCN+LOIqezl5VSm7XWEa72yR2BED4uszWDVao+iR+GtHC5aE/VUoWcH04pVVQPv9fDOZTyyQ7VnauZeeKu8PL8ue2kx+2z4zQlOe1wvfAQQH0VRT/jUjoat9GZLQw3/c0ZXYztjuoUU5cJVWcoQTwmlXEewmFHOc7o4uzXldjqqMl2XY3zugiFuUZxdYkQFccJXZJDuqJXfrfckEQghHDLaFB8fF8YG/47R+vqJTEYMv/GO//Z2zgdl4DRoPjzqTYcOHOJ48kdnMPbV2dU11rOO4zNr3bmnx2nk8ATYgAAC4tJREFUaFChCH9tO8n3647y11NtCKtUjO4NyqaZtTusXTW+XnnEe79ost06lNdsg8EGhbjG3cbVtDLspr46yhmKs8LeiLMUI0qXJUYX47IOIEhZqKzO0su4njLqAk0NB3nUtMDtNf5zlOEsxQkgkWJc5ioBnNIlKKMu8L29a4bJe6mVdTMRLbfk0ZAQwqsW7znDY99HMvWRCDrVLeO8a0j9iCNlRm6jitfLaQz9PpJOdUsTezmRpzrW4Gx8Ah/MT5r38FqvukxccZivV2QvORQyG7mSjWUyc8KEjUbqCNUNJynCFRLwJ1YX4RJBNFEHaWo4QABWLPgRTxCFSKCCiuWcLsL39q4scDRze+6URJkTmT0akkQghPC61GWfXSWCnHK1oE/U2J5pthcP8uPCVSvD2ldj27GLbPjvfIZjUjzbuSafLT6Y67i85fB7PTBmcVfmTmaJQBamEUJ4Xera/1/0b8ykh5p65To/PtYCgGHtqwHQO6w8W1/vyk+Pt2BU19p80b9xmvZTH4mgaqphualXNAOY9FBTZgxpjidmDXU9cS4v5TQJZEUSgRDihrozrDzdG7ieSJZdi1Ot8NalXhlaJ3dml0xeQKZ0sD8ArauXws9oSFPsbWi7anSqWyZNcT8Ac6rRU61rlOS2miGseKEDb/Wu7zaOmqUL06JaSeY87X5Ia0EmiUAIcdOqUTqYl7rXAeDlHnWdo5x6h1WgQrFAHmpZJcMxjSsnPWMfmGpfpzql+ejeRgDMe6YtZYsEUD2kEEUCkiacVSlZiEdahzrbP9u5Jg0rFGXm4y0pWcjsXOSmUcVihJZMWzAwtdw8Dvs4B9VQPSV9BEKIm5rDoTl7yULZop6NqDkbn8CC3acZ2Co029c6G5+AxeagUgn3H/ZrD8Xy4Dcb+LxfOJ3qlmHlgRg61y2Dn1GhlErTf/HP/7WlUokgCvub+HfXKSYuP8zuk/G82rMuj7apyuUEGwFmA39tPcm9TStmOWorM9JZLIQQN1D0hatpSoenNmH5IT5MHv2U/g7B7tAs3XeWznVLp5nDkRfybUKZUqo78DlgBL7RWo9Nt98f+B5oCpwDHtBaR3kzJiGE8DZ3SQDgyQ41KBLgR4MKGRe6NxoUXeq5LvfhTV5LBEopI/AV0AWIBjYppeZorVOX3RsCXNBa11BK9QM+AB7wVkxCCFEQuOq7yE/e7CxuDhzSWh/RWicCPwN90rXpA3yX/Po3oJPK6/shIYQQmfJmIqgApF4aKTp5m8s2WmsbEAdkKASilBqqlIpUSkXGxLhfh1QIIUT2eTMRuPpmn75n2pM2aK0na60jtNYRISE5W8ZPiP9v79xj7SqqMP770tLytA+KpkkJbZOKVqO0EaSCpFFSLSHEGA23mtiIRkVR0RjSSkKif6EmBk2MQBQ1plQQBJsGLQSoxBr6oLSltZZepcZG+vAB+IiGx/KPWdeent57e297zrlz2d8v2dkz68ye+c6Zc87ae/bsNcaYwemmI9gPnNuSnwW0hxT8fxlJE4EpwNDPfxtjjOk43XQEm4F5kuZImgT0AWvayqwBlmf6A8AjMd7msxpjzDina7OGIuIlSdcB6yjTR++IiF2SvgpsiYg1wPeBH0vqp1wJ9HVLjzHGmMHp6nMEEfEA8ECb7aaW9H+AD3ZTgzHGmOFxrCFjjGk44y7EhKTDwB9P8PAZwF86KKcb1K6xdn1Qv8ba9UH9GmvXB/VpPC8iBp12Oe4cwckgactQsTZqoXaNteuD+jXWrg/q11i7PhgfGgfw0JAxxjQcOwJjjGk4TXMEt4+1gBFQu8ba9UH9GmvXB/VrrF0fjA+NQMPuERhjjDmWpl0RGGOMacOOwBhjGk5jHIGk90raI6lf0ooetnuHpEOSdrbYpkt6SNLe3E9LuyR9OzXukLSw5ZjlWX6vpOWDtXUSGs+V9Kik3ZJ2Sfp8TTolnSppk6Ttqe8raZ8jaWO2dVfGtELS5Mz35+uzW+pamfY9kt7TCX0tdU+Q9KSktZXq2yfpKUnbJG1JWxV9nPVOlXSPpN/ld3FRZfrOz89uYHtB0vU1aTxhIuJVv1FiHf0emAtMArYD83vU9mXAQmBni+3rwIpMrwC+lukrgF9QwnNfDGxM+3TgD7mflulpHdQ4E1iY6bOAp4H5tejMds7M9CnAxmz3bqAv7bcC12b608Ctme4D7sr0/Oz7ycCc/E5M6ODn+EXgTmBt5mvTtw+Y0Waroo+z7h8BH8/0JGBqTfratE4ADgDn1apxVO9nLBvv2ZuERcC6lvxKYGUP25/N0Y5gDzAz0zOBPZm+DVjWXg5YBtzWYj+qXBf0/pyyxGh1OoHTga3A2ylPbU5s72NKoMNFmZ6Y5dTe763lOqBrFvAw8C5gbbZXjb6sbx/HOoIq+hh4DfAMOYGlNn2D6F0CbKhZ42i2pgwNjWS1tF7yuoh4FiD3r037UDp7pj+HKRZQzrqr0ZnDLtuAQ8BDlLPl56KsbNfe1lAr33Xzc7wFuAF4JfNnV6YPyqJPD0p6QtIn0lZLH88FDgM/yOG170k6oyJ97fQBqzNdq8YR0xRHMKKV0CpgKJ090S/pTOBe4PqIeGG4okPo6ZrOiHg5Ii6gnHlfBLxxmLZ6qk/SlcChiHii1TxMW2PVz5dExEJgKfAZSZcNU7bXGidShlC/GxELgH9RhlmGYsx+K3mv5yrgp8crOoSW6v6PmuIIRrJaWi85KGkmQO4PpX0onV3XL+kUihNYFRE/q1VnRDwHrKeMuU5VWdmuva2hVr7rlr5LgKsk7QN+QhkeuqUifQBExJ9zfwi4j+JQa+nj/cD+iNiY+XsojqEWfa0sBbZGxMHM16hxVDTFEYxktbRe0roy23LKmPyA/SM52+Bi4Pm81FwHLJE0LWckLElbR5AkyiJBuyPim7XplHSOpKmZPg24HNgNPEpZ2W4wfYOtfLcG6MtZO3OAecCmk9UXESsjYlZEzKZ8tx6JiA/Xog9A0hmSzhpIU/pmJ5X0cUQcAP4k6fw0vRv4bS362ljGkWGhAS21aRwdY3mDopcb5Q7+05Sx5Rt72O5q4FngRcqZwMco48EPA3tzPz3LCvhOanwKeFtLPdcA/bl9tMMaL6Vcmu4AtuV2RS06gbcAT6a+ncBNaZ9L+aPsp1ymT077qZnvz9fnttR1Y+reAyztQn8v5sisoWr0pZbtue0a+A3U0sdZ7wXAluzn+ykzaqrRl3WfDvwVmNJiq0rjiWwOMWGMMQ2nKUNDxhhjhsCOwBhjGo4dgTHGNBw7AmOMaTh2BMYY03DsCExjkfTP3M+W9KEO1/3ltvxvOlm/MZ3EjsCYEhRwVI5A0oTjFDnKEUTEO0apyZieYUdgDNwMvDNjzH8hA9x9Q9LmjCP/SQBJi1XWbbiT8oAQku7PIG67BgK5SboZOC3rW5W2gasPZd07VdYGuLql7vU6Eo9/VT7xbUzXmXj8Isa86lkBfCkirgTIP/TnI+JCSZOBDZIezLIXAW+OiGcyf01E/C1DX2yWdG9ErJB0XZQgee28n/IE7VuBGXnMY/naAuBNlLgzGygxjH7d+bdrzNH4isCYY1lCiRGzjRKO+2xK3B+ATS1OAOBzkrYDj1MCic1jeC4FVkeJpnoQ+BVwYUvd+yPiFUqYj9kdeTfGHAdfERhzLAI+GxFHBQKTtJgSHrk1fzll8Zh/S1pPiSN0vLqH4r8t6Zfx79P0CF8RGAP/oCzROcA64NoMzY2k12fEznamAH9PJ/AGSmjsAV4cOL6Nx4Cr8z7EOZSlTDsSYdSYE8VnHMaUaJcv5RDPD4FvUYZltuYN28PA+wY57pfApyTtoEQLfbzltduBHZK2RglJPcB9lGUrt1Mivt4QEQfSkRgzJjj6qDHGNBwPDRljTMOxIzDGmIZjR2CMMQ3HjsAYYxqOHYExxjQcOwJjjGk4dgTGGNNw/gc937AT7fMvywAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(cost_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEJCAYAAAB7UTvrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXhU1fnA8e+bPSEJgSwQEkLY9xBCWFRkdUNR3BF3qtL6c621Sm3VamurrW3V2trigmhVtCgFcRdBRZEdwhIgAZIQEpIQyL4n5/fHHWKAACHJzJ0k7+d55pm5d2bufecG7nvPOfecI8YYlFJKKQAPuwNQSinlPjQpKKWUqqdJQSmlVD1NCkoppeppUlBKKVVPk4JSSql6TksKIvKaiOSKyLYG67qKyBcikuJ47uJYLyLygoikikiSiCQ4Ky6llFIn58ySwuvARcetmwssN8b0B5Y7lgGmAf0djznAS06MSyml1EmIMzuviUgssMwYM8yxvAuYZIzJFpFIYKUxZqCI/Nvx+p3jP3eq7YeFhZnY2Finxa+UUu3Rhg0bDhljwht7z8vFsXQ7eqJ3JIYIx/ooYH+Dz2U61p2QFERkDlZpgpiYGNavX+/ciJVSqp0RkfSTvecuDc3SyLpGizDGmHnGmERjTGJ4eKOJTimlVDO5OinkOKqNcDznOtZnAj0bfC4ayHJxbEop1eG5OiksBW5xvL4FWNJg/c2Ou5DGAYWna09QSinV+pzWpiAi7wCTgDARyQQeB54G3hOR24AM4BrHxz8GLgZSgTJgdnP3W11dTWZmJhUVFS2IXjXk5+dHdHQ03t7edoeilHIypyUFY8ysk7w1tZHPGuCu1thvZmYmQUFBxMbGItJYU4U6E8YY8vPzyczMpHfv3naHo5RyMndpaG41FRUVhIaGakJoJSJCaGiolryU6iDaXVIANCG0Mj2eSnUcru6noJRS6hQqqmtZujmL0qoaeoUGENM1gOguAfh5e7pk/5oUWll+fj5Tp1rNJgcPHsTT05Oj/SnWrl2Lj4/Pabcxe/Zs5s6dy8CBA0/6mX/84x+EhIRwww03tE7gSilblVbW8PaaDOZ9u5e84spj3hOB7sF+9OwaQK+uAfQKDWDKoG4M6RHc6nFoUmhloaGhbN68GYDf/va3BAYG8uCDDx7zGWMMxhg8PBqvvZs/f/5p93PXXa3SLq+Uaobc4go+257DkMhg4nuG4OnR/CrWwvJq3lydxqur9nGkrJpz+4fx91kj6RcRSHp+GfsPl5GeX0bG4TIyDpfy9e48cosrCQ/y1aTQlqWmpnL55Zczfvx41qxZw7Jly3jiiSfYuHEj5eXlzJw5k8ceewyA8ePH8+KLLzJs2DDCwsL42c9+xieffEJAQABLliwhIiKC3/zmN4SFhXH//fczfvx4xo8fz1dffUVhYSHz58/n7LPPprS0lJtvvpnU1FSGDBlCSkoKr7zyCvHx8TYfDaXaro+3ZvPrxVs5UlYNQGd/b87tH8bEAeFMHBBORLBfk7ZzuLSK11btY8H3aRRX1nDe4AjumtyPkTFd6j8TFujLqF5dTvhueVVt6/yYRrTrpPDEh9vZkVXUqtsc0iOYxy8d2qzv7tixg/nz5/Ovf/0LgKeffpquXbtSU1PD5MmTufrqqxkyZMgx3yksLGTixIk8/fTTPPDAA7z22mvMnTv3hG0bY1i7di1Lly7lySef5NNPP+Xvf/873bt35/3332fLli0kJOiI5Eql55fy9poMPth0gJiuAdwzpR8TB4Sf9oaKwrJqHl+6jf9tzmJ4VGdevXUoB46U8/XuPL7enceyJKu/7eDIYCYNDCcuqjMVNbUUV9TUP0oqqympqKGooobVe/KpqKll2rDu3DW5H0N7dG7yb/D3cV77QrtOCu6mb9++jB49un75nXfe4dVXX6WmpoasrCx27NhxQlLw9/dn2rRpAIwaNYpvv/220W1feeWV9Z9JS0sDYNWqVTz88MMAjBgxgqFDm5fMlGrramrrWL4zl//8kM63KYfw9BAmD4wgObuIW+evI75nCPdN7c+kgY0nh6935/HwoiTySiq5/7z+3DW5H96eHiTEdOHSET0wxpCcXczXu/NYuSuXl7/ZS03dscO3eXsKQX7eBPl5EejrxcXDI/nZxD707xbkqsPQJO06KTT3it5ZOnXqVP86JSWF559/nrVr1xISEsKNN97YaF+Ahg3Tnp6e1NTUNLptX1/fEz7jzGHRlWoLDhZWsHBdBgvX7udgUQWRnf144PwBzBzdk27BflTV1PH+xkxe/CqV2a+vY0R0Z+6d2p8pgyIQEUora/jDx8m8tSaDfhGBzLt5FHHRISfsR0QY0iOYIT2CuXNSX4orqknPLyPQ14tARxJw1d1DLdWuk4I7KyoqIigoiODgYLKzs/nss8+46KLj5yRqmfHjx/Pee+9x7rnnsnXrVnbs2NGq21fKbsYYCsqqOVBQTtbRR2FF/XJSZiF1xjChfzhPzhjKlEEReHn+eIOHj5cHs8bEcFVCNB9szOTFFanctmA9w6M6c+3onrzy7V4yDpdx+/jePHjhwCaf2IP8vBkW1fTqIHeiScEmCQkJDBkyhGHDhtGnTx/OOeecVt/HPffcw80330xcXBwJCQkMGzaMzp3b5j9UpRoqqqjmmU928sHGA5RXH9vo6uPlQVSIPz1C/JgzoQ+zRscQExpwyu35eHlw3ZgYrhoVzeKNB3hxRSqP/m8b0V38eeeOcYzrE+rMn+NWnDrzmrMlJiaa4yfZSU5OZvDgwTZF5F5qamqoqanBz8+PlJQULrjgAlJSUvDyOvNrAT2uqjVU19axN6+UHdmF7MgqIi2/jPMGR3BlQjTenk0bYOGrnTk88sE2cosruCohmsGRwfQI8ScqxJ/IED9CO/m0uBd+dW0dmzIKGNIjmEDf9nftLCIbjDGJjb3X/n6tqldSUsLUqVOpqanBGMO///3vZiUEpZorObuINXvz2ZFdxI7sInYfLKGqtg6wrs7DA335YkcOL65I5Z7J/bkiIeqkyeFIaRVPLtvB4k0HGNAtkH/ddA7xPU+s328N3p4ejOnd1Snbdnd6hmjHQkJC2LBhg91hqA6orKqGP326iwWr0zAGQjv5MKRHMLPPibUaZCOD6R3WCU8PYcWuXJ77MoWH3k/ixRWp3D2lH1eOjDqm7v/jrdk8tmQbBWXV3Du1P3dN7ouvV9touG1rNCkopVrVmr35PPR+Eun5Zdx6dix3TupLRJDvSat0pgzqxuSBEXy105EcFiXxjxWp3D25H+P7h/HE0h18uv0gw6M688ZPxjqlF6/6kSYFpVSrOFo6eP37NGK6BrBwTtMbaEWEqYO7MWVQBMuTc3lu+W5+uSgJsKqZHr5oEHec2/uY0oNyDk0KSqlG1dTWsSO7iJyiSnqHdaJXaMBJ6/tX78nn4feTyDhslQ4eumggAT5nfnoREc4b0o2pgyP4MjmX71IPcdNZvegbHtjSn6OaSJOCUgqwksD2rCJ+2JvPD3vzWZ92hOLKHztLensKvUI70T8ikH6OR5+wQP67YT9vrE6nV2gA784Zx9hWuH1TRDh/SDfOH9KtxdtSZ0aTQiubNGkSv/rVr7jwwgvr1z333HPs3r2bf/7zn41+JzAwkJKSErKysrj33ntZtGhRo9t99tlnSUxs9C6y+v3MmTOHgADrnuyLL76Yt99+m5AQ59yhodq+vOJK3t+YWZ8EShxJoE94Jy6N78G4PqFEd/FnX14pqXklpOaWsPNgMZ9tP8jRURxE4Cfn9OaXFw506pg8yjU0KbSyWbNmsXDhwmOSwsKFC/nzn/982u/26NGj0YTQVM899xw33nhjfVL4+OOPm70t1b7ll1Qy75u9LFidRkV1Hf0iApnhSAJj+3QlIujYkT4TYo4dqbOyppa0Q2Wk5pbQKzSgzfbeVSfSVptWdvXVV7Ns2TIqK61JMtLS0sjKyiI+Pp6pU6eSkJDA8OHDWbJkyQnfTUtLY9iwYQCUl5dz3XXXERcXx8yZMykvL6//3J133kliYiJDhw7l8ccfB+CFF14gKyuLyZMnM3nyZABiY2M5dOgQAH/9618ZNmwYw4YN47nnnqvf3+DBg7njjjsYOnQoF1xwwTH7Ue3PkdIq/vTpTs790wpe/nYv04ZF8tUvJvLlAxN56orhXDqixwkJoTG+Xp4M7B7EJXGRmhDaGVtKCiJyH3AHIMDLxpjnRKQr8C4QC6QB1xpjjrRoR5/MhYNbWxbs8boPh2lPn/Tt0NBQxowZw6effsqMGTNYuHAhM2fOxN/fn8WLFxMcHMyhQ4cYN24cl1122Ulv03vppZcICAggKSmJpKSkY4a9fuqpp+jatSu1tbVMnTqVpKQk7r33Xv7617+yYsUKwsLCjtnWhg0bmD9/PmvWrMEYw9ixY5k4cSJdunQhJSWFd955h5dffplrr72W999/nxtvvLF1jpVyG4Xl1bz67V5e+y6N0qoapsf14L6p/egX4V4jdCr7ubykICLDsBLCGGAEMF1E+gNzgeXGmP7Acsdym3S0CgmsqqNZs2ZhjOGRRx4hLi6O8847jwMHDpCTk3PSbXzzzTf1J+e4uDji4uLq33vvvfdISEhg5MiRbN++/bQD3a1atYorrriCTp06ERgYyJVXXlk/BHfv3r3rJ91pOOy2atuMMRwsrOCb3Xn85fNdjH/mK174KpUJA8L49L4Jjpm9NCGoE9lRUhgM/GCMKQMQka+BK4AZwCTHZxYAK4GHW7SnU1zRO9Pll1/OAw88UD+rWkJCAq+//jp5eXls2LABb29vYmNjGx0qu6HGShH79u3j2WefZd26dXTp0oVbb731tNs51fhWR4fcBmvYba0+anvyiivZebCI3TklpOQUszunmJTcEoorfrxz6Pwh3fj5eQO045c6LTuSwjbgKREJBcqBi4H1QDdjTDaAMSZbRCIa+7KIzAHmAMTExLgm4jMUGBjIpEmT+MlPfsKsWbMAawa1iIgIvL29WbFiBenp6afcxoQJE3jrrbeYPHky27ZtIynJ6shTVFREp06d6Ny5Mzk5OXzyySdMmjQJgKCgIIqLi0+oPpowYQK33norc+fOxRjD4sWLefPNN1v/hyunM8awJ6+U9WmHWZd2hPXph0nPL6t/v0uAN/27BTEjvgcDugXRPyKIAd0CCQ30PcVWlfqRy5OCMSZZRJ4BvgBKgC1A4zPHNP79ecA8sEZJdUqQrWDWrFlceeWV9dVIN9xwA5deeimJiYnEx8czaNCgU37/zjvvZPbs2cTFxREfH8+YMWMAawa1kSNHMnTo0BOG3J4zZw7Tpk0jMjKSFStW1K9PSEjg1ltvrd/G7bffzsiRI7WqyM0ZY8grqWRPbilJmQWsSzvChvTD9XMDd+3kQ2KvLtwwNoZhPTrTv1sQYYEtHyFUdWy2D50tIn8AMoH7gEmOUkIksNIYM/BU39Whs11Hj6vzGGNIy7du79zj6AuwJ6+EPbklFDWoAuod1onEXl0YHduVUbFd6BPWSROAaha3GzpbRCKMMbkiEgNcCZwF9AZuAZ52PJ94z6ZS7dAji7fxztqM+uWIIF/6hgdyWXwP+oUH0jcikEHdgwkP0iog5Xx2dV5739GmUA3cZYw5IiJPA++JyG1ABnCNTbEp5TKfbM3mnbUZXD82hmtGRdMnPJDO/t52h6U6MFuSgjHm3EbW5QNTW2n7WqxuRXZXMbZXOUUV/GrxVuKiO/PEZUObPPOYUs7U7v4V+vn5kZ+fryeyVmKMIT8/Hz+/0/dyVU1njOGXi5KoqK7lbzPjNSEot9Huxj6Kjo4mMzOTvLw8u0NpN/z8/IiOjrY7jHbljdXpfLM7j9/NGKrDQiu30u6Sgre3N71797Y7DKVOKjW3mD98nMykgeHcOK6X3eEodQwtsyrlQlU1ddz/7mYCfDz501Vx2val3E67Kyko5c6eX76bbQeK+NeNo4gI1nYa5X60pKCUi6xPO8xLK/dwzahoLhrW3e5wlGqUlhSUaqF9h0r5LvUQUSH+DI0KbnQ+guKKan7+3maiuwTw+GVDbYhSqabRpKBUM+zNK+Hjrdl8tPUgydlFx7wXEeTLsKjODO0RzNAenRkWFcxzX6Zw4Eg5//3ZWQT66n875b70X6dSTbQnr4SPk7L5aGs2Ow8WAzCqVxcenT6EKYMiyCuuZNuBQrZlFbL9QBErd+XWz2MMcPfkfozq1dWm6JVqGk0KSp3G7pxiHlqUxOb9BcCPiWDasO70CPGv/1zvsE6M6f3jSb+8qpadB4vYllVEQWkVP53Y1+WxK3WmNCkodRLGGN5Ync4fPk4myM+LR6cP4eLh3Yns7H/6LwP+Pp6MjOnCyOMmvVfKnWlSUKoRh0oq+eV/t7BiVx6TB4bzp6tH6CilqkPQpKDUcVbsyuWX/91CUUUNT1w2lJvP6qWdzFSHoUlBKYeK6lqe/mQnr3+fxqDuQbx1+zgGdtfJ7VXHoklBdXjGGDZmFPDIB1vZlVPM7HNiefiiQfh5e9odmlIup0lBdUgV1bWs3pPPl8k5LE/O5WBRBWGBvrw+ezSTBkbYHZ5SttGkoDqMvOJKVuzM5cvkHFalHqKsqpYAH08m9A9n6uAILhjSnc4BOuuZ6tg0Kah2zxjD75YlM//7fRgDkZ39uCohmqmDIxjXJ1SriZRqQJOCateMMTy5bAfzv0vjutE9uemsXgyJDNa7iZQ6CU0Kqt0yxvDHT3Yy/7s0fnJObx6dPliTgVKnoUNnq3bJGMOzn+9i3jd7uWlcL00ISjWRLUlBRH4uIttFZJuIvCMifiLSW0TWiEiKiLwrIj52xKbahxeWp/KPFXuYNaYnT1w2VBOCUk3k8qQgIlHAvUCiMWYY4AlcBzwD/M0Y0x84Atzm6thU+/CPFan87cvdXD0qmqcuH46HhyYEpZrKruojL8BfRLyAACAbmAIscry/ALjcpthUG/byN3v582e7mBHfg2euitOEoNQZcnlSMMYcAJ4FMrCSQSGwASgwxtQ4PpYJRLk6NtW2zf9uH099nMwlwyP5yzUj8NSEoNQZc/ndRyLSBZgB9AYKgP8C0xr5qGlkHSIyB5gDEBMT46QolbupqK7lwy1Z5BZXUlBWRUFZNQXl1RSWVVNQXkVheTU5RZVcMKQbz10Xj5en3kOhVHPYcUvqecA+Y0wegIh8AJwNhIiIl6O0EA1kNfZlY8w8YB5AYmJio4lDtS9f787j0f9tI+NwGQABPp509vems783IQHe9AkLpLO/NzGhAdxxbh+8NSEo1Wx2JIUMYJyIBADlwFRgPbACuBpYCNwCLLEhNuVGcooqeHLZDj5KyqZPeCf+c9tYRvfugq+X9kBWyllcnhSMMWtEZBGwEagBNmFd+X8ELBSR3zvWverq2JR7qK0zvLE6jb98vpvq2jp+cf4A5kzso8lAKRewpUezMeZx4PHjVu8FxtgQjnKR6to6isqrCfb3PmkVz5b9Bfz6f1vZdqCIiQPCeXLGUHqFdnJxpEp1XDrMhXKJnKIKrpv3A/sOlQLQyceTYEe7wNFngC+TcwgP9OUf1ydw8fDu2ulMKRfTpKCcLr+kkhteWUNuUQW/mjaIypo6Csurj3nsP1xGSWUNt5wVyy8uGECQnw5hrZQdNCkopyosr+bm19ay/3AZC34yhnF9Qu0OSSl1CnrvnnKasqoafvL6OnbnFPPvm0ZpQlCqDdCkoJyiorqWO95Yz6aMI7xw3Uid4lKpNkKrj1Srq66t4+63N/Fdaj7PXjOCacMj7Q5JKdVEWlJQraq2zvCL97bwZXIOv5sxlKtHRdsdklLqDGhSUK3GGMOvF29l6ZYsHr5oEDedFWt3SEqpM6RJQbWaF5ansnDdfu6e3I87J/W1OxylVDNoUlCtYm9eCS+uSOGyET34xQUD7A5HKdVMmhRUixljeHzpdvy8PHl0+hDthaxUG6Z3H6kW+3TbQb5NOcRvLx1CeJCv3eG0f5UlkJ8ChxyP/FSoqwHvAPD2b/DsZ732DYbB08G/i92Rtx8//Au+ew5ixsGIWdB3Cng2sRd+TSWkfwdVZRCVAME9nBvrGdKkoFqkrKqGJ5ftYHBkMDeO62V3OG1PTSWseg5SvwAvP8fJ3P/EE3z5kR+TQHGDqUbEA0J6Wd+tLoPqcsejFEzdj5/78rdw3m8h/gbw0AqCZqurhc9+DWtegqhRsPdr2L4YOoXDsKthxHUQOQKOLy0X7Lf+xilfwN6V1t/qqKBIa1tHHz1Ggl+wS39WQ5oUVIv8/atUsgsr+Puske17trPqcvDwavrVYFOkr4YP74VDu6HnWOuEU5rnOKk3PMGXgU8QhPWHPhOt59D+EDYAuvYGr0ZKZ8ZAbbX13fxU+OwRWHo3bFwAFz8LPeJb73ecicoSOJLmeOyDomwI6wc9x0H4IPdOWFVl8MEdsHMZjL0TLnzK+pulfglb3oH1r1rJInywIznEWQkg5QvI3WFto3MMxF8P/S+0Sm5ZGyFzPRzYYG0XAIHwgTD4Mhj7M+jk2pEAxJi2O3lZYmKiWb9+vd1hdFipuSVMe/4bZsRH8ew1I+wOxzmMgbUvwxePWSffARfCwIuh31TwDWreNisKrSv39a9ZJ4npf4P+5506Bjjx6vNM1NVB0kLrd5QegsSfwJTfQEDXk3+nshiyNluJqc8k8PI5s32W5ELSu3BwKxzeZyWC0txjP+PpC7WV1mu/zhA9BmLGQsxZ0CMBfALObJ9NUVsNyR/C5rcgOArOvtdKTKf8LXnwzkw4sBEu+iOMu/PEz5QdtkoNWxZC5lprnYeX9VsGXAj9L7AS+cn+jmWHrSRxYCOkfw97V4B3J0icDWffA0HdW/a7GxCRDcaYxEbf06SgmsMYw02vrmVLZgErHpxEWKCbtyWU5FpF+KiEpp9ci7Jhyf/Bnq+g71QI7Aa7P4Xyw+DpA70nwqCLrSTR1P+wyR/Cx7+EkhzranPyI+Ab2PzfdabKC2DlH2HtPOtKderjMPImMLXW1eyBDY7HRshNpn6q9IAw6+p35I0QMfjk2zcG0r61El7yMqirhuBoq0TTJdZ61L/ubcVweC/sXwMZP1jPeTutbXl4WfvyC2mkvcTxulM49DobIoacvpRRkgsbXrdiK8624irNg9oqq83lnPshupHzZN5ueOtq6/tXvwqDLjn9cT6Uav2umHHNrwrKTYZv/wrbFoGHNyTcDOfcByE9m7e9BjQpqFb3UVI2d729kSdnDOXmpnZSy9pk1YFHurhUsXclLLoNyg5Zdbbn3G/9x/Y4xUxu2xfDh/dbJ4wLfm9dWYtAbY114tr1Mez8yKoCAeuqNnwQdI52PKKgc0/rStQ30EowHz9oVRF0Gw6XPW/FYpeDW63klLHaKq2U5kFNufWef1fr5Hi0jruu1rqq3vWJdZKPSoSEm2DolT+e8MoOO6pQ5luN4H4hVvvFqFsh/AxvUS47DPvXwv4fIDvJUZV2XHXa0eej/LtAr3Mg9lyIHX9skshcbyXB7Yutv2ffKTDmp9D/fCjLhzX/gnWvWCW4XuNh/P3Q7zzr7532HSy83qo2nPUuRNvwN8vfYzVqb34HMFZyHv8AhDa/L5AmBdWqSitrmPqXrwkN9GHp3ePx9GjClXdJHvw9waqSSJwNUx9z/t0wdXXw3d/gq99bdfAjb7TqfY+kQWg/q0ged511l85R5QXwyUNWtUfUKLhi3smrFoyxruZ2fQR7VljbLc4+toEXrN9ZU2VdjU+aC2fd3bptE81ljPU7t/7XqtY4mgS6xDZemio9ZFWNbHrTupr3DoChV1i/d9sHVjVQ9BgrgQ693Lqad3b8hZmQtsrx+BYK0q33/LtC7DlQeMCqkvEJsuryx9xhtckcr7IYNiyAH/4JRQcgYigMvAi+/7t1PG74r/Vsp8JM+O4Fq12otspqGxp9W7M2pUlBtao/fpLMv7/ey/t3ns2oXk08sS+917rajL8BNv3Hqsu+8A8w/JqW1ZWfTPkRWPwzq7pn2FVw6QvWFXtdLexYYl15ZW+xqoTG/sz6z5W9BRbfaZ3YJz4M5/4CPM/wXozaGuv7hZnWo8jxXFtllVBacHXnNoyxrr43vWElAwTirrWSfffh9sZ2JN263fNokvDuZCWp+FlNawOqqbKqa7573kp8vcbDdf9xr9t5S3Jh9YsQf+OZl8IcNCmoVpOaW8xFz33LlQlR/OnqJlYDZSfBvydYjXMX/dFaXna/VXfdewJc8tfGr94aMgYqCqxqidMlkewt8O5N1hXfhX+AMXNO/I4xVrXSd8//2KBXXWqVIK6cZ2/VTltSXWE9NyxttQd1dZC9yarqO9MG9jZAk4JqFcYYbnhlDdsOFLLiwUmENqVx2Rh4fTrkJcM9G3684qqrtRr9vnzCqsse/3OrntTbz/oPeXgvHNxineCzk6zn8sPQKcK6O6XnOKsRr3vcsf9pN74JH/0CAkLh2gXQc8zpY8zabNU5dwqDiXOdc8eLUm7kVElB+ymoJimprOEvn+/i+z35/O7yYU1LCGBV1aSvskoDDYvgHp5Wlc2g6fD5r+HrZyDpPesunoNboarE8Tlv6DbEahgO7Qu5O60GyOQPrfe9/K2r+pixVmPulretu4KuehUCw5sWY494uPyfTT8YSrVjp00KInI38JYx5khr7FBEBgLvNljVB3gMeMOxPhZIA65trX2q5jPGsHRLFk99lExucSXXj43h+jExTftydQV88ajVaJdwS+OfCeoGV71itTUsf9JqtIy/3ioBRI6w7uhprPhefPDHWxgzVlu9gk0tnPugdZvnqe4sUkqdVFNKCt2BdSKyEXgN+My0oM7JGLMLiAcQEU/gALAYmAssN8Y8LSJzHcsPN3c/quV2HSzmsSXbWLPvMHHRnZl3cyLxPUOavoHVL0JBBty89PQNtn0nW4+mCupu3eEy9HJruarUuqXQzcaRUaqtOW1SMMb8RkQeBS4AZgMvish7wKvGmD0t3P9UYI8xJl1EZgCTHOsXACvRpNDqDpVYvUdDO/mcdDTToopq/vbFbt5YnU6Qnxd/uGI4M0f3bNqtp/UbybY63gyabg3N4Gw+nayHUqpFmtSmYIwxIk79Mr0AABmNSURBVHIQOAjUAF2ARSLyhTHmoRbs/zrgHcfrbsaYbMf+skWk0ZneRWQOMAcgJqaJ1RgKgJyiCib9eSXl1bX4eHnQo7MfPUL86x9RIX5U1dTx/PJU8ksruX5MDA9eMJAunZpx98XyJ6yOThf8vvV/iFLKaZrSpnAvcAtwCHgF+KUxplpEPIAUoFlJQUR8gMuAX53J94wx84B5YN191Jx9d1TzvtlLVW0dv5o2iMOlVRwoKCeroJxVKYfIKa6oH2InvmcI828dzfDozs3bUeYGq3frOfdbQxoopdqMppQUwoArjTHpDVcaY+pEZHoL9j0N2GiMyXEs54hIpKOUEAnknuK76gzll1Ty1pp0Lo+P4qcTT+xAVV1bR05RBYXl1QzuHozHmVQVNWQMfDrX6hQ24cEWRq2UcrWmjFP7MXD46IKIBInIWABjTHIL9j2LH6uOAJZilUhwPC9pwbbVcV5dtY/Kmjr+b3LjPWq9PT2I7hLA0B6dm58QALYuskaInPpY80cRVUrZpilJ4SWgpMFyqWNds4lIAHA+8EGD1U8D54tIiuO9p1uyD/WjwrJq3lidziXDI+kb7sQROatKraGZI+NhxPXO249SymmaUn0kDW9BdVQbtajTmzGmDAg9bl0+1t1IqpW9/n0aJZU13DX5NGPGN0dlMWSus/oMpH5pzQp29WvuPVmKUuqkmnJy3+tobD5aOvg/YK/zQlKtqaSyhte+28f5Q7oxOLIVpvgrPGB1FjvaaSxnu9XhTDyg21C48I/Q66yW70cpZYumJIWfAS8Av8GacWM5jltClfv7zw/pFJZXc/fkftY49QtvsIYUnvjwmQ3fXFUKHz8Em/9jLXsHWGPuT/ilNZVk9Ghb55VVSrWOpnRey8XqT6DamPKqWl75di8TBoQzIioI3rrGGjco43trNrErX27aUM4Ht8J/Z1tz/Z59jzW5Svfh7jEngFKqVTWln4IfcBswFKgfH9cY8xMnxqVawTtrMzhUUsU9U/rBN3+GPcth+nPgHwIf3mcNZz3tT9ZYQ431bjbGmpHqs19bg9ndvMQ1vZOVUrZpSmvgm1jjH10IfA1EA8XODEq1XGVNLf/+Zg9je3dldPVGWPk0jJhlTY849Aq483vrLqEl/weLZluT0jRUdhjevdGaQrLPRLjzO00ISnUATUkK/YwxjwKlxpgFwCWAzdMrqdNZtCGTnKJKHhwTAB/cbs1Ze8lffywRdI6GW5Za/QmSP4SXxlvz0QKkr4Z/nQu7P7MmqZn1rjXXgFKq3WtKQ3O147lARIZhjX8U67SI1ClVVNdSUllD2CnmM6iureOllXtIjO5E4rr7rAltZr554uQxHp7WlJN9JsH7t8OC6TDwYmuC9pAYuO1ziEpw6u9RSrmXpiSFeSLSBevuo6VAIPCoU6NSjaqormXWyz+wZX8BkwdGMGtMDJMHRZwweumSzVlkHilnYY9lyJ5NMPOtUzcoR42Cn34Lnz5szZ88/BqrVKF3EynV4ZwyKTgGvStyTHbzDdaEOMoGxhge+WArmzIKuDYxmhW78lj+xnoiO/sxc3RPZo7uSWRnf2rrDP9ckcpdXdcTvedtOOc+GNyEIap8A2HGP2DKo9a4RaebB1kp1S6dMik4ei/fDbznonjUSbzy7T4+2HSAB84fwL1T+1NdW8fy5FzeXpvB88tTeGF5ClMGRdAvIgjv/GQeCPgn9BoPUx47sx0FdXfOD1BKtQlNqT76QkQexJoqs/ToSmPM4ZN/RbWmFbty+eMnyVwyPNK6vRRrALuLhnXnomHd2X+4jHfWZvDe+kzWJKfxScALePiHWMNNnG7GM6WUaqApZ4yj/RHuarDOoFVJLpGaW8K9b29iUPdg/nxNXKOzpfXsGsBDY3z5RUgqJWveJLggB7lmmTX/sVJKnYGm9GjWWVJsUlhWzZw31uPj5cHLtyQS4NPgz1VTCenfQ8oXkPI55KfgCXTu2tdqG+h1tm1xK6Xarqb0aL65sfXGmDdaPxx1VE1tHfcs3MT+I2W8fcc4okL8rTfSVsEPL8HelVBVAp4+EDseRt8O/c9v2rAVSil1Ek2pPhrd4LUf1vDWGwFNCk70x0928s3uPJ65ajijY7taKze+Ccvuh4AwiLsW+l8AvSfohPVKqVbTlOqjexoui0hnrKEvlJO8t34/r67ax61nxzJzdIw1BtFXv4dvn4U+k+HaN7QPgVLKKZpza0oZ0L+1A1GWDemH+c3ibYzvF8ZvLhlstR0suRu2vgcjb4Lpf9PRSZVSTtOUNoUPse42AmuspCFovwWnyDxSxk/f3EBkiB8vXj8Sr6pCWHgjpK+yOpWd+wvtVKaUcqqmlBSebfC6Bkg3xmQ6KZ4Oq7SyhtsXrKeyuo6FcxIJqcyy5j84kgZXvgJx19gdolKqA2hKUsgAso0xFQAi4i8iscaYNKdG1oHU1Rl+/u5mducU89qto+lXtRsWzITaarhpsXV3kVJKuUBThs7+L1DXYLnWsU61kr98sYvPd+Twm0uGMImN8Pol4O0Pt32hCUEp5VJNSQpexpiqowuO1z4t2amIhIjIIhHZKSLJInKWiHQVkS9EJMXx3KUl+2gr/rfpAP9YsYdZY3oyO2gNLLwewgfC7cshfIDd4SmlOpimJIU8Ebns6IKIzAAOtXC/zwOfGmMGASOAZGAusNwY0x9Y7lhu1zZlHOGh95MY27srv+v+HbL4pxB7Dty6DAIj7A5PKdUBNaVN4WfAWyLyomM5E2i0l3NTiEgwMAG4FepLHlWOZDPJ8bEFwErg4ebux91lFZRzxxsb6B7ky+u9l+P1+Z9h0HS46lXw9jv9BpRSygma0nltDzBORAIBMca0dH7mPkAeMF9ERgAbgPuAbsaYbMc+s0Wk0UtlEZkDzAGIiYlpYSj2KKuy7jSqqq7my8Gf4f/9fIi/AS59QUc1VUrZ6rTVRyLyBxEJMcaUGGOKRaSLiPy+Bfv0AhKAl4wxI7GG425yVZExZp4xJtEYkxgeHt6CMOyRW1TBA+9uIfXgYT7v/Q4hW+fDWXfDZS9qQlBK2a4pZ6FpxphHji4YY46IyMVY03M2RyaQaYxZ41hehJUUckQk0lFKiARym7l9t1FdW8eOrCI2ZhxhY0YBG9OPcKCgHF+q+DL6NbqnfaOd0pRSbqUpScFTRHyNMZVg9VMATj5r/GkYYw6KyH4RGWiM2YU1wN4Ox+MW4GnH85Lm7sNOxhheXbWPz7YfJCmzkMoa627e7sF+JMYE8ciQQ0zMepnAg+useZBH32ZzxEop9aOmJIX/AMtFZL5jeTZWQ3BL3IPVeO0D7HVs0wN4T0Ruw+ow1ya78CZlFvL7j5IZ1D2IG8b2YnSUN2PrttA1YymkfAapR8DLH656BYZfbXe4Sil1jKY0NP9JRJKA8wABPgV6tWSnxpjNQGIjb01tyXbdweJNB+juWcwHYw8SsPdF2LwSaivBLwQGXASDLoa+U8E30O5QlVLqBE1t2TyI1av5WmAf8L7TImrDqmvr+GbzLr70/SUBnxVBSC+remjgxRBzljYkK6Xc3knPUiIyALgOmAXkA+9i3ZI62UWxtTmrUg8xoXIFgd5FcNP/oM8kbUBWSrUpp7p03Ql8C1xqjEkFEJGfuySqNmrxhkzu8v6ausiRePTV3KmUantO1U/hKqxqoxUi8rKITMVqU1CNKKms4UDyagaSjkfCjXaHo5RSzXLSpGCMWWyMmQkMwhpy4udANxF5SUQucFF8bcZn2w4yw6ygztMXhuldRUqptum0PZqNMaXGmLeMMdOBaGAzHWCwujP10ca9XOH1PTLkMvAPsTscpZRqlqaMklrPGHPYGPNvY8wUZwXUFuUUVRCU9hlBlCIjtepIKdV2nVFSUI37cEsW13isoDqoJ8ROsDscpZRqNk0KrWDVuo2M99yO96ibwEMPqVKq7dIzWAvtOlhM/OGPMAjEz7I7HKWUahFNCi30v037ucbzG6p7TYCQtjm/g1JKHaXjLrRAXZ3hwMZPiZJDMPoWu8NRSqkW05JCC6xNO8zUii+o8g6GgZfYHY5SSrWYJoUW+Gz9Ti7yWIdH3EydV1kp1S5o9VEzVVTX4rVjEb5SDYk32R2OUkq1Ci0pNNNXO3O5rO4rSroMhsgRdoejlFKtQpNCM63/4WuGe6QRMHa23aEopVSr0aTQDEdKq+iVsZga8cYjrk3OGqqUUo3SpNAMn2xJ5zKPVZT2vggCutodjlJKtRptaG6G7DXv00VKMGdr1ZFSqn3RksIZ2n+4jMTDH1Hs2w3pM8nmaJRSqnXZkhREJE1EtorIZhFZ71jXVUS+EJEUx3MXO2I7nc/XbmW8x1ZM3Ezw8LQ7HKWUalV2lhQmG2PijTGJjuW5wHJjTH9gOW44kY8xhpLNH+AphuDE6+wORymlWp07VR/NABY4Xi8ALrcxlkZtzypibNnXFHTqAxFD7A5HKaVanV1JwQCfi8gGEZnjWNfNGJMN4HiOaOyLIjJHRNaLyPq8vDwXhWtZvnYzY2QnviOuBhGX7lsppVzBrruPzjHGZIlIBPCFiOxs6heNMfOAeQCJiYnGWQEer7bOULttMR5i8B+pfROUUu2TLSUFY0yW4zkXWAyMAXJEJBLA8ZxrR2wns2ZvPhOrV1HYeRCED7A7HKWUcgqXJwUR6SQiQUdfAxcA24ClwNFJCW4Blrg6tlNZuXYjozxSCNBSglKqHbOj+qgbsFisOnkv4G1jzKcisg54T0RuAzIAtzn7VlTX4rNrCQh4x11ldzhKKeU0Lk8Kxpi9wAnDihpj8oGpro6nKVbszOUC8x3FocMJ6trb7nCUUspp3OmWVLf13bq1xHnsIyDBbQovSinlFJoUTqOwrJqu+z4CwHPYlTZHo5RSzqVJ4TQ+2ZbNNFlNacQoCOlpdzhKKeVUmhROY+261Qz2yNCqI6VUh6BJ4RSyC8uJyf4UgyBD3G7UDaWUanWaFE5h6aYDTPf4gcoe4yA40u5wlFLK6TQpnMLmDd/TzyMLP+2wppTqIDQpnMTunGKGHvmSOvGEITPsDkcppVxCk8JJ/G9jJpd6rqYmZjx0CrM7HKWUcglNCo2oqzPs2vQtvSQXnxFX2x2OUkq5jCaFRmzIOMKYspXUiRcMmm53OEop5TKaFBrx4eYDTPdcQ12fKRDQ1e5wlFLKZeyaZMetFaWsJkoOgY6IqpTqYLSkcJzC8mqGF35FjfjAwIvtDkcppVxKk8JxNu8vYKxHMiXdEsEv2O5wlFLKpTQpHGfL3mwGSQYBfcbZHYpSSrmctikcp3DPWrykDnqNsTsUpZRyOS0pNFBbZ/DP2WgtRCXaG4xSStlAk0IDuw4WM8SkUBoQDYHhdoejlFIup0mhgQ0ZR4j3SIXo0XaHopRSttCk0MCe1N30kMME9Nb2BKVUx2RbUhARTxHZJCLLHMu9RWSNiKSIyLsi4uPqmGoy1lmx9dSkoJTqmOwsKdwHJDdYfgb4mzGmP3AEuM2VweQVVxJdtp1a8Ybuw125a6WUchu2JAURiQYuAV5xLAswBVjk+MgCwKXzX27MOMJIj1TKw4aCl68rd62UUm7DrpLCc8BDQJ1jORQoMMbUOJYzgajGvigic0RkvYisz8vLa7WANqXlMVz24RerVUdKqY7L5UlBRKYDucaYDQ1XN/JR09j3jTHzjDGJxpjE8PDWu2300J7NBEglXjFjW22bSinV1tjRo/kc4DIRuRjwA4KxSg4hIuLlKC1EA1muCqiyppaAvE3gCUSNctVulVLK7bi8pGCM+ZUxJtoYEwtcB3xljLkBWAEcnebsFmCJq2LanlXEcJNCpW9X6BLrqt0qpZTbcad+Cg8DD4hIKlYbw6uu2vHG9CPEe+yxSgnSWE2WUkp1DLYOiGeMWQmsdLzeC9jSyrtj735u9zgAvWbbsXullHIb7lRSsIUxhsqM9dZCtA6Cp5Tq2Dp8Usg8Uk6fih0YBKIS7A5HKaVs1eGTwsYMqz2hMqQf+HW2OxyllLKVJoW0wyR4pOKjk+oopZTOvHYgLZkuUgw9dbhspZTq0CWF0soagg5ttha0kVkppTp2UtiSWUAcqdR4BUD4YLvDUUop23XopLAx3RoZlch48OzwNWlKKdWxk8KWtByGeqTjFaONzEopBR04KdTVGcozNuNNjbYnKKWUQ4dNCnsPlTCgeqe1EKVJQSmloAMnhQ3pR4j3SKU6sAcER9odjlJKuYUOnRRGeabipf0TlFKqXoe95Wbvvn1Ekaed1pRSqoEOWVIoKKsi5EiStaDtCUopVa9DJoVNGQXEe6RSJ14QOcLucJRSym10yKSwJbOAkR57oNtQ8AmwOxyllHIbHTIp3De5D2f5puGh7QlKKXWMDtnQLPkpSHWJticopdRxOmRJgcx11nO0lhSUUqqhjpkUAkJh4CUQ2tfuSJRSyq24vPpIRPyAbwBfx/4XGWMeF5HewEKgK7ARuMkYU+WUIAZdYj2UUkodw46SQiUwxRgzAogHLhKRccAzwN+MMf2BI8BtNsSmlFIdmsuTgrGUOBa9HQ8DTAEWOdYvAC53dWxKKdXR2dKmICKeIrIZyAW+APYABcaYGsdHMoGok3x3joisF5H1eXl5rglYKaU6CFuSgjGm1hgTD0QDY4DG5sI0J/nuPGNMojEmMTw83JlhKqVUh2Pr3UfGmAJgJTAOCBGRow3f0UCWXXEppVRH5fKkICLhIhLieO0PnAckAyuAqx0fuwVY4urYlFKqo7OjR3MksEBEPLGS0nvGmGUisgNYKCK/BzYBr9oQm1JKdWguTwrGmCRgZCPr92K1LyillLKJGNNoe26bICJ5QPpJ3g4DDrkwnDPlzvFpbM2jsTWPxtY8LYmtlzGm0Tt12nRSOBURWW+McdsR79w5Po2teTS25tHYmsdZsXXMsY+UUko1SpOCUkqpeu05KcyzO4DTcOf4NLbm0diaR2NrHqfE1m7bFJRSSp259lxSUEopdYY0KSillKrXLpOCiFwkIrtEJFVE5todT0MikiYiW0Vks4istzmW10QkV0S2NVjXVUS+EJEUx3MXN4rttyJywHHsNovIxTbF1lNEVohIsohsF5H7HOttP3aniM32YycifiKyVkS2OGJ7wrG+t4iscRy3d0XEx41ie11E9jU4bvGujq1BjJ4isklEljmWnXPcjDHt6gF4Yg3F3QfwAbYAQ+yOq0F8aUCY3XE4YpkAJADbGqz7EzDX8Xou8IwbxfZb4EE3OG6RQILjdRCwGxjiDsfuFLHZfuwAAQIdr72BNViDYb4HXOdY/y/gTjeK7XXgarv/zTniegB4G1jmWHbKcWuPJYUxQKoxZq+xpvNcCMywOSa3ZIz5Bjh83OoZWJMcgY2THZ0kNrdgjMk2xmx0vC7GGtAxCjc4dqeIzXbG4pYTbJ0iNrcgItHAJcArjmXBScetPSaFKGB/g+WTTthjEwN8LiIbRGSO3cE0opsxJhusEwwQYXM8x7tbRJIc1Uu2VG01JCKxWGN5rcHNjt1xsYEbHLuWTLDl6tiMMUeP21OO4/Y3EfG1IzbgOeAhoM6xHIqTjlt7TArSyDq3yfjAOcaYBGAacJeITLA7oDbkJaAv1tze2cBf7AxGRAKB94H7jTFFdsZyvEZic4tjZ1owwZazHR+biAwDfgUMAkYDXYGHXR2XiEwHco0xGxqubuSjrXLc2mNSyAR6Nlh2qwl7jDFZjudcYDHuNzJsjohEAjiec22Op54xJsfxH7cOeBkbj52IeGOddN8yxnzgWO0Wx66x2Nzp2DnicdsJthrEdpGjOs4YYyqB+dhz3M4BLhORNKzq8ClYJQenHLf2mBTWAf0dLfM+wHXAUptjAkBEOolI0NHXwAXAtlN/y+WWYk1yBG422dHRE67DFdh07Bz1ua8CycaYvzZ4y/Zjd7LY3OHYiRtPsHWS2HY2SPKCVWfv8uNmjPmVMSbaGBOLdT77yhhzA846bna3qDvjAVyMddfFHuDXdsfTIK4+WHdDbQG22x0b8A5WVUI1VgnrNqy6yuVAiuO5qxvF9iawFUjCOgFH2hTbeKyiehKw2fG42B2O3Slis/3YAXFYE2glYZ1cH3Os7wOsBVKB/wK+bhTbV47jtg34D447lOx6AJP48e4jpxw3HeZCKaVUvfZYfaSUUqqZNCkopZSqp0lBKaVUPU0KSiml6mlSUEopVU+TglKnICK1DUbI3CytOOquiMQ2HAVWKXfgdfqPKNWhlRtr6AOlOgQtKSjVDGLNi/GMYwz+tSLSz7G+l4gsdwygtlxEYhzru4nIYsd4/VtE5GzHpjxF5GXHGP6fO3rTKmUbTQpKnZr/cdVHMxu8V2SMGQO8iDUWDY7Xbxhj4oC3gBcc618AvjbGjMCaJ2K7Y31/4B/GmKFAAXCVk3+PUqekPZqVOgURKTHGBDayPg2YYozZ6xiA7qAxJlREDmENIVHtWJ9tjAkTkTwg2lgDqx3dRizWEM39HcsPA97GmN87/5cp1TgtKSjVfOYkr0/2mcZUNnhdi7bzKZtpUlCq+WY2eF7teP091kiWADcAqxyvlwN3Qv1kLsGuClKpM6FXJUqdmr9jNq6jPjXGHL0t1VdE1mBdXM1yrLsXeE1EfgnkAbMd6+8D5onIbVglgjuxRoFVyq1om4JSzeBoU0g0xhyyOxalWpNWHymllKqnJQWllFL1tKSglFKqniYFpZRS9TQpKKWUqqdJQSmlVD1NCkopper9P/i1kzmPcFkkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation ACC: 71.60%\n",
      "Test ACC: 72.37%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_acc(model=model,\n",
    "                           data_loader=test_loader,\n",
    "                           device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_acc(model=model,\n",
    "                            data_loader=valid_loader,\n",
    "                            device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training 2: Increasing Batch Size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(RANDOM_SEED)\n",
    "\n",
    "model = AlexNet(NUM_CLASSES)\n",
    "model.to(DEVICE)\n",
    "\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_sizes = np.arange(256, 5121, 512)\n",
    "batch_size_index = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/040 | Batch 000/188 | Cost: 2.3029 | Batchsize: 256\n",
      "Epoch: 001/040 | Batch 150/188 | Cost: 1.7115 | Batchsize: 256\n",
      "Epoch: 001/040\n",
      "Train ACC: 33.36 | Validation ACC: 33.00\n",
      "Time elapsed: 0.21 min\n",
      "Epoch: 002/040 | Batch 000/188 | Cost: 1.7286 | Batchsize: 256\n",
      "Epoch: 002/040 | Batch 150/188 | Cost: 1.6143 | Batchsize: 256\n",
      "Epoch: 002/040\n",
      "Train ACC: 44.84 | Validation ACC: 45.55\n",
      "Time elapsed: 0.42 min\n",
      "Epoch: 003/040 | Batch 000/188 | Cost: 1.5018 | Batchsize: 256\n",
      "Epoch: 003/040 | Batch 150/188 | Cost: 1.4893 | Batchsize: 256\n",
      "Epoch: 003/040\n",
      "Train ACC: 50.73 | Validation ACC: 50.55\n",
      "Time elapsed: 0.64 min\n",
      "Epoch: 004/040 | Batch 000/188 | Cost: 1.4247 | Batchsize: 256\n",
      "Epoch: 004/040 | Batch 150/188 | Cost: 1.2653 | Batchsize: 256\n",
      "Epoch: 004/040\n",
      "Train ACC: 56.70 | Validation ACC: 57.35\n",
      "Time elapsed: 0.85 min\n",
      "Epoch: 005/040 | Batch 000/188 | Cost: 1.0885 | Batchsize: 256\n",
      "Epoch: 005/040 | Batch 150/188 | Cost: 1.1472 | Batchsize: 256\n",
      "Epoch: 005/040\n",
      "Train ACC: 60.30 | Validation ACC: 58.45\n",
      "Time elapsed: 1.06 min\n",
      "Epoch: 006/040 | Batch 000/188 | Cost: 1.0394 | Batchsize: 256\n",
      "Epoch: 006/040 | Batch 150/188 | Cost: 1.0907 | Batchsize: 256\n",
      "Epoch: 006/040\n",
      "Train ACC: 62.62 | Validation ACC: 60.85\n",
      "Time elapsed: 1.27 min\n",
      "Epoch: 007/040 | Batch 000/188 | Cost: 1.0348 | Batchsize: 256\n",
      "Epoch: 007/040 | Batch 150/188 | Cost: 1.0401 | Batchsize: 256\n",
      "Epoch: 007/040\n",
      "Train ACC: 66.19 | Validation ACC: 65.60\n",
      "Time elapsed: 1.48 min\n",
      "Epoch: 008/040 | Batch 000/188 | Cost: 1.0627 | Batchsize: 256\n",
      "Epoch: 008/040 | Batch 150/188 | Cost: 0.9297 | Batchsize: 256\n",
      "Epoch: 008/040\n",
      "Train ACC: 64.11 | Validation ACC: 63.90\n",
      "Time elapsed: 1.70 min\n",
      "Epoch: 009/040 | Batch 000/188 | Cost: 1.0361 | Batchsize: 256\n",
      "Epoch: 009/040 | Batch 150/188 | Cost: 0.8127 | Batchsize: 256\n",
      "Epoch: 009/040\n",
      "Train ACC: 69.89 | Validation ACC: 65.45\n",
      "Time elapsed: 1.91 min\n",
      "Epoch: 010/040 | Batch 000/188 | Cost: 0.7913 | Batchsize: 256\n",
      "Epoch: 010/040 | Batch 150/188 | Cost: 0.7620 | Batchsize: 256\n",
      "Epoch: 010/040\n",
      "Train ACC: 69.22 | Validation ACC: 66.50\n",
      "Time elapsed: 2.12 min\n",
      "Epoch: 011/040 | Batch 000/188 | Cost: 0.8304 | Batchsize: 256\n",
      "Epoch: 011/040 | Batch 150/188 | Cost: 0.8406 | Batchsize: 256\n",
      "Epoch: 011/040\n",
      "Train ACC: 71.92 | Validation ACC: 68.50\n",
      "Time elapsed: 2.33 min\n",
      "Epoch: 012/040 | Batch 000/188 | Cost: 0.6939 | Batchsize: 256\n",
      "Epoch: 012/040 | Batch 150/188 | Cost: 0.9586 | Batchsize: 256\n",
      "Epoch: 012/040\n",
      "Train ACC: 73.86 | Validation ACC: 67.45\n",
      "Time elapsed: 2.54 min\n",
      "Epoch: 013/040 | Batch 000/188 | Cost: 0.7050 | Batchsize: 256\n",
      "Epoch: 013/040 | Batch 150/188 | Cost: 0.6281 | Batchsize: 256\n",
      "Epoch: 013/040\n",
      "Train ACC: 77.54 | Validation ACC: 70.90\n",
      "Time elapsed: 2.76 min\n",
      "Epoch: 014/040 | Batch 000/188 | Cost: 0.6453 | Batchsize: 256\n",
      "Epoch: 014/040 | Batch 150/188 | Cost: 0.6312 | Batchsize: 256\n",
      "Epoch: 014/040\n",
      "Train ACC: 76.89 | Validation ACC: 69.80\n",
      "Time elapsed: 2.97 min\n",
      "Epoch: 015/040 | Batch 000/188 | Cost: 0.6457 | Batchsize: 256\n",
      "Epoch: 015/040 | Batch 150/188 | Cost: 0.7908 | Batchsize: 256\n",
      "Epoch: 015/040\n",
      "Train ACC: 78.62 | Validation ACC: 71.50\n",
      "Time elapsed: 3.18 min\n",
      "Epoch: 016/040 | Batch 000/188 | Cost: 0.7273 | Batchsize: 256\n",
      "Epoch: 016/040 | Batch 150/188 | Cost: 0.5583 | Batchsize: 256\n",
      "Epoch: 016/040\n",
      "Train ACC: 80.89 | Validation ACC: 70.75\n",
      "Time elapsed: 3.39 min\n",
      "Epoch: 017/040 | Batch 000/188 | Cost: 0.5611 | Batchsize: 256\n",
      "Epoch: 017/040 | Batch 150/188 | Cost: 0.5131 | Batchsize: 256\n",
      "Epoch: 017/040\n",
      "Train ACC: 83.01 | Validation ACC: 71.25\n",
      "Time elapsed: 3.60 min\n",
      "Epoch: 018/040 | Batch 000/188 | Cost: 0.5365 | Batchsize: 256\n",
      "Epoch: 018/040 | Batch 150/188 | Cost: 0.4436 | Batchsize: 256\n",
      "Epoch: 018/040\n",
      "Train ACC: 81.85 | Validation ACC: 71.55\n",
      "Time elapsed: 3.81 min\n",
      "Epoch: 019/040 | Batch 000/188 | Cost: 0.4803 | Batchsize: 256\n",
      "Epoch: 019/040 | Batch 150/188 | Cost: 0.4372 | Batchsize: 256\n",
      "Epoch: 019/040\n",
      "Train ACC: 84.76 | Validation ACC: 73.60\n",
      "Time elapsed: 4.03 min\n",
      "Epoch: 020/040 | Batch 000/188 | Cost: 0.4905 | Batchsize: 256\n",
      "Epoch: 020/040 | Batch 150/188 | Cost: 0.4021 | Batchsize: 256\n",
      "Epoch: 020/040\n",
      "Train ACC: 85.10 | Validation ACC: 71.25\n",
      "Time elapsed: 4.24 min\n",
      "Epoch: 021/040 | Batch 000/188 | Cost: 0.4978 | Batchsize: 256\n",
      "Epoch: 021/040 | Batch 150/188 | Cost: 0.4828 | Batchsize: 256\n",
      "Epoch: 021/040\n",
      "Train ACC: 87.19 | Validation ACC: 72.75\n",
      "Time elapsed: 4.45 min\n",
      "Epoch: 022/040 | Batch 000/188 | Cost: 0.3978 | Batchsize: 256\n",
      "Epoch: 022/040 | Batch 150/188 | Cost: 0.4588 | Batchsize: 256\n",
      "Epoch: 022/040\n",
      "Train ACC: 87.93 | Validation ACC: 72.20\n",
      "Time elapsed: 4.66 min\n",
      "Epoch: 023/040 | Batch 000/188 | Cost: 0.3476 | Batchsize: 256\n",
      "Epoch: 023/040 | Batch 150/188 | Cost: 0.3774 | Batchsize: 256\n",
      "Epoch: 023/040\n",
      "Train ACC: 90.10 | Validation ACC: 72.35\n",
      "Time elapsed: 4.87 min\n",
      "Epoch: 024/040 | Batch 000/188 | Cost: 0.3039 | Batchsize: 256\n",
      "Epoch: 024/040 | Batch 150/188 | Cost: 0.4589 | Batchsize: 256\n",
      "Epoch: 024/040\n",
      "Train ACC: 89.20 | Validation ACC: 72.00\n",
      "Time elapsed: 5.09 min\n",
      "Epoch: 025/040 | Batch 000/188 | Cost: 0.2648 | Batchsize: 768\n",
      "Epoch: 025/040 | Batch 150/188 | Cost: 0.3186 | Batchsize: 768\n",
      "Epoch: 025/040\n",
      "Train ACC: 91.24 | Validation ACC: 72.55\n",
      "Time elapsed: 5.30 min\n",
      "Epoch: 026/040 | Batch 000/188 | Cost: 0.2093 | Batchsize: 768\n",
      "Epoch: 026/040 | Batch 150/188 | Cost: 0.3252 | Batchsize: 768\n",
      "Epoch: 026/040\n",
      "Train ACC: 90.77 | Validation ACC: 71.80\n",
      "Time elapsed: 5.51 min\n",
      "Epoch: 027/040 | Batch 000/188 | Cost: 0.3375 | Batchsize: 768\n",
      "Epoch: 027/040 | Batch 150/188 | Cost: 0.2307 | Batchsize: 768\n",
      "Epoch: 027/040\n",
      "Train ACC: 92.61 | Validation ACC: 73.15\n",
      "Time elapsed: 5.72 min\n",
      "Epoch: 028/040 | Batch 000/188 | Cost: 0.2307 | Batchsize: 768\n",
      "Epoch: 028/040 | Batch 150/188 | Cost: 0.2596 | Batchsize: 768\n",
      "Epoch: 028/040\n",
      "Train ACC: 90.78 | Validation ACC: 70.25\n",
      "Time elapsed: 5.94 min\n",
      "Epoch: 029/040 | Batch 000/063 | Cost: 0.2773 | Batchsize: 1280\n",
      "Epoch: 029/040\n",
      "Train ACC: 96.33 | Validation ACC: 75.60\n",
      "Time elapsed: 6.11 min\n",
      "Epoch: 030/040 | Batch 000/063 | Cost: 0.0958 | Batchsize: 1280\n",
      "Epoch: 030/040\n",
      "Train ACC: 96.87 | Validation ACC: 74.95\n",
      "Time elapsed: 6.28 min\n",
      "Epoch: 031/040 | Batch 000/063 | Cost: 0.1020 | Batchsize: 1280\n",
      "Epoch: 031/040\n",
      "Train ACC: 97.30 | Validation ACC: 74.40\n",
      "Time elapsed: 6.44 min\n",
      "Epoch: 032/040 | Batch 000/063 | Cost: 0.0750 | Batchsize: 1280\n",
      "Epoch: 032/040\n",
      "Train ACC: 97.54 | Validation ACC: 75.00\n",
      "Time elapsed: 6.61 min\n",
      "Epoch: 033/040 | Batch 000/038 | Cost: 0.0687 | Batchsize: 1792\n",
      "Epoch: 033/040\n",
      "Train ACC: 98.05 | Validation ACC: 76.20\n",
      "Time elapsed: 6.79 min\n",
      "Epoch: 034/040 | Batch 000/038 | Cost: 0.0607 | Batchsize: 1792\n",
      "Epoch: 034/040\n",
      "Train ACC: 98.19 | Validation ACC: 75.25\n",
      "Time elapsed: 6.96 min\n",
      "Epoch: 035/040 | Batch 000/038 | Cost: 0.0577 | Batchsize: 1792\n",
      "Epoch: 035/040\n",
      "Train ACC: 98.34 | Validation ACC: 75.00\n",
      "Time elapsed: 7.13 min\n",
      "Epoch: 036/040 | Batch 000/038 | Cost: 0.0546 | Batchsize: 1792\n",
      "Epoch: 036/040\n",
      "Train ACC: 98.30 | Validation ACC: 75.35\n",
      "Time elapsed: 7.30 min\n",
      "Epoch: 037/040 | Batch 000/027 | Cost: 0.0610 | Batchsize: 2304\n",
      "Epoch: 037/040\n",
      "Train ACC: 98.56 | Validation ACC: 75.15\n",
      "Time elapsed: 7.47 min\n",
      "Epoch: 038/040 | Batch 000/027 | Cost: 0.0544 | Batchsize: 2304\n",
      "Epoch: 038/040\n",
      "Train ACC: 98.78 | Validation ACC: 75.30\n",
      "Time elapsed: 7.64 min\n",
      "Epoch: 039/040 | Batch 000/027 | Cost: 0.0431 | Batchsize: 2304\n",
      "Epoch: 039/040\n",
      "Train ACC: 98.84 | Validation ACC: 76.75\n",
      "Time elapsed: 7.81 min\n",
      "Epoch: 040/040 | Batch 000/027 | Cost: 0.0455 | Batchsize: 2304\n",
      "Epoch: 040/040\n",
      "Train ACC: 98.84 | Validation ACC: 74.80\n",
      "Time elapsed: 7.98 min\n",
      "Total Training Time: 7.98 min\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "cost_list = []\n",
    "train_acc_list, valid_acc_list = [], []\n",
    "\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    ### INCREASE BATCH SIZE\n",
    "    if epoch > (NUM_EPOCHS//2) and not epoch % (NUM_EPOCHS//len(batch_sizes)):\n",
    "        train_loader = DataLoader(dataset=train_dataset, \n",
    "                                  batch_size=int(batch_sizes[batch_size_index]),\n",
    "                                  num_workers=4,\n",
    "                                  shuffle=True)\n",
    "\n",
    "        batch_size_index += 1\n",
    "    \n",
    "    \n",
    "    model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "        \n",
    "        features = features.to(DEVICE)\n",
    "        targets = targets.to(DEVICE)\n",
    "            \n",
    "        ### FORWARD AND BACK PROP\n",
    "        logits, probas = model(features)\n",
    "        cost = F.cross_entropy(logits, targets)\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        cost.backward()\n",
    "        \n",
    "        ### UPDATE MODEL PARAMETERS\n",
    "        optimizer.step()\n",
    "        \n",
    "        #################################################\n",
    "        ### CODE ONLY FOR LOGGING BEYOND THIS POINT\n",
    "        ################################################\n",
    "        cost_list.append(cost.item())\n",
    "        if not batch_idx % 150:\n",
    "            print (f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d} | '\n",
    "                   f'Batch {batch_idx:03d}/{len(train_loader):03d} |' \n",
    "                   f' Cost: {cost:.4f} | Batchsize: {batch_sizes[batch_size_index]}')\n",
    "\n",
    "        \n",
    "\n",
    "    model.eval()\n",
    "    with torch.set_grad_enabled(False): # save memory during inference\n",
    "        \n",
    "        train_acc = compute_acc(model, train_loader, device=DEVICE)\n",
    "        valid_acc = compute_acc(model, valid_loader, device=DEVICE)\n",
    "        \n",
    "        print(f'Epoch: {epoch+1:03d}/{NUM_EPOCHS:03d}\\n'\n",
    "              f'Train ACC: {train_acc:.2f} | Validation ACC: {valid_acc:.2f}')\n",
    "        \n",
    "        train_acc_list.append(train_acc)\n",
    "        valid_acc_list.append(valid_acc)\n",
    "        \n",
    "    elapsed = (time.time() - start_time)/60\n",
    "    print(f'Time elapsed: {elapsed:.2f} min')\n",
    "  \n",
    "elapsed = (time.time() - start_time)/60\n",
    "print(f'Total Training Time: {elapsed:.2f} min')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzddXzV1f/A8de5sYCNHF2jm9Et3SAGqICiCCgGXxCMHwIioigqFopBKoiEgoF0Tbq7O0bnqO3uxvn9cbexuNvu4i7Y+/l47MG9n3xT933P55zzPkprjRBCCJEYQ0YHIIQQIvOTZCGEECJJkiyEEEIkSZKFEEKIJEmyEEIIkSRTRgeQXAEBATowMDCjwxBCiCxlx44d17TWBVJ6fpZLFoGBgWzfvj2jwxBCiCxFKXUmNefLYyghhBBJkmQhhBAiSZIshBBCJCnL9VkIIVLOarUSEhJCeHh4RociPMTHx4fixYtjNpvT9LqSLITIRkJCQvD39ycwMBClVEaHI9KY1prr168TEhJC6dKl0/Ta8hhKiGwkPDyc/PnzS6J4SCmlyJ8/v0dajpIshMhmJFE83Dz195utksXG49c4evlORochhBBZTrZJFntDbtFryhbafbU2o0MRIltTStG7d+/o9zabjQIFCtClSxcA/vnnH8aNG5foNS5cuED37t0B+Pnnnxk4cGCyYvj444+TPKZPnz788ccfybpuSuzevZvFixd7/D6plW2SxZaTN6JfW+2ODIxEiOwtZ86c7N+/n7CwMABWrFhBsWLFovd37dqVYcOGJXqNokWLpuqD3J1kkV4kWWQyLzUrE/36yh1LBkYihOjYsSOLFi0CYPbs2fTs2TN6X8yWQp8+fRg0aBCNGzemTJky0Qni9OnTVKtWLfqcc+fO0aFDBypWrMgHH3wQvf3xxx+nTp06VK1alUmTJgEwbNgwwsLCqFmzJs8++ywAM2bMoEaNGgQFBcVq9axduzbeveNyde6ZM2do3bo1NWrUoHXr1pw9exaA33//nWrVqhEUFESzZs2IiIhg1KhRzJ07l5o1azJ37tzU/cF6ULYaOvvFU0G8+fsewq32jA5FiAz3wcIDHLxwO02vWaVoLt5/tGqSx/Xo0YMxY8bQpUsX9u7dS9++fVm3bp3LYy9evMj69es5fPgwXbt2jX78FNPWrVvZv38/OXLkoF69enTu3Jm6desybdo08uXLR1hYGPXq1aNbt26MGzeO7777jt27dwNw4MABxo4dy4YNGwgICODGjRtu3zuhcwcOHMjzzz/PCy+8wLRp0xg0aBB//fUXY8aMYdmyZRQrVoxbt27h5eXFmDFj2L59O999953bf84ZIdu0LADy5nROUrkbbsvgSITI3mrUqMHp06eZPXs2nTp1SvTYxx9/HIPBQJUqVbh8+bLLY9q2bUv+/Pnx9fXlySefZP369QBMmDCBoKAgGjZsyLlz5zh27Fi8c1evXk337t0JCAgAIF++fG7fO6FzN23aRK9evQDo3bt3dDxNmjShT58+TJ48Gbs9a31pzVYtCz9vZ7K4I8lCCLdaAJ7UtWtX3nrrLYKDg7l+/XqCx3l7e0e/1lq7PCbucFGlFMHBwaxcuZJNmzaRI0cOWrRo4XL+gdY6weGmSd07sXNdxffjjz+yZcsWFi1aRM2aNaNbN1lBtmpZ+Hk7c+NdizWDIxFC9O3bl1GjRlG9evVUX2vFihXcuHGDsLAw/vrrL5o0aUJoaCh58+YlR44cHD58mM2bN0cfbzabsVqdnwOtW7dm3rx50Qkr5mOopCR0buPGjZkzZw4As2bNomnTpgCcOHGCBg0aMGbMGAICAjh37hz+/v7cuZP5h/Rnq2Th7+NMFtKyECLjFS9enMGDB6fJtZo2bUrv3r2pWbMm3bp1o27dunTo0AGbzUaNGjV47733aNiwYfTxL7/8MjVq1ODZZ5+latWqjBgxgubNmxMUFMTQoUPdvm9C506YMIHp06dTo0YNZs6cyTfffAPA22+/TfXq1alWrRrNmjUjKCiIli1bcvDgwUzfwa0SatZlVnXr1tUpXfzo5r0Ian24gvcfrcKLTdK2booQWcGhQ4eoXLlyRochPMzV37NSaofWum5Kr5mtWhY5Ix9DfbDwYAZHIoQQWUu2ShZmo9TEEUKIlMhWySLmqIWLoWEZGIkQQmQt2SpZxNRlwvqMDkEIIbKMbJcsKhbyB+D6vYgMjkQIIbKObJcsjkiJciGESLZslyxiCg2zYndkraHDQmR1RqORmjVrUq1aNR599FFu3brlkfs0btzYI9fNrrJ1sgj6YDn9ftmW0WEIka34+vqye/du9u/fT758+Zg4caJH7rNx40aPXDctZLW6UJDNkwVA8JGrGR2CENlWo0aNOH/+PADBwcHRCyCBs3Lrzz//DEBgYCDvv/8+tWvXpnr16hw+fBiA0aNH07dvX1q0aEGZMmWYMGFC9Pl+fn7R123RogXdu3enUqVKPPvss9F1nhYvXkylSpVo2rQpgwYNinX/KKdPn+aRRx6hdu3a1K5dOzoJPfPMM7HWoejTpw/z58/Hbrfz9ttvU69ePWrUqMFPP/0UHUfLli3p1atXdIkTVyXUAaZOnUqFChVo0aIFL730UnTJ9qtXr9KtWzfq1atHvXr12LBhQyr+9JMnWxUSFELEsGQYXNqXttcsXB06Jr7KXRS73c6qVavo16+fW8cHBASwc+dOvv/+e8aPH8+UKVMAOHz4MGvWrOHOnTtUrFiRV199FbPZHOvcXbt2ceDAAYoWLUqTJk3YsGEDdevWZcCAAaxdu5bSpUvHWlMjpoIFC7JixQp8fHw4duwYPXv2ZPv27fTo0YO5c+fSqVMnIiIiWLVqFT/88ANTp04ld+7cbNu2DYvFQpMmTWjXrh3woJR66dLOChKuSqhbLBY+/PBDdu7cib+/P61atSIoKAiAwYMHM2TIEJo2bcrZs2dp3749hw4dcuvPL7UkWQgh0lXUwkOnT5+mTp06tG3b1q3znnzySQDq1KnDggULord37twZb29vvL29KViwIJcvX6Z48eKxzq1fv370tqh7+/n5UaZMmegP7p49e8b6dh/FarUycOBAdu/ejdFo5OjRo4BzAadBgwZhsVhYunQpzZo1w9fXl+XLl7N3797oxZJCQ0M5duwYXl5e1K9fP/p+4Kwh9eeffwJEl1C/dOkSzZs3jy53/tRTT0Xfc+XKlRw8+KACxe3bt7lz5w7+/v5u/RmmhiQLIbIrN1sAaS2qzyI0NJQuXbowceJEBg0ahMlkwuF4sORx3HLiUeXCjUYjNpst3nZX+xI7xt26eF999RWFChViz549OBwOfHx8APDx8aFFixYsW7aMuXPnRrdMtNZ8++23tG/fPtZ1goODyZkzZ6z3rkqoJxaXw+Fg06ZN+Pr6uhV7Wsr2fRZCiIyRO3duJkyYwPjx47FarZQqVYqDBw9isVgIDQ1l1apVHr1/pUqVOHnyJKdPnwZIsOJraGgoRYoUwWAwMHPmzFid0z169GD69OmsW7cuOjm0b9+eH374IboE+tGjR7l3757L67oqoV6/fn3+++8/bt68ic1mY/78+dHntGvXLtaKeum5HoYkCyFEhqlVqxZBQUHMmTOHEiVK8PTTT0eXDq9Vq5ZH7+3r68v3339Phw4daNq0KYUKFSJ37tzxjnvttdf45ZdfaNiwIUePHo3VOmjXrh1r166lTZs2eHl5AdC/f3+qVKlC7dq1qVatGgMGDHDZ2kmohHqxYsUYPnw4DRo0oE2bNlSpUiU6rgkTJrB9+3Zq1KhBlSpV+PHHHz3xR+NStipRDhA4bFG8bafHdU5NSEJkGVKiPLa7d+/i5+eH1prXX3+d8uXLM2TIkIwOKzoum83GE088Qd++fXniiSfcPj9LlShXSpVQSq1RSh1SSh1QSsVb5UQ5TVBKHVdK7VVK1fZUPEIIEdfkyZOpWbMmVatWJTQ0lAEDBmR0SIBzSHDUxMXSpUvz+OOPZ3RIHu3gtgFvaq13KqX8gR1KqRVa65iLSXQEykf+NAB+iPxVCCE8bsiQIZmiJRHX+PHjMzqEeDzWstBaX9Ra74x8fQc4BBSLc9hjwAzttBnIo5Qq4qmYhBC4PQpIZE2e+vtNlw5upVQgUAvYEmdXMeBcjPchxE8oKKVeVkptV0ptv3o1dTOuv+vl2U4zITIzHx8frl+/LgnjIaW15vr169HDe9OSx+dZKKX8gPnAG1rr23F3uzgl3r9irfUkYBI4O7hTE0/n6kUYyK7UXEKILKt48eKEhISQ2i9dIvPy8fGJNykxLXg0WSilzDgTxSyt9QIXh4QAJWK8Lw5c8HBMnry8EJma2WyONYNYCHd5cjSUAqYCh7TWXyZw2D/A85GjohoCoVrri56KSQghRMp4smXRBOgN7FNKRU0zHA6UBNBa/wgsBjoBx4H7wIsejEcIIUQKeSxZaK3X47pPIuYxGnjdUzG4y2Z3YDLKZHYhhEiIfEICyw9ezugQhBAiU8s+yeLKYVj5AUTEL+g1f0dIBgQkhBBZR/ZJFjdPwfov4eJeWlYsEGvXqsNX+CH4RAYFJoQQmV/2SRZFI8tOXdjJ9Bfrx9v96dLD6RyQEEJkHdknWfgXglzF4fyOjI5ECCGynOyTLACK1YLzOxPc3Wf6VimDIIQQLmSzZFHH2Xdx/wZ9m8SfxRp85CrhVoeLE4UQInvLXskiRr/Fu50q0a9p/IThkJaFEELEk82SRU3nr+d3YTYaeK9LlXiHhFnt8bYJIUR2l72ShU9uCKiQaCf3jE1n0jEgIYTIGrJXsgDno6jzOyCBx00RNumzEEKIuLJfsihWB+5dgdvnXe62OyRZCCFEXNkvWRSv4/z17GaXu2P2WYSGWbkYGpYeUQkhRKaW/ZJFkZrgmw+OrXC5+9fNZwmPTBgtxwfT6JPV6RmdEEJkStkvWRiMUL4tHF0Ktgj8veNXaf8++ASXb4dz415EBgQohBCZT/ZLFgBVn4TwW3ByDXYXHd0TVh2jwcerMiAwIYTInLJnsijbCrxzw4G/KODvneThf+8+z/Erd9IhMCGEyJyyZ7IweUHlLnD4X2b3qZ7k4YPn7KbNl2vTITAhhMicsmeyAKjVGyy3KXpmIW+0KZ/R0QghRKaWfZNFyYbO2dwHFjC4deqSxeS1Jwk+ciWNAhNCiMwn+yYLpaBSZzi9ARV+K1WXGrv4EH2mb0ujwIQQIvPJvskCoNKjoO1wZGlGRyKEEJla9k4WRWtB3kDY+Ytbh7f6IpgTV+96NiYhhMiEsneyMBig/gA4u4ntfXInefjJq/do/cV/6RCYEEJkLtk7WQDUfh58chOw+4eMjkQIITItSRbeflDvJTi8iHIqJKOjEUKITEmSBUDD18AnF6NMM4Gkl1Ut8+4iPl58yPNxCSFEJiHJAiBnfmjxLs2M+2ht2Jnk4Q4Nk9aeTIfAhBAic5BkEaVefyLyluc906+Ysbl1SuCwRYSGWT0cmBBCZDxJFlGMZrw6jSPQcJnBpvlun7b7XPwJfZXeW8KT329Iy+iEECJDSbKIqXwb/rQ34RXjQsqoC26dYjaoeNvCrQ52nk3drHAhhMhMJFnEMdb6HGF4M8I0y63jDS6ShRBCPGwkWcRxjdx8b3uM1sZd1FTHkzz+29XH0iEqIYTIWEkmC6WUMT0CyUxm2NtyTedihtcneJP40qobjl9PcN+WkwnvE0KIrMSdlsVxpdTnSqkqHo8mE9g7uh3r3uvKjUc+IJcKY5hpNi0qFkjRtZ6ZtDmNoxNCiIzhTrKoARwFpiilNiulXlZK5UrqJKXUNKXUFaXU/gT2t1BKhSqldkf+jEpm7B6Ry8dMvpxe5G3wLHNsLehpXM0zVXzdOnfj8WsEDlvk4QiFECL9JZkstNZ3tNaTtdaNgXeA94GLSqlflFLlEjn1Z6BDEpdfp7WuGfkzxu2o00EBf28m2zvjo6zUufy7W+f8JBP1hBAPKbf6LJRSXZVSfwLfAF8AZYCFwOKEztNarwVupFWgGeGELsZSez0K7JtCYZLuf/jv6NV42/7efd4ToQkhRLpy5zHUMeAx4HOtdS2t9Zda68ta6z+A1K4a1EgptUcptUQpVTWhgyIffW1XSm2/ejX+B7KnHP6wA48MnITSdkaa3RtKG9fgObu5cS/xTnIhhMjs3Oqz0Fr301pvjLtDaz0oFffeCZTSWgcB3wJ/JXSg1nqS1rqu1rpugQIp62xOCR+zkZyFykDToXQxbqaR4UCKrvPUj/H+6IQQIktxJ1kUVEotVEpdi+yw/lspVSa1N9Za39Za3418vRgwK6UCUntdj2gyiLOOAnxsmkIBkj8z+8TVe/zfH3tZf+yaB4ITQgjPcydZ/AbMAwoDRYHfgdmpvbFSqrBSSkW+rh8ZS+acmGD25W3rKxRVN3jPPDNFl5i7/RzPTd2SxoEJIUT6cCdZKK31TK21LfLnV9xY9EEpNRvYBFRUSoUopfoppV5RSr0SeUh3YL9Sag8wAeihtU56MYkMskVX5kf7o3Q1buJp45qMDkcIIdKVyY1j1iilhgFzcCaJZ4BFSql8AFprlyOetNY9E7uo1vo74LvkhZuxvrU9Tm11lPdNM9joqEqILpjRIQkhRLpwp2XxDDAAWAMEA68CfYEdwHaPRZbJ+JqN2DCxMHA4DgyMM01G4Ujx9dYfu8bNexHsOnuTM9fvJXjcvpBQLt8OT/F9hBAiLSTZstBal06PQDI7FVlcdtRz7Rk7phefmKfynv6VMbbnk32tcKud56ZuIah4bvaEhAJwelxnl8c++t16vE0GjnzUMcWxCyFEaiWZLJRSZpytiWaRm4KBn7TW2WqJuKhC5FprZttbUVGdo69pKcvs9diiKyfrWhars0Vy9PJd9463pbwFI4QQacGdx1A/AHWA7yN/6kRuy1by+XkBoJSiQiF/xtl6cs5RgIle35CX225fp9lna3h++lZPhSmEEB7hTgd3vciJc1FWR45gylZmv9SQ/45exc/bxPIhzQF44t2zzPMawyjzTIZYX3frOmdv3OfsjfsA6KQHlQkhRKbgTsvCrpQqG/UmckKe3XMhZU7F8+bg2QalYm3bpcsz0f4YTxg30M6wLdnXDLcm/nhpxJ/7kn1NIYTwBHeSxds4h88GK6X+A1YDb3o2rKxjou1xDjhK8Zl5Es0NKW9wNRm3mo/+PRhr26wtZ1MbnhBCpIlEk4VSygCEAeWBQZE/FbXWMistkhUT1zpN4bLOy2TzeBqoQym6zvlbYUxZf4op66TMuRAi80k0WWitHcAXWmuL1nqv1nqP1tqSTrFlGb6FyvFUxCjO6MJM8vqCCupciq/10SLXySaxuRhCCOFp7jyGWq6U6hZVx0k8EBA5QgrgNn70iXgHC17M8fqQUupSiq97/MqdeNuafx6MxZbtuoqEEJmEO8liKM7igRal1G2l1B2llPtjRR9iSwY345+BTaLfn6cAPSNGoIAp5i/IRcpaA22+XMuivRfjbXfIdAshRAZxZ1lVf621QWvtpbXOFfk+yTW4s4MC/t7UKJ6HmPUPT+hivGYdTKC6xBfmHzCksCTI67/tjLdN2nZCiIzizrKqq9zZlp3ZHLHnS2xyVOVz29O0Ne7kHdOcNL2XxWbnzXl7uBgalqbXFUKIxCQ4KU8p5QPkAAKUUnl5UPEiF851LUSkqkXjN7Qm2R8lUF3iFdO/HHAEstDROE3utebwFebvDOFOuJVJz9dNk2sKIURSEmtZDMBZWbZS5K9RP38DEz0fWtaRJ8eDju7Haz7Iox/YXmCXoxxjzVMphMtK7skSs0bU+VthZOLlP4QQD5kEk4XW+pvIirNvaa3LaK1LR/4ERa5FIVwY/9SDyigWvHjD+hre2JjqNZ4AQlN17c+WHiYqPxy4cJup60/F2n/2+n0en7iB0PvZqsajECIduNPB/a1SqrFSqpdS6vmon/QILisyGmL3Qp/RhXnH+hLVDKdZ6D2CVob4HdfumrXlLB8veTAPY+up2K2V79YcY/e5WyzZH38klRBCpIY7HdwzgfFAU6Be5I88LE+Aq+kofzua8qRlNOHazDSv8Qwx/ZHiUVLnbjzo2I56CHXwwm0mrz3JtbsRAPzw34kUXTshRy7d4TcpPSJEtuZO1dm6QJXMvD52ZvB2+4p8vuxIgvt36gp0iPiUseZpDDYtoJ9xMQOsQ9jgqJ7ie16JXEGv04R1ABT09wbgzPX7CZ4TcvM+p67d45HyBdy+T/uv1wLQq0HJlIYqhMji3JmUtx8o7OlAsrrXW5ZLcLW7KBa8eMs6gBHWvtgwMsvrE74yT8SfhD/cExO1yl6UK3ceVGKx2l23XNp+uZbeU2U9DSFE8riTLAKAg0qpZUqpf6J+PB1YVlapsD/DOlZKYK9ilr0NDSwTmWtrwRPGDSzzfifFfRmXQl2vz33f4ro0SJhVSoYIIZLPncdQoz0dxMNm6RvOFWh/3nCaS7ddf5hb8OL/bC/zu70ZE70mMM1rPGvt1elvfYsIzG7fq+EnMj9SCOF5CbYslFKVALTW/wGbtdb/Rf0AUnnWDQteS3oi3nZdia6Wj7ik89LMuI9FXsOprY6m+t5D5+0mcNgigo9cAeB+hI1fN59J9Byr3UGEzYHdoVm6/5LM4xBCREvsMdRvMV5virPvew/E8tApmseXl5uVibe9QiG/WNsvk4+Glon0iXibnCqMX70+4Rlj6pYMWXXYmST6TN/GXYuNv3ZdYORf+6P3rz92Ld45zT5bQ4WRSxg0Zxev/LqDhS6KGQohsqfEkoVK4LWr9yIBwzpU4v1Hq0S/D8yfg+VDmjO8U+V4xwY7avG45UN2O8ryqXkyw0y/QRqs0/3i9K1sPxN7TsZzU7fEO+5iZP9HVMXba3csRNik1K0QIvFkoRN47eq9SIDBoKhcxFk7alCrcqx+s0Wix18hL72t7/KbrSWvmP5lo/f/KKMupCqGbadvsmDneZf77oRbqfvRSraddl2OZNTf++NtO3v9PjOTeKQlhHi4JNbBXVwpNQFnKyLqNZHvi3k8sodIwzL5WTiwKVWL5sJgSLpRZsfIcFt/DupAPjJP5zvzt7xsHUqIdn9uhDvuWWws2Hmea3ct9Pt5W7z9Y+KsCR7lqZ82cvm2hafqFMfHbEzTmIQQmVNiyeLtGK+3x9kX971IQvXiuZN5huJXe1uu6dz86PU1670HM9r6PD/b25NWTwGrvr8s+vXtcJvb592S2lNCZDsJJgut9S/pGUh25u9j4k4CH9ZLHfXpE/EO35knMNo8g67GjbxtHcAJnfGNO7tDnkYKkV24MylPeMjGYa1Y9kYzkhqhGuyoSXXLFEZZX6CsusDfXu/xlDEYbyLSJ9A4okqlfxKjqKEQ4uEmySIDFc3jS8XC/jQplz/JYzUGZtjb08HyKUd1cT43T2Kj9/943/QLxbiaDtHGny2+eN+leMdMXnuS2Vul6KAQDxtJFpnANz1q0bthqej3IzvHH1Yb5SL56R4xmlcjBnPQUYoXTctY7f0mI00zqaWO4cmBak/9tDHW+xv3IrDaHZy78aC21djFh3h3wT6PxSCEyBjulCj/TCmVSyllVkqtUkpdU0o9lx7BZRc+ZiOl8ueIfp9UdVcHBpY4GtDbOpzmli9Z7qhLf9MS/vR+nzleH1GAmx6J89yNME5evRtr25iFB3nkszXM2XrW7Rnf45cdkZLnQmQx7rQs2mmtbwNdgBCgArFHSok0ULtU3ujXBqXo37R09Pu6MfbFdUYX5n/WQTxpGc1vtlbUU4f52/s96qrDHomz1Rf/xXofNd9i2IJ9Cc7liOu7NccZ/mfCrY8b9yJYd+wqG0/En2UuhMgY7iSLqKp2nYDZWmu3FpNWSk1TSl1RSsWf1eXcr5RSE5RSx5VSe5VStd2M+aFUu+SDhKAUNCkXEP2+X4zEkZCdugLDbf3pFvEBFm1mltfHdDCkbynymEUTd0TOGL8YGkaHr9dyOYGCijFNWnuCwGGLqP3hCnpP3UqvyfFnmQshMoY7yWKhUuowzkWQVimlCgBJ/8+Hn4EOiezvCJSP/HkZ+MGNaz7UopZkNShFy0oFeadDRR4pH0CFwv5uX2O3LscTEWPYp8vwrflbuhv/I70m3Mdc/KnbD85yYr9tOcvhS3eYu+1ckud/vNgzrSEhROq5swb3MKARUFdrbQXuAY+5cd5aILFWyGPADO20GcijlCriXtjZw2styjGzXwPKFvBj1ZvN3T7vFv70jXibnbo8480/8Zt5LAW5icKBkay1nsXcbQn3bVy/ayEsImv9foTIqtzp4H4KsGmt7UqpkcCvQNE0uHcxIObXzRASKCOilHpZKbVdKbX96tX0GSaaESb2qkVQiTyYXJQE8Y0sq1Ekt49b17pNTnpEjORHWxcaGw+y1ed1Tvk8xx7vlxho/BOVwjXAk6P31C3sOx+a9IGJ+L/5+7hyx3VDts5HK+n+40aX+4QQacudxY/e01r/rpRqCrQHxuN8ZNQglfd2VbPC5fMSrfUkYBJA3bp1H9ppwx2qFaFDtbRrXGkMjLP1Yo29Fo8Z1+OvwiiqrvOW+XdaGXcx294KL2z8YW+GBa80u2+UdTHKoEcNlLLYkt8ScCSS1w5cuJ3s6wkhks+dPouo/92dgR+01n9DmnyyhAAlYrwvDqSuvOpDrKC/Nw3L5OPLp2vG27dnVLtEz92iKzPc9hL/sw6iW8Rohln7U9twnM/NkxhrnsYRnz58b/6aJoZ95HCrOyr5vlp5lG2nb3Dz3oO6Umev30drzXerj8UbkhuTkoL4QmQ4d1oW55VSPwFtgE+VUt6kzWS+f4CBSqk5OFspoVprWW0nASajgTkvNwJgVJcqhFntfL7sCBN71SZ3DveXYQXFHHsr5tub0dywh6LqGiNNv9LJuJVOxq2EaS8W2hvxnu3FNG9tvPrrDsoE+EW/v2OxcuyKnfHLjzJ+eeKrA14KDSdfTi9MBkWTT1fzToeK0fs+W3qYdzoktOa5ECItuJMsnsY5qmm81vpWZCd0kvMslFKzgRZAgFIqBHifyGG4WusfgcU4h+MeB+4DL6bkN5Ad9Y0cSvt6y3IJHtO/aWmmrAj96MQAACAASURBVD+V4H4rJlY66gAww96ePNyhvuEwrQy76GEKpqThCi9HDOE2fgleI7mu3Y3g2t0HYx60dq8YYYTNwSOfreHxmkUZ160GF0PDY80S/z74hCQLITzMndFQ94ETQHul1ECgoNZ6uRvn9dRaF9Fam7XWxbXWU7XWP0YmCiJHQb2utS6rta6utZay52mgeYUCvN2+IkPbVUjwmN9faRRv2y38We6oxzDby3xi7UlDwyF2er/C/5lmE0DqOqkTss7F0q6uRNidnRbLD172SBxCiKS5MxpqMDALKBj586tS6n+eDkwkz9bhrcmTw8yb7Srwesty5PAyMbyT62/b9QLzJXqtn+yP8mLE26xzVOdV00L+836DRw1pP+ro06WHCbkZlubXFUKkPXceQ/UDGmit7wEopT4FNgHfejIwkTwFc/mwO05Hd05v519vq0oFWX34CgBlAnK6db01jlqscdSimu0ko80z+NbrO3rZV7POUZ11jurs14HoNOi6emlG0g3KNZGxx6RkGXgh0pU7/9sVxJrJZSetlmoT6aJQrgdzM2b2T96I5/26DH0i3mGpvR5VDad5xzyXhd4j2eP9Mo0N+0mP2eHTEul7ibLzrGeKJwohnNxpWUwHtiil/ox8/zgw1XMhibSn+eqZIGqXzEuxPL7JPvsuOXjFOgTQlFPnGWL6g87Grfzm9TEHHaUYZ+vBWkdQ2ocd6ULkOhr3I+yEhjmH3uo4SerJ7zdyelxnl+ffs9jI4WVEyRhcIVLMnQ7uL3GOVLoB3ARe1Fp/7enAROrFfFTzRK3ilMr/4BHUtD51U3TF47o4r1vfoHL4NEZaX8SXcGZ4fUp/46J0KSXS4ONVAIRbE5+Bfvb6fULvW5mx6TRV31/GuCWpqzt1/a6FTSeup+oaQmRliSYLpZRBKbVfa71Taz1Ba/2N1npXegUn0oarZSZaVSqUYAe4O8Lw4Vd7WzpEfMpWR0VGmmdxwqc3P5q/orwKSUW0aaPZ52sIGrOcUX8fAOCntSdTdb2ekzfTc/LmtAhNiCwp0WShtXYAe5RSia/GIzKljtUKU6mwP680L5vAftelRcxG9x/XWPCiV8QIRlj7cl9708Swn3+9RtDKsDNFMWdWRy87Z5hfuR3ONyuPcf2uJYMjEiJ9udPBXQQ4ELlK3j9RP54OTKRe3pxeLH2jGYEJjIAqkS8Hn3evEX973hwujk6YDROz7G2oYplOC8uXHNHFmeY1ntM+vXjEsDdFsafG1TuuP8gnrT3BsgPx1w0HsNrdK6xY/+NVfLXyKMNSuXTsPYuNyWtP4nBjUqIQmYE7HdwfeDwKkSl88mR13l2wj251itOncSD1xq7kfjJLgF8nN70j3mWYaTY9TWuY6TWOLY5KrLXX4IwuRGF1g9LqElsdFfnb0TRN44+wOfAyGTgbY03wmKLWyzjyUQfsDk0OL+c//0+WHOKn/07y9+tN2H8hlIL+PrStUijRe6W2NPqnSw8zY9MZiuX1pVN1qcwvMr8Ek4VSqhxQSGv9X5ztzQD31s8UWUbZAjnpWb8k3esUx2RQKKXw9zElO1kAhOLHu7aXGG17gVdN/9DDuIa3zfNiHfMsq2hv384pXZg19pps16kv1xE1QiqpQU8VRy4FnEnD22Tkp/+c/Rl/7jrPzxtPAyQ4siqt3Am3ARBulfU4RNaQWMvia2C4i+33I/c96pGIRIYIKpEHALPxwZNJQ4xP3XqBedl2OnlzGSx48bWtOxNsT1LfcBgfLFzTubmo8/OReRqdjM5lX183/cNlnYe59hbMs7ckRBdI0e9h0Oxd/NS7rtuTgCw2B94mY/R7R5yRAKev3SOXr5l8OdO+fLsQWU1iySJQax3vgbPWertSKtBjEYl0lTeH84Mw5sS9KFEfuiuHNqdYHl8qj1oavW/PqHZsOnkNh4bXZiXeme3AwGZHlVjbXrUOoaLtLLUNxxho+ou83GWQ6S/6G5cwyd6ZH2xdk131dtmBy/y5K4Qhc/e4dbwC5u9IeORWi/HB+Hmb2Dqidbx964+7V9cqKa5GqgmRGSXWwZ3YkmzJn9klMqXWlQvyTY+aDGkTv/BgVHXbonl88PUy8lSd4gDUL52P3DnMdKhWhE7Vi1DA3ztF9z6iSzLb3pomlm+pYplOU8vX7HSU4w3TAo749GGQcUGyr+luogCw2jVv/v7g+B1n4rec7lpsPDdlS7LjEOJhk1iy2KaUeinuRqVUP2CH50IS6UkpxWM1i+Fliv9Pof8jZTg9rnN0R/C4bjXo26Q0Xz0TewGmfwY2SZNHNSG6IM9ZR9A/4k0Ahpr/YIjpDzxVUuSTxYdivY+56t7xKw8WY9p59layrht85Aoj/3owWuqvXecTHGr7794LXJNhuCILSCxZvAG8qJQKVkp9EfnzH9AfGJw+4YnMxGhQjHq0SrySIUVy+9K8grOfoXud4uTwMsbaP71PvWTdZ6WjDpXDp3HAUYrBpgXM8fqI2irxxZFSYvG+hNfaWn046XLogcMW8b/ZD+aoztl6lie+30Cf6dv4dfNZwLlo0xtzdzNgpuvvV2uOXHWrmKIQGS3BZKG1vqy1boxz6OzpyJ8PtNaNtNauB6uLbK9x2fyUzBd7nkbLSgXJ5ePOKO0HwvChS8RYvrY9SRV1hgXeo3nbNAcD7s2HSK2oYbZJWbjnwUrAwxbsY1ecVsjJa84WynYXj7iiXAr1zFK2QqSlJP8Ha63XAGvSIRbxkOhasyiHlx5J9XU0Br62dWearQMzvD7lddM/9DKu5rCjJCE6gGO6GL/Y26f58q/JNW/7Ob50sSys1pol++R7lXg4JO/rnhAJqF0qL3/uOk/pgJyUK+jHZ6Q+WUS5jR+PR4yhvWE7LQy76Wl68N2ln2kJv9ub842tG9Zk/nO+l8qJdVHe+cP1LPU3f98T75GcK1ILV2QFqV+9RgjguQYlWft2S2qVzOtyOGidUnlTeQfFMkc93rW9RGD4LCqG/8xLEUO5r70ZaPqbhV4j6GlcRVl1nvRYY8MdC3ZmzNzVedvOsS/EM0vhiuxLkoVIE0opSuZ39lWUK+gX3UfRtFwAAN/1qp2Wd8OCFyscdWkb8TmvRgwml7rHJ+aprPJ+m93eL7PM6x1eNi7EC2sa3jclkbrXbrDaHbT/aq3LVQGT6535e3n0u/Wpvo4QMcljKJHmcnqb2Du6Paeu3aNQLu/obV4mAzWL52FGv/pUem9pEldxjw0TSxwNWGmpQx3DUUqriwSpE/QwBTPcMJu+pqVMs3VguaMuF3X+dO/fuBXmXrK6dtfCkct3eHfBPjYPjz8JEMBmd3DjfgQF/RObAiWEZ0jLQnhM6YCc0XM0AI5+1JF5rzTCxxz/Of6U5+uycGDswoI96pVw+15WTGx2VGG2vTXDbC8TGD6LVyMGc8pRhOHm2QR7v8kG70G8bvyLRoYD6dbiiDlaKiFKKRp9sjrydcLHvff3fuqPXcX9CFtahSeE26RlITIFo0FRvXhuDnzQHptdM2X9SV5qVobVh69wJYGS44lTLHE0YImjQXRZkY6GrbEKGm5xVKKBwTlE9inLKHboCjjS6ftT8JEHj5ti1qQyxMkWN+9F4Otl5NpdC8sOOOd+VBm1jLfbV+T1luXSJVYhQJKFyCB9GgdGV3iNKae385/km+0qujxvw7BWNBm3Oln3OqJLcsTuLC1SwXaOAaaF5OUuedVdDjtKUMlwjt+9xwDwm60lY23Pcc8DFW2u3AnHZDCwaN9Fbt5/0LKJlSxi5Kq/d59n8Jzd5PY1ExpmjTVL/vNlR5JMFla7I1ZhSCFSQ5KFyBAjOlfmmXol6PjNOrfPqV0yD4Xi1KHqUqMI/+5NeCZ2XEd1Cd60vhZrW35C6WjcyhDTH/QyreEp41q+tT3BD/auyR6Om5j6Y1e53H759oOWU1SH+Mbj11h5yNn6CI3s97hxLyJZ9xv4205+6p2StdaFiE++dogMYTYaqFwkV/R7ncRw15eblWHBa00wxfimXK1YLr54OijVsVwnN7/a21LH8hNPWkaz0VGVoeY/OObzPKu83mSsaWrkkFzPMyg4c/0evaZsSbK/48a9CH7dfCbB/VGPrYRIC9KyEFlC3yal423rUqMoxkR6hLvUKMKt+9ZklRPfqSvwgvX/6GjfyjPGYPxUGD2Nq3nWtIoZtrbkV6GRHemtsHngv49BKU5du+fWsW/M3c3ao1epG5iXYnl8qT56OR8+Xi3NYxICpGUhMljLiokvdFStWG4AvF1UxVWAyWhgx8g2LB70SLz9WkP7qs7lUZ9rWDIZUTk7x/tY/4/uEaNpYpnACnsdnjOupJlhHx+af+a4z/Ps9e7PG6Y/qKzOpFnNKqWgz/Rtbh17LbLj32bXXL7trC/13l/7Ezw+cNgixiw8mPogRbYkLQuRof6vYyWu3Y2gfun8LvdP6FmLQxdvk9dFCfSoRkV+P+/oZUpjKpk/R/TDLYViwWuNefL7jcmO8SL5ecn6Jj5YCMeLlobd1DccprlhL2+YFvCGaQG7HWX5yPosh3SpVHWOq6TWhI3h4MUHJdXfmLs70WNXHXI+kpq24RQVCvmRJ4eZjSeuM2PTGcY+UY1nG5RKWcAi25BkITJUpcK5WPi/pgnu9/M2US8wn8t9MYeZBrhYgGlo2wr8tsVZKlwpqF0ydSVHwnHeY42jFmsctficZ6iozlHHcJR3THP4I3JE1R3tyzx7Cz60PUdyKz+ltE7U/vO3E93fP0YZ9GEL9sXaN+LP/S6ThdaaiWuO83TdEhR0sZKiyF7kMZTIsnrWf/Boyc/bxH9vt4i132w0oN1ct/TP1xrj723i0aCibt/fgYFDuhS/2tvS3PIVb0a8wgJ7U67rXPQzLeFX88e0NWxHJeMR1fVkjniCxCfyRa3ZkZLlW/edD2X88qMMmZd4q+V2uJVFiYxIe/L7DXy21L2S7yLzkmQhsqy4M8GL5fElwM/5uKp/5JKw7aoWxsdsiP7m/ONzrmtU1SqZl30ftOfxms5kUTiZ36Rvkov5jmYMtb5Gy4gv+NLanSDDSSZ7fckG70F0Mmx2a9Z4cofHAvT9OeE+jqTWR0+M1e5Mcvcj7By7fIcvVxx1mXyHzt3D67/t5OTVu/H2gXOlwe+DT6Q4DpE5SLIQWU5C36RNRgPbR7bl9LjOjOxSBYCieXw5/GFHKhb2B6BDtSKc+qRTgteuUMh53DsdYk8KbFjG9aMwVzQGJtifpLblJ96IeA0HBr73msBRnxdY6zWYUaYZlFYXSavquDHnaaSlqLxgUIoekzYzYdUxbsfpG7pxL4KNJ5yjzb5YcVQWcnqISZ+FyHJm9WvAb1vPYkjhA36lFFtHtEZrOHjhdvSHHUCJfDk48XEnjAbF0Hl7HmzPm4PN3EjWfayY+MvRlGWWujQz7OVV00KKqav0NS2lr2kpV3QeltjrsclRlYs6Hxa8OKyTM2rLM6asO0m9wHz8d/QqADvO3MQvcmZ93JZF88/XcD9yXZBFey9yKTSc+a82Tt+ARbrwaLJQSnUAvgGMwBSt9bg4+/sAnwNRM56+01pP8WRMIutrXC6AxpGlz1MqqnJroVw+tKxUMNY+Y0qzUALC8GGZoz7LIuoDUFxdoa1hB52NW3ja+B8vmFZEH7vBXpW/HY1ZYa/DTXIldMlkefv3PUkec+t+BHlyeLHn3C0+WnQo3v5wqzMhxH0KFXcU2o4zN9FaJ2tUl8gaPJYslFJGYCLQFggBtiml/tFaxx3oPVdrPdBTcQiRUrP6N+DZKVtSdY2gEnnYcy72utwhuiDT7R2Zbu+IGRvV1UnyqLs0Mhyks3EznxknYzdNIdhRk7WOGuxxlOWgLkUE5hTF8PuOkCSPefXXnfRsUJLhcUZKRbE5nFniw38PcvDibZa+0SzBa924F0F+v/ij00LvW8mdI2W/B5HxPNmyqA8c11qfBFBKzQEeA2RWkMgSmiTQevHzNnHX4l6ZcHMSrRQrJnbqCqBhtaM2Y23PUlWdoZNxMy8Yl9PauCvW8WccBfnO/ji/21u4dX93bTp5nU0nryd53IJdzocAF0PD8DG5XjL2ie83svadlvG2B41ZzulxnVMXqMgwnuzgLgaci/E+JHJbXN2UUnuVUn8opVwuYKCUelkptV0ptf3q1aueiFWIJBXN7YPRoPjtpQZunzO8c+Vk3kVxQAfyua0HdSw/0jh8AoMjXuMb2xOE6ADu4svn5km8Z5pJB8NWAtVFjNgpyjWMpM2a4u5o9Mlq6o1d6XLf2Rv3+XL5Ee6EJzz66+odC2uOpH5VQJF+PNmycPWVKu7wj4XAbK21RSn1CvAL0CreSVpPAiYB1K1bN3MssCyyFaVg7Tst0eCy7HfxvL6E3AyLfj+0bQV+CD5B7ZJ5Ccyfg9PX7yf7nha8uEAAfzuaggO+sj2FETtfmyfSz7SEfiwBwKYNmJSDcG1mjaMm020d2KdLE4ZnJ9JFPZpyZcLq4+w6d4uZ/Vwn1p6TN3P8yt3owQQi8/NkyyIEiNlSKA7EKqOptb6utY4a9zcZqOPBeIRIMa2dQ3OjEsWg1uX587UHo37+er1J9Ouvn6nJoNblOfRhBwB6NUi7EU52jPzPOoh64RPpZnmfD63P8ZO9CyOtL7LY0YBWht3M8/6QQz59GWj8k5LqMmk1RDe51h27Fq91MXurc0b9icg5GVprNp24zj03H+slx7T1pzh6+U6aXze78mSy2AaUV0qVVkp5AT2Af2IeoJQqEuNtVyD+MAwhMtBjNV3P6B7atgK1YpQPyeXj7LhtW6UQj9eK/bS1f9MySQ4nLVsgZ7LiukpeduiKTLV34nNbD361t2Wo9TWaWr5hvt1ZVPEt8++s9R7CZu+BfGKazGDjfAIITdZ9UuulGGVGAN5dsI/AYYuiR1VdDA2n5+TNDI0zS3zlwcs889MmFuxMunM+3GrH7qKVM+bfg3Se4P56KSJxHksWWmsbMBBYhjMJzNNaH1BKjVFKdY08bJBS6oBSag8wCOjjqXiESIke9ZytgqpFEx/G6mUyEPxWC77tWSvePoNBUadUXt7tWCnB89OqtPhV8vCm9VUCw2fR1fIhY6y92e8I5GljMEPM81nr/QYfmabS1bCBAtxMk3smZvPJxOem3ItcT/zIpTtcvWPh4AVnjav+M7az5dQNhs7bEz1sN0rbL/9j2vpT0e8rvbeUEX+6HsVltT9IIiE378c6TySPR+dZaK0XA4vjbBsV4/W7wLuejEGI1GhUNj+LBz1C5SL+SR4bGJB462BA87L8tvUsZ+L0X3z0eDUal3Vv3kj/pqWZ4tYHnmKvLstee1mm2TsCUFadZ5DpT3oY1/Ccyblq3yp7LT6zPcORDJoMGNXCuHLHwiOfrSbc6uCDrlUTPefYlbuM+fcgVruDT5Y4a07N2XaOcd1qxLjugySx/tg16pfOR++pWzl17R6P1SzqcmivSJzM4BYiCVUSaVX89XoTjl9xXRPJlZxezv9yk5+vS34/LwJyelMyfw4AXmtRlu+DTzCsYyWerF3M5TKsr7Us52ayiO+ELsZg60BG0Ycy6iItjbt4wbicf71GEOyoyUldmH/sjTmg4y805SlfrjgKED0LHJxzOWJyaM2RS3coWyBnrJUSoxKFKzEnDz43dQsvNgmMLkUSbkubtUeyG0kWQqRCzRJ5qFkij9vHT36hLn/vPk+bygXjzXJ+p0Ml3ukQ/1HViY87sfvcTaZvOE0e39RPagvFj126PLts5Zlm68hH5ulUU6doZtjDANMijjqKscFRjRvan5v485u9NQ4PPbFecTD+0q9xR1l9tvQIP288Tb+mpXkvsuZXUuL2YJy4eo+wyMdZk9eeZHQSrRcRnyQLIdJRsTy+vNainFvHbh/ZBofWGA2KOqXyUaeU+8UMi+Xx5fytsCSPu4U/A62DAMjFXR4zbqSvcQnPGlfipZwfrsNMs1nhqMMMWzt26fJux5BWzlx3LjO740zifSwxy4wkVpre5pCWRUpIshAikwpI4XP1t9pV4LGaxXjkszXJOu82fsy0t2OmvR2g8SOMtoYdtDTuprVhF094b+C8zs8mR1UW2hthwsYFHcAhXZKUL9uUtO2RSSKpAcAnrt6lXEF/l8e6u66JSJgkCyGyqNPjOnPXYqPa+8tibR/Yyv1v///+ryldvl3vYo/iLjn40/EIfzoeIRd36WdaShV1mscMG+huXBt95AFHKWbZ27DLUY4QXYA75Ejpb8ml6GKFSXzgOzQs2BlCx2pFuBiacKtKa9hw/Bq5fMxUL547LUN9qEmyECILiyod7q4nahUjX04vpkZ2klcr5t6H5W38+MrWHYAchNPYcAADDgqpm7xu+puPzVMBsGojqxy1WeeozhWdh2s6N17Y2KXLpbgQYpQ9IaEEDluU4P5l+y/xxYqjsUrLR3HESDSztpxlVuRyuwCHP+wQayGtP3aEUCKvLw3KuF4XPruSZCFEFrNwYFNWHHrQMTz7pYaE2+x4mwzRpdcT4tCaIW0rRCeLlLiPDysdD4ot/GpvQw11kkqGc1RSZ3nGGEwHY+zV+y7ofKyy12afLs1OR3lO6KLoNO40/yJyZJUrG44nXCTx793nKZLbl2YVCgDwVmRJ97hFD3/ecIrRCw/Su2EpPny8GmERdkb/c4DhnSpni2q6kiyEyGKqF88d6/FJo7KJfwNuVCY/wztV5tHv1lMvMF+yWyNJ0RjYo8uxx+7suB9re5bS6hK+WAhQoQSoULoYNtPb9KDw4C2dk22OipzSRZhrb8EJ7arGaPr4v/nOCX1Fc/sws3/CRSJHL3QO6Z25+QyPBhXl4IVQ5m4/h6+XMVuMrpJkIcRDzmRUVC+em43DWlEkd/yWx8mPO1Fm+OJ42397qQHjlx1h59lb8fYlxoaJY7q4803k05959pZ4WyMooy5Sx3CU6uoUdQ1HaGvcSX/jYlY5arHZUZnVjtqc0wUwY/N4IcS4LoSG88HC+CsoWGx2wq2xR1ANmbs7er13RzbpPJdkIcRD7vPuQYBzPfIoSsVYY9tF1ddFg5pStWhu5r+an9Lvxk8kKWHBi0O6FIfspaK35eM2/U2L6WFcTVvjTt5jFhZtwlvZOOYoxjZHRSbaHuM8BdIkhqSsPfpgCYR528/xdN0SNPpkNTfuRcQ67vytsOihyZIshBBZ2u+vNCIswk5hF62JFUOaszck4RZD1aLOx1yeXh71Brn4zNaDz2zPUDJyudmS6jK38KOKOkM34zq6GDczxdaJU7owCs1dfDngCOQy7s87SYl3/tjLO3/sTfK4RCq1P1QkWQjxkKoXmPCHabmCfpQr6Bf9PrevmWfqlWDVoct816t2su/VsVphluy/lKI4nRRndSGm2jvF2lpCXWa8+SeGmv+Itd2hFdfJxVpHdebZWnJMF+NGGq1ZnlxRZUQediqrTVapW7eu3r59e9IHCiHSxNHLdzAoaPPlWpf7T4/rzKS1J/h4ccK1mlJHU4xr5FAWHCgKqls0NhygpLpCG8MOcioLDq04ootjQHNYl+S4oygrHXU4qYtgwctDcT2QFZaLVUrt0FrXTfH5kiyEEO64dT8Cfx8zaw5foX/kOhWVCvuz9I1mOBzaZSe5K3tGtSNozPI0iSk3d2lgOERVwxlqq6MoNKXUFYqpaxiUxqEVp3RhNjuqsNNRnu26Amd04TS5d0zZIVnIYyghhFvy5HB+Q69TKi8mgyJfTi++eqYm4Owk//nFengZDfSasiXeuZUK+3P40h2algtIcE5CUPHc5PAyselkwnMi4grFj+WOeix31Iu1vQA3aWbYRwnDFWqokzxpXMezplVYtZGVjtrsdJTntC7MJZ2PMLzIzT0O6VLcT+cRWFmJJAshRLLkzenF8Y87xdveomLByF8LEHzkKo+UD+Do5Ttcvm3h5xfr4+9jwsvknIj3aFBRFu6JtcoybasUov8jZaj03tLobWUL5KRHvZKMXZy8RTSvkpf5jmYQOeLVmwiKq6v0NS6ltXEnHeNMGgQI12aO6WLcx4cTjiLMtzdjh66YrPs+zCRZCCHS1ItNShN8xDkENeopt1KQM8ZkQF9z7Nnbf7/ehOrFcmMwKErmy8HZG/eZ/2pjqhbNhY/ZGC9ZvNysDJPWnnQ7JgtenNDFGGHrxwhbPwpwk5LqCnnVXfKqO9zQ/jQz7KWcuoC3stLFuJlepjXMsLVlrO3ZdOn3yOwkWQgh0lQuH+fHSvG8OThy6Y7LY0Z0rsK87c71tT/tVp2gGGuCjH8qiPHLj1CjeG7MRtclQVI7ovcqebmq88YqT7sqRgkTX8IZavqDl0yLCTKc4NWIN7iAe6sZPqw8tga3ECJ7qlUyLz8+V4f3H014oaLcvma2j2zD8iHNeKZe7CVd65fOx7wBjRJMFLP6N0B5sCQ6QBg+jLU9x4CIIZRRF/nZ61Py4DrxAdy12DwaT2YgyUIIkeY6VCuMj9mY6BoUAX7eVCiU9Nrm4GxtRGlSLoAXmwRSu2QeHikf/9v+L33rs+C1xuwY2SZ62+EPO7D6zeZuxx9lmaMeL1uHUkpd4Wevz8iJ69Lnr/66I9nXzmokWQghPOa9LlXw9zGRN0fqnvl3r1OcV1uUZfLzzpGfhXL5sOC1Ji4XiArMn4PaJfOSP8Y+H7ORMgX84h3rjk2Oqgy0/o9q6hQ/mb/EiD3eMeuOXUvRtbMSSRZCCI/pGlSUfaPbR4+CSo3/61CJtlUKxdoW9TAqZoHEUvlzRr9e81YLNr/b2q3r50mkzPgKR13etfWnqfEAb5jmuzym0zfr+Ojf+IUIHxbSwS2EyLKGtK3AxdBwJj1fhyt3LDjiFGoqHZDT5XktKxZgXLcabD55ncFzdgPQqmJBhneujM2uOX/rPt1+2BTrnN/tLaijjvI/01+c0wWYZ28Za//Bi7c5ePE2IzpXJuRmGCXype2KgRlNWhZCvcUIegAACtVJREFUiCyrRL4czH65If4+ZsoW8KN8En0g20a0YUDzMvzYuw6FcvnwWM1i/DOwCQANy+QnwM+bwrl9qFMqH0VjtFZ61i/J0jceoUCPiay1V+dj01T29rDGa+kAvP/PAR75bA07ItcOf1hIuQ8hRLZ35XY4Bfy9Y1XZnbr+FB9GPlYa0KwM73aq7NxhuQszusKVQxzp9Dvt54YmeN3MVAYkteU+pGUhhMj2CubyiVeOvV/T0swb0AhwtjqieftBj9ngm5cKawZQgIRLvY/8ax9nrt+j7PDFzNt2jnnbzhE4bBGj/zngkd+HJ0nLQgghEnEn3Iq/j4vO74t7YFoHdlqK0jNiZLJneR8f2xG71nibjGkUaeKkZSGEEB7kMlEAFAmCJ36ktuE4q7zf4j3TTKqrk5Do7JIHOk1YR8WRS5M+MJOQZCGEEClV5THo/SeWPOV5zriShd4jWeb1f/QzLiYftxM99ejlu+kUZNqQx1BCCJFKDoem5vDf6WzczNPG/6hlOE6ENhLsqMm/9kYsd9QhnPgTCAFebVGW0vlz0rBMfvx8TOTNYcahwehibfTUkMWPhBAiE7hrsVHt/WUAbHupKH9P/5Quxs0UVjexaDNbHRUJdgTxr72RW+uHn/qkU5qugS7JQgghMokl+y6Sw9tE8woFOHTxNna7nUqWvfw8/QceMeyjoiEEmzaw3lGdYEcQ2xyVOKRL4nDRI+DvbeKDx6py7kYYDcvko3LRXORKqP/EDZIshBAiizh6cDerfvuCTobNlDJcAeCu9qGZ5WtukCvJ8w980D7WuiDJIcuqCiFEFlGhSk0qfDSTE1fu0OjLP3m19CXun9vLDdyrvlv1/WUZNtFPkoUQQqSzsgX92TTu+ej3/ewOwq12rt+NwGwy0GTcapfndapeOL1CjMejyUIp1QH4BjACU7TW4+Ls9wZmAHWA68AzWuvTnoxJCCEyG7PRgNloiJ7TEdV6sNkdbD19gwEzd1Asjy/f9KiVYTF6LFkopYzARKAtEAJsU0r9o7WOWcO3H3BTa11OKdUD+BR4xlMxCSFEVmIyGmhcNoB9o9tndCgenZRXHziutT6ptY4A5gCPxTnmMeCXyNd/AK1VWo4VE0IIkSY8mSyKAedivA+J3ObyGK21DQgF8sc5BqXUy0qp7Uqp7VevXvVQuOL/27v3GLnKMo7j319aWhCQtrSaJiW2TeqlGqVNQSpIGiWNNMQYo2HRxEY0KorXGNJKQqJ/oSYGTYzQKGpMQQQEmwYtBKhEDL1Q2rK11i5SY8OlNQjeouHy+Mf7bDid7u7Zbnd25sz8PsnkvOedM6fv05zZZ86ZM89rZjaadiaLkc4QWu/THc82RMSGiFgRESvmzZs3KYMzM7Pxa2eyOAycU1lfADw12jaSpgNnAc+1cUxmZjYB7UwWO4AlkhZJmgEMAJtattkErM32h4AHomm/EjQz6wNtuxsqIl6SdDWwhXLr7M0RsU/SN4CdEbEJ+BHwM0lDlDOKgXaNx8zMJq6tv7OIiHuAe1r6rqu0/wt8uJ1jMDOzk+f5LMzMrFbjCglKOgr8ZYIvnwv8bRKH0w0cUzP0Wky9Fg/0fkxviIgJ307auGRxMiTtPJmqi93IMTVDr8XUa/GAY6rjy1BmZlbLycLMzGr1W7LY0OkBtIFjaoZei6nX4gHHNKa++s7CzMwmpt/OLMzMbAKcLMzMrFbfJAtJ75N0QNKQpHWdHs9YJN0s6YikwUrfHEn3STqYy9nZL0nfy7j2Slpeec3a3P6gpLUj/VtTQdI5kh6UtF/SPklf7IGYTpW0XdKejOnr2b9I0rYc321ZFw1JM3N9KJ9fWNnX+uw/IKmjs9xImibpMUmbc73p8RyS9Lik3ZJ2Zl9jj7scyyxJd0j6Y76nVk5JTBHR8w9KbaongMXADGAPsLTT4xpjvBcDy4HBSt+3gHXZXgd8M9trgF9Tyr1fAGzL/jnAn3M5O9uzOxTPfGB5ts8E/gQsbXhMAs7I9inAthzrL4CB7L8RuCrbnwVuzPYAcFu2l+bxOBNYlMfptA4ee18BbgE253rT4zkEzG3pa+xxl+P5KfDJbM8AZk1FTB0JtgP/uSuBLZX19cD6To+rZswLOTZZHADmZ3s+cCDbNwFXtG4HXAHcVOk/ZrsOx/YrynS7PRET8BpgF/BOyq9lp7ced5SCmiuzPT23U+uxWN2uA3EsAO4H3gNszvE1Np789w9xfLJo7HEHvBZ4krw5aSpj6pfLUOOZta/bvT4ingbI5euyf7TYujLmvFyxjPJJvNEx5SWb3cAR4D7Kp+jno8z62Dq+0WaF7KaYbgCuAV7J9bNpdjxQJlO7V9Kjkj6VfU0+7hYDR4Ef5+XCH0o6nSmIqV+Sxbhm5Guo0WLrupglnQHcCXwpIv4x1qYj9HVdTBHxckScS/lEfj7wlpE2y2VXxyTpMuBIRDxa7R5h00bEU3FhRCwHLgU+J+niMbZtQkzTKZeofxARy4B/Uy47jWbSYuqXZDGeWfu63bOS5gPk8kj2jxZbV8Us6RRKotgYEb/M7kbHNCwinge2Uq4Jz1KZ9RGOHd9os0J2S0wXAu+XdAj4OeVS1A00Nx4AIuKpXB4B7qIk9SYfd4eBwxGxLdfvoCSPtsfUL8liPLP2dbvqrIJrKdf9h/s/lnc9XAC8kKehW4DVkmbnnRGrs2/KSRJloqv9EfGdylNNjmmepFnZPg24BNgPPEiZ9RGOj2mkWSE3AQN5d9EiYAmwfWqieFVErI+IBRGxkPL+eCAiPkpD4wGQdLqkM4fblONlkAYfdxHxDPBXSW/KrvcCf2AqYurUF08d+GJoDeUunCeAazs9npqx3go8DbxI+QTwCcr14PuBg7mck9sK+H7G9TiworKfK4GhfHy8g/FcRDnF3Qvszseahsf0duCxjGkQuC77F1P+OA4BtwMzs//UXB/K5xdX9nVtxnoAuLQLjr9VvHo3VGPjybHvyce+4fd9k4+7HMu5wM489u6m3M3U9phc7sPMzGr1y2UoMzM7CU4WZmZWy8nCzMxqOVmYmVktJwszM6vlZGF9S9K/crlQ0kcmed9fa1n//WTu32yqOVmYlaKNJ5QsJE2r2eSYZBER7zrBMZl1FScLM7geeHfOefDlLBD4bUk7cg6ATwNIWqUyL8ctlB84IenuLFK3b7hQnaTrgdNyfxuzb/gsRrnvQZV5Fi6v7HtrZZ6CjfnLd7OuML1+E7Oetw74akRcBpB/9F+IiPMkzQQelnRvbns+8LaIeDLXr4yI57Lkxw5Jd0bEOklXRyky2OqDlF/gvgOYm695KJ9bBryVUqPnYUq9pt9NfrhmJ85nFmbHW02pp7ObUkr9bEqNI4DtlUQB8AVJe4BHKIXZljC2i4Bbo1SsfRb4LXBeZd+HI+IVSkmUhZMSjdkk8JmF2fEEfD4ijimsJmkVpSR0df0SyuQ+/5G0lVIzqW7fo/lfpf0yfn9aF/GZhRn8kzLd67AtwFVZVh1Jb8yqpa3OAv6eieLNlBLlw14cfn2Lh4DL83uReZQpdDtSldXsRPiTi1mp3vlSXk76CfBdyiWgXfkl81HgAyO87jfAZyTtpVRYfaTy3AZgr6RdUUp9D7uLMj3pHkol3msi4plMNmZdy1Vnzcysli9DmZlZLScLMzOr5WRhZma1nCzMzKyWk4WZmdVysjAzs1pOFmZmVuv//6vJ0Oy2FFgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cost_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(cost_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3dd3hUZdr48e+dThqBhIROqFJiaAGpimLBCiqK2EBR7PXnruyur6776i67r20tq4IIuCrFgmBDXYQVpIcSunQICakkpNfn98cZQoA0ksmcSXJ/rmuumTlz5pw7Rzn3PM95nvuIMQallFIKwMPuAJRSSrkPTQpKKaXKaFJQSilVRpOCUkqpMpoUlFJKlfGyO4C6CAsLM5GRkXaHoZRSDUpsbGyqMaZVRZ816KQQGRnJxo0b7Q5DKaUaFBE5XNln2n2klFKqjCYFpZRSZeotKYjIhyKSLCLbyy1rKSI/ichex3MLx3IRkTdFZJ+IxInIgPqKSymlVOXq85rCHOBt4KNyy6YBy4wx00VkmuP9s8DVQHfH4yLgXcfzeSsqKiI+Pp78/Pw6hK7K8/Pzo3379nh7e9sdilKqntVbUjDG/CIikWctHguMcryeC6zASgpjgY+MVYhprYiEiEgbY0zi+e43Pj6eoKAgIiMjEZHahq8cjDGkpaURHx9P586d7Q5HKVXPXH1NIeLUid7xHO5Y3g44Wm69eMeyc4jIVBHZKCIbU1JSzvk8Pz+f0NBQTQhOIiKEhoZqy0upJsJdLjRXdAavsHyrMWaGMSbGGBPTqlWFw2w1ITiZHk+lmg5Xz1NIOtUtJCJtgGTH8nigQ7n12gMJLo5NKaXqTX5RCWk5haRlF5CWXUhWQTFFxaUUl5ZSVGIoLimluNSUvS41YBy/jY0p9yvZcbuD0b0i6NshxOlxujopLAEmAdMdz4vLLX9UROZjXWDOrM31BHeQlpbG6NGjATh+/Dienp6catGsX78eHx+fardxzz33MG3aNC644IJK13nnnXcICQnhjjvucE7gSqnzVlJqSMsuIOlkAclZ+Wc8p2QVkJZjJYC07AJyCkuctl8RCA/2a1hJQUTmYV1UDhOReOAFrGSwUESmAEeAWxyrfwdcA+wDcoF76iuu+hYaGsqWLVsA+POf/0xgYCDPPPPMGesYYzDG4OFRce/d7Nmzq93PI488UvdglWrEsguKOZ6ZR1Z+MdkFxWTnF5PleM4usB5Z+cVk5ReVvc52vM9yfC6Ap4fg6SF4eXiUvfb0EIwxpOcUUlpBR3dogA+tgnwJC/SlY0d/QgN8CQ30ISzQp+x1kJ8XXh4eeHkKPp4eeHlar70dyzwd3banem9d1Y1bn6OPJlby0egK1jVAoz7L7du3j3HjxjFixAjWrVvHN998w4svvsimTZvIy8tjwoQJPP/88wCMGDGCt99+m6ioKMLCwnjwwQf5/vvv8ff3Z/HixYSHh/Pcc88RFhbGk08+yYgRIxgxYgQ///wzmZmZzJ49m2HDhpGTk8Pdd9/Nvn376N27N3v37uWDDz6gX79+Nh8NpZzrZH4R249lsuPYSbYdy2T7sUwOpuVQ1Y0lfbw8CPbzIsjPmyA/LwJ9vQgL8yfIz5tAX+u9CBSXGkpLDcWlhhLHo9iRCcICfQgP9iM8yJfwIF8igv0IC/TFx8tdLteevwZd+6g6L369g50JJ526zd5tg3nh+j61+u7OnTuZPXs27733HgDTp0+nZcuWFBcXc+mllzJ+/Hh69+59xncyMzO55JJLmD59Ok8//TQffvgh06ZNO2fbxhjWr1/PkiVL+Mtf/sLSpUt56623aN26NV988QVbt25lwACdE6gaHmMMmXlFpGQVkJJdQGp2IamO10fTc9l+LJNDabll67dp7kdUu+aM69+OTqH+BPt5E+g46Z96BPh6NegTd31q1EnB3XTt2pVBgwaVvZ83bx6zZs2iuLiYhIQEdu7ceU5SaNasGVdffTUAAwcOZOXKlRVu+6abbipb59ChQwCsWrWKZ599FoC+ffvSp0/tkplSrnAip5DfkrLYm5zNvuRsfkvK4mBqDqnZBRSVnPuT38tDaN3cj6i2zRk/sD1R7ZoT1a45YYG+NkTfeDTqpFDbX/T1JSAgoOz13r17+ec//8n69esJCQnhzjvvrHAuQPkL056enhQXF1e4bV9f33PWMVW1nZWySUmpYX9KNnHxmWyLz+C3pGz2JmeRml1Ytk6AjyfdIoIY2jW0rEsmLNCHVoG+ZX31zZt54+Ghw6WdrVEnBXd28uRJgoKCCA4OJjExkR9++IExY8Y4dR8jRoxg4cKFjBw5km3btrFz506nbl+p6hhjOJKey9b4TOKOZhB3LJMdxzLLRuL4+3jSIyKIy3qG0z08iO4RgXSPCKJtcz+dH2MTTQo2GTBgAL179yYqKoouXbowfPhwp+/jscce4+677yY6OpoBAwYQFRVF8+bNnb4fpSpSVFLK+HdXszU+E7Au7PZuE8z4ge2Jbh9CdPvmdGkViKf+2ncr0pC7GGJiYszZN9nZtWsXvXr1siki91JcXExxcTF+fn7s3buXK6+8kr179+Lldf6/BfS4qvO1aHM8Ty3YyhOju3NF7wh6RATpxV03ISKxxpiYij7TlkIjlp2dzejRoykuLsYYw/vvv1+rhKDU+TLG8N6KA/SICOSJ0d21778B0TNEIxYSEkJsbKzdYagmaPmeZPYkZfHarX01ITQw2pZTSjnduyv20y6kGdf3bWt3KOo8aVJQSjnVhkPpbDh0gvtHdsbbU08xDY3+F1NKOdV7K/bTwt+bWwd1qH5l5XY0KSilnGbP8SyW7U5m8rDO+PvoJcuGSJOCk40aNYoffvjhjGVvvPEGDz/8cKXfCQwMBCAhIYHx48dXut2zh9+e7Y033iA393QNmGuuuYaMjIyahq5Unb3/3/34+3hy99BOdoeiakmTgpNNnDiR+fPnn7Fs/vz5TJxYWdHY09q2bcvnn39e632fnRS+++47QkKcX29dqYrEn8hl8dYEJg7uSIuA6u8botyTJgUnGz9+PN988w0FBQUAHDp0iISEBPr168fo0aMZMGAAF154IYsXLz7nu4cOHSIqKgqAvLw8brvtNqKjo5kwYQJ5eXll6z300EPExMTQp08fXnjhBQDefPNNEhISuPTSS7n00ksBiIyMJDU1FYDXXnuNqKgooqKieOONN8r216tXL+6//3769OnDlVdeecZ+lDofH6w8iIfAfSM72x2KqoPG3en3/TQ4vs2522x9IVw9vdKPQ0NDGTx4MEuXLmXs2LHMnz+fCRMm0KxZMxYtWkRwcDCpqakMGTKEG264odL6Lu+++y7+/v7ExcURFxd3Rtnrl19+mZYtW1JSUsLo0aOJi4vj8ccf57XXXmP58uWEhYWdsa3Y2Fhmz57NunXrMMZw0UUXcckll9CiRQv27t3LvHnzmDlzJrfeeitffPEFd955p3OOlWrQSkoNM1ceYFt8Jn+8thftQppVum5adgHzNxxhXL92tGle+XrK/WlLoR6U70I61XVkjOGPf/wj0dHRXH755Rw7doykpKRKt/HLL7+UnZyjo6OJjo4u+2zhwoUMGDCA/v37s2PHjmoL3a1atYobb7yRgIAAAgMDuemmm8pKcHfu3Lnspjvly26rpi3+RC4TZ65l+ve7+WHHcca88QuLtxyrdP25qw+RX1TKA5d0cWGUqj407pZCFb/o69O4ceN4+umny+6qNmDAAObMmUNKSgqxsbF4e3sTGRlZYans8ipqRRw8eJBXXnmFDRs20KJFCyZPnlztdqqqb3Wq5DZYZbe1+0gt3nKM577ajjHw6i19iYlswVMLtvDE/C0s353Mi2OjaN7Mu2z9nIJi5q45zJW9I+gWHmRj5MoZtKVQDwIDAxk1ahT33ntv2QXmzMxMwsPD8fb2Zvny5Rw+fLjKbVx88cV88sknAGzfvp24uDjAKrkdEBBA8+bNSUpK4vvvvy/7TlBQEFlZWRVu66uvviI3N5ecnBwWLVrEyJEjnfXnqkYiM6+IJ+Zv5on5W+gREcT3T4zk5oHt6RQawMIHhvL0FT34Oi6Ra/65krUH0sq+N2/9ETLzinhwVFcbo1fO0rhbCjaaOHEiN910U1k30h133MH1119PTEwM/fr1o2fPnlV+/6GHHuKee+4hOjqafv36MXjwYMC6g1r//v3p06fPOSW3p06dytVXX02bNm1Yvnx52fIBAwYwefLksm3cd9999O/fX7uKmoDEzDzeWb6PYD9verYJpmfrIDqHBZwz03jdgTSeXriV4yfzefqKHjw8qite5dbx8vTg8dHdGdk9jKcWbGHizLU8cHFXHrusGx+sPMiQLi0Z0LGFq/88VQ+0dLaqET2uDc/q/ak89ulmsgqKy248D+Dj6UG38EB6tgmiZ+sgUrMLmbnyAB1b+vPGhH70r+bknlNQzEvf7mTe+qOEBfqSml3A3HsHc0mPVq74s5QTuF3pbBF5ArgfEGCmMeYNEWkJLAAigUPArcaYE3bEp1RDZoxhxi8H+PvS3XRpFciCB4bSsaU/+1Oy2X38JLuPZ7E7MYtf96Xy5Sbr4vGEmA48f31vAnyrPyUE+Hrxt5uiufSCcKZ9uY2+7Ztzcfewar+nGgaXJwURicJKCIOBQmCpiHzrWLbMGDNdRKYB04BnXR2fUu7mRE4hOYXFtG/hX+26WflF/O6zOJbuOM61F7bh7+OjCXSc6Hu1CaZXm+Bztp2ZV0RkWEBFm6vSlX1aM7xbGIaKB0WohsmOlkIvYK0xJhdARP4L3AiMBUY51pkLrKCWScEYo/+TOlFD7mJs6FbuTeHhTzaRlV9M3w4hXB/dhuui29K6ud856+5NyuKBj2M5nJbLc9f2YsqIztX+O2gR4FOn2cc1aVmohsWO0UfbgYtFJFRE/IFrgA5AhDEmEcDxHF7Rl0VkqohsFJGNKSkp53zu5+dHWlqansicxBhDWloafn7nnoRU/fpozSEmz95A2+bN+N1VF1BcUspL3+5i6PRlTHh/DR+vPUxatjVz/pu4BMa+8ysn84r45L6LuG9kF/1hpGrFlgvNIjIFeATIBnYCecA9xpiQcuucMMZUecWrogvNRUVFxMfHVzt2X9Wcn58f7du3x9vbu/qVVZ0VlZTy4tc7+HjtES7vFc4bt/Uv6wLan5LNN1sTWbL1GPtTcvD0EKLaBrM1PpMBHUP41x0DK2xFKFVeVReabR99JCJ/BeKBJ4BRxphEEWkDrDDGXFDVdytKCko1ZJm5RTz8aSy/7kvjgUu68PureuJZwe0sjTHsPp7F11sT+Hl3MsO7hfHsmJ74eOnUI1U9dxx9FG6MSRaRjsBNwFCgMzAJmO54PrdinFKN2IGUbO6bu5GjJ3L5v/HR3BJT+U1qRKTswvHvx1Q950Wp82HXVaIvRCQUKAIeMcacEJHpwEJH19IR4BabYlPK5VbtTeXhT2Lx8vTg0/uHMCiypd0hqSbKlqRgjDmnxoIxJg0YbUM4Stkip6CY1fvT+Hl3Mgs3HqVbq0A+mBRDh5bVDz1Vqr7oeDKlXMQYw/6UbFbsSWHFnhTWH0ynsKSUAB9PxvVrx4tj+5RdUFbKLvp/oFL1LCEjj3dX7Gf5nmTiT1hVaHtEBDJ5eCSjerQiJrKlXiBWbkOTglL1aO2BNB75ZBM5hcWM7N6Kh0Z1ZdQF4VXesEYpO2lSUKoeGGOYs/oQL327i8hQfxY+OJSurQLtDkupamlSUMrJ8otK+NOi7XyxKZ7Le0Xw+oS+BPnpxD/VMGhSUMqJEjLyePDjWOLiM3ny8u48fll3PCqYfKaUu9KkoJSTnLp+UFBcysy7Y7iid4TdISl13jQpKFUHRSWlJGbk8+PO40z/fjcdQ/2ZcVcM3cL1+oFqmDQpKFUDR9Nz2RqfwZH0XI6m53LE8UjIyKfEcUez0T3Def22fgTr9QPVgGlSUKoaa/anMenD9RSWlAIQGuBDh5b+9O/QgrF9/enY0p/IsABiOrXQ6weqwdOkoFQVdiWeZOpHG+kYat2/ODIsQGcdq0ZN/+9WqhLHMvKYPHs9Ab5ezL13sE44U02CJgWlKpCRW8ikD9eTW1jCZw8O1YSgmgwtuKLUWfKLSrhv7kaOpOUy464YerYOrv5LSjUSmhRUo5dXWMLTC7Zwyf8t54OVBziZX1TpuiWlhifmbyb2yAlem9CXoV1DXRipUvbTpKAatYSMPG55fzWLthwjyM+Ll77dxbC//cz/frOTo+m5Z6xrjOHPS3bww44k/ufa3lwX3damqJWyj15TUI3WxkPpPPjxJvKLSvjg7hhG94pgW3wms1YdYO7qQ8z+9SBXR7VhysjODOjYgn+t2M+/1x7mgYu7cO+IznaHr5QtxBhjdwy1FhMTYzZu3Gh3GMoNzV9/hP9ZvJ12Ic34YFIM3cKDzvg8MTOPuasP8+m6w5zML6Zn6yB2H89iXL+2vHZrP51voBo1EYk1xsRU+JkmBdWYFJWU8vK3u5iz+hAju4fx9sQBNPevfIZxTkExn8fGM2f1ISJD/Xn/rhi94Y1q9KpKCtp9pBqNEzmFPPLpJlbvT+O+EZ2ZdnVPvDyrPsEH+HoxaVgkk4ZFuiZIpdycJgXVKPyWlMWUuRtIyizglVv6Mn5ge7tDUqpBsqWdLCJPicgOEdkuIvNExE9EOovIOhHZKyILRMTHjthUw7N6Xyo3v7ua/KJS5j8wRBOCUnXg8qQgIu2Ax4EYY0wU4AncBvwdeN0Y0x04AUxxdWyq4flyUzyTZq+ndbAfix4exoCOLewOSakGza4ral5AMxHxAvyBROAy4HPH53OBcTbFphoAYwxvLtvL0wu3EtOpJZ8/NIz2LfztDkupBs/l1xSMMcdE5BXgCJAH/AjEAhnGmGLHavFAu4q+LyJTgakAHTt2rP+AldspKinlj19u47PYeG7q347pN0friCGlnMSO7qMWwFigM9AWCACurmDVCsfKGmNmGGNijDExrVq1qr9AlVs6mV/EPbM38FlsPE+M7s6rt/bVhKCUE9kx+uhy4KAxJgVARL4EhgEhIuLlaC20BxJsiE25sYSMPO6ds4F9ydn83/hobonpYHdISjU6dvzEOgIMERF/ERFgNLATWA6Md6wzCVhsQ2zKTf26L5Ub//Urx07kMeeewZoQlKondlxTWCcinwObgGJgMzAD+BaYLyIvOZbNcnVsyv1kFxTz1+928em6I3QJC2DuvYO1lLVS9ciWyWvGmBeAF85afAAYbEM4yk2t2pvKs1/EkZCZx9SLu/D0FT3w8/a0OyylGjWd0axc5vtticSfyGNIl1B6tw3Gs5Kic1n5Rfz1u93MW3+ELq0C+PzBYQzspPMPlHIFTQrKJTYdOcGj8zZTUmoNKgvy9WJw55YM6RJ6RpJYuTeFaV9sI1FbB0rZQpOCqnc5BcU8tWALrYP9mHvvYHYkZLL2QDrrDqSxbHcyYCWJHq2DiD18gi6tAvhMWweqKctJA99A8PJ1+a41Kah695evd3IkPZf59w+hW3gg3cIDGdvPmpuYfDKftQfTWXsgjS1HMnjwkq48eXl3bR2opikvA1a+CuvegxaRcMtciOjt0hA0Kah6tXT7cRZsPMrDo7pyUZdz73ccHuzHDX3bckNfvfWlcqKSIti5GApzoNf14N/S7oiqVlIEGz+EFdMh7wRE3QSHVsHMy+Ca/4P+d4K45sZPmhRUvUk6mc+0L+OIahfMk5f3sDsc1RQUZMGmj2DNv+BkvLXsu2egx1UQfRt0vxK83KgAszGw5zv46XlI2wedL4YrX4I2fSE7Gb64D5Y8aiWIa1+1upTqmSYFVS9KSw3PfLaV/KIS3pjQX0tRNEXGwI/PQXE+RI6ATiMgsJ5K02QnW10uGz6A/ExrX9e9DkERsHUBbFsIu76GZi0h6mboexu0G+iyX98VStgMPzwHh1dBWA+YuMBKXqdiCgyHuxbBL6/Air9BwiaXdCfp7ThVvfhw1UH+8s1O/ndcFHcN6WR3OMoOB/4LH90AHt5QWmQta9XTShDOShJp+2H1W7DlUygptLqKhj8B7c+602RJMez/GeLmw+5vrUQV2g06DLFOvkGtrefA1qff+wRA/kk4cRBOHLIe6adeH4SsJPALBv9Q69GsxenX/qHg7QcF2VBw0tpOwUmrJZOfCfkZcHybtd6oP8DAyeBZ+W1jOfiL1WrIP+mU7iS9R7NyqT3Hs7j+7VWM6BbGrEkxiJ2/xpR95lxndYk8uhFSdsOhlVY3yOE1UJRjrdOqJ7QfZP1qbx8DrXqBZxUdGFlJcHQtHFlnPR/bBJ4+0G8iDH0MwrpVH1d+JuxcAts+s+LLToLS4nPX8/Kzkkd5zVpaF4BbdoagNtZJPjcNctOt57x067UpKfclAd9gK4GUPQdBm34w7FHwa159zHC6O+ngf62usDp0J2lSUC5TUFzC2Ld/JSWrgKVPXkyrINcPqVNu4PAamD0GrvobDH34zM9KiiBxqyNJ/ArHNloXVwG8mkHbflaSaDfAOgEnxsHRdXBkrfULHcDT11qnyyUw8B6rm6i2Skutk3l2EmQdP/2cmwYBrU4ngRaRNTuBl5ZCQSYU5Vknf59A53VTlZac7k66/M8w4slabUaTgnKZl77ZyQerDjJrUgyje9XhH6pq2D6+GRK2wJPbwKeamx8ZY53sj22CY7HWI3Hrmb/S/cOg4xDocJH13KavLWP43cbRDVbyrKrLqQpVJQW90KycZvnuZD5YdZA7h3TUhNCUHYuFff+xfslWlxDA+hXdsov1uNBRKLmkCJJ3Wn34rS+0PtNuyNM6DKq3TWtSUHVWWmp475f9vPrjb/SICORP17h2sk29y02HZX+BbqOtC5mqar+8Cn4hMOi+2m/D09tqDbTp67y4VI3oOEFVJ2nZBdwzZwP/WLqHMVGt+fyhYTTzaUSzkY9vgxmjIHY2LLgL1r1vd0SudeKwdVG3po5vhz3fwpCHrf501eBoS0HV2roDaTw+fzMncot4aVwUd1zUsXGNNIpbCEset4YaTvoa1r4H3//euhB52f/UvDujtBSyEqzRKh4NJGGeTIBf/s+aCFZaArfMhj43Vv+9la+ATxBcNLX+Y1T1QpOCOm8lpYZ/Ld/H6//5jU6hAXw4eRB92tZwWF1DUFJkTbpa9x50Gg63zLHGrnccBt8+bdWmyU6C6/5Z9fBJgPiN8P2z1ggbn6ByI2scj+bt6udvKC2F9ANW/37KLoiIgq6XVV/uIScVVr1uTQIrLYEBkyBpB3w51brY23lk5d9N+Q12fAUjnrISqWqQqk0KIvIo8Ikx5oQL4lFuLiWrgKcWbGHVvlTG9mvLyzdeSKBvI/ptkZUEn02GI6thyCNwxYunR3h4esH1/4TACPjlH9YJdPzsii+mnkyA/7xoTZYKjIDRz1vLjsXCmndOT+YKamMlh5BOUFh+olPWma+9/awhkS0ioUXnM4dJBraGnJTTI3eOxVqzX/MzHcEIYEA8oP1g6H6FVe6h9YWnWzt5GbDmbVj7LhTlQt+JcMnvre3npsOHY2D+7XDP99A6quJjt/JV8G4GQx9xzn8LZYtqh6Q6bo95G9btMz8EfjBuMo5Vh6S61pajGdz/0UZO5hXx4g19mDCoQ+PqLjq6HhbebZ0gx759eiRMRdbPhO9+Z028un3B6V/gRfnWyXXla9aJf+ijMPLpM/vXi/IhafuZJ/GTCdY6vo6JTWdMdAq2CrudmlmbGQ+m9PT2ys8YFk+rDEK7gdAuxnoO7WYN8dz3E+z90SqvAFYy6X65lZjWz7CSSJ8bYdQfodVZtaoyjsKsKwEDU36CkLPukZ1+AN6KgSEPwVUv1+boKxeq8zwFsf7lXwncA8QAC4FZxpj9zgz0fGlScJ3txzK5feZamvt7M/PuGNffJ7koD/Z8b9WIiehTt+GJxlgnwJPHrBNsZrw1s3X9TGjeHiZ8XPmv4fJ2fAVf3m/9cr/rS+vk/uNzkHEEel5nFTZr2bn2cVamuBAyj54ut3DisNUaaR8DraOrHwaanWwNGd37I+z72Zpo1WMMXPonaBNd+feSdsCHV1slIO5demZX1JLHYet8eDLO+ly5NadMXhORvlhJYQywHBgC/GSM+b2zAj1fmhRc47ekLCa8v4Zm3p4sfHAo7VvUYOy5Mx1eA4sfgXTHb5Cgtqe7QLpcUvkol5JiSN1jzYg9HgepvzmSwDEozDpzXQ8v68Q49u3z6w8/uNLqVikugJICCO8NY6ZbcTUEJcVW11Nwm5qtf2gV/PtGaNsf7l5sdRdlHIU3+1v1e659pV7DVc5Rp6QgIo8Dk4BU4APgK2NMkYh4AHuNMV2dHXBNaVKofwdSsrn1/bV4CCx8YCiRYQG139iWT61yBv1ur9mJtyAblr1o/YIP6WCdbPNOWL9w9y+3+tw9vKHTMCtBtOlrnfiPx1mJIHnn6VmxXs2sLpHmHazWQPP2ENzO8b6d9Uu7tiODjm+DpX+APuNgwOTqLz43dDu+sq67XHAN3PoR/PAH2DgbHt98breSckt1TQp/weoqOlzBZ72MMbvOM5gLgAXlFnUBngc+ciyPBA4Bt1Z3cVuTQv06mp7Lre+vobC4lAUPDKFbeB3Gne/9CT5x9NF7NYPoW2Hw1Mq7afb/DEuesLpJLnrAGgJavvhXSZFVD2fvj9a2k3ee/swvxOoGaR1tJYrW0Va/emM/WbvSuhnw/e8garxVkrrvBLjhLbujUjVU16QwBNhhjMlyvA8CehtjzmNGS6Xb9gSOARcBjwDpxpjpIjINaGGMebaq72tSqD+JmXnc+v4aTuYVM+/+IfRuW4drCJnx8N5ICG5rjd7ZNBfiPoPiPGuY50VTrT54T2/rIu+Pf4LNH0Nod6s7p+OQ6veRcdSqxBnWA0I6akkEV/jPi7DqNWtU02OxVikK1SDUNSlsBgacGnHk6DbaaIwZ4ITArgReMMYMF5E9wChjTKKItAFWGGMuqOr7mhTOjzGGWasO4uvlwdCuYXRtFVDh6KGUrAImvL+G5KwCPrnvIvp2CKn9TkuKYM611kXKqf89Xdo4Nx22fGJ1DWUctkbARN0M2z63+riHP4VIxgYAABldSURBVA6XTLOGYir3ZIzVvecdAJf8zu5o1Hmoa0E8KT8E1RhTKiLOaoffBsxzvI4wxiQ69pEoIuEVBiMyFZgK0LFjRyeF0TSsPZDOS9+e7u2LCPZlWNcwhnYNZVjXUNq38Cc9p5A7P1hHYmY+H00ZXLeEAFbNoKPr4OZZZ9a6928Jwx6zyiHs/ckaErnmbWuS1e3zrQuZyr2JWEXvVKNSk5bCl8AK4F3HooeBS40x4+q0YxEfIAHoY4xJEpEMY0xIuc9PGGOqvBqpLYXzM2XOBjYfzWDe/UOIPXyC1ftTWbM/jbScQgA6tvTHQyAhM585kwcxrFtY3Xa4ZynMm2DVu7/+jerXz023xuRr379S9aquLYUHgTeB5wADLMPxS72OrgY2GWOSHO+TRKRNue6jZCfsQznsS85i2e5knhjdnQtaB3FB6yBuv6gjxhh+S8pm9f5UVu9P41BqDu/fNbDuCSHjKCx6wJo1O2Z6zb5TXQkGpVS9qzYpGGOSsbp5nG0ip7uOAJZgDX2d7nheXA/7bLJmrTqIj5cHdw09837JIlKWJO4ZXs1Eq5w06xaIkSOtmbaVKS6Ez+9xFFKbq9cFlGpAalL7yA+YAvQByv51G2Pure1ORcQfuAJ4oNzi6cBCEZkCHAFuqe321ZlSswv4YtMxbh7QnrDAWt6t6uh6WDjJqvbpHWCVgIi5p+K+/2UvQvwGq5BcqG3TWJRStVCT+yn8G2gNXAX8F2gPZFX5jWoYY3KNMaHGmMxyy9KMMaONMd0dz+l12Yc67d9rDlNYXMqUEbUouWCMVSRt9tXg5WOd6KNutMpKzxhlPTZ9ZNXmAdj9rXXBeND9NSu1rJRyKzUakmqM6S8iccaYaBHxxiqKd5lrQqycXmiuXn5RCcOm/0z/DiHMmnyet/AryIIlj8GORdbs1XHvQjPHWIC8DCsxxM62Jo75Bluth+1fWLWApvzYtO+hq5Qbq+uFZkf5RTJEJAo4jjXrWDUAX2yKJz2nkPtGnufEouRd1p3G0vfD5S/C8CfOnBDWLMSadDb4fmvI6cbZsPkT8PKzWhOaEJRqkGqSFGaISAus0UdLgEDgf+o1KuUUpaWGWSsPcmG75gzpch4je+IWwtdPgE8g3L2k6huriFgzjjsOgTF/swrD1bS4mlLK7VSZFByzl086ahD9glWnSDUQP+9O5kBqDv+8rV/N7nuQkwrLX4aNH1rlJ26ZfX5lkHVIqVINXpVJwTF7+VGs+ycoN7DneBZH0nO5ondEtevOXHmAts39uObCKn65GwOHf7USwa6voaTQmmk8+oXTdxxTSjUZNek++klEnsGqYJpzaqGODnK91OwC7py1jpSsAm6/qCMvXN8bX6+Kyz3HxWew7mA6z13bC2/PCgaZ5abD1nkQO8cqN+3XHGKmWDXxw3vW69+hlHJfNUkKp+YjlL/xqkG7klyqtNTwzGdbiclfw20djvHoutHsTjzJe3cOJDz43MlhM1ceJMjXiwmDzqpvH7/RKkK3Y5F1U5j2g6xRRb3HVX/HLqVUo1eTGc31cD9Bdb5mrz7Epj0H2RA4A9+Uk2wIWcZTiZO57q083r1zIAM7nS4TdSwjj++2JXLv8EiC/BxdQCcTrFtFbv8CfIKg/53W5LPWF9r0Fyml3FFNZjTfXdFyY8xHzg9HVWT7sUymf7+LtyKW4ZOZBePexW/127yb/w9+Kr2Yh2bcwZM3DOP2i6yqsbNXHQRg8vDOVsmJtf+C//4DSoutctTDHjvzhjVKKeVQk+6j8jOe/IDRwCasO6WpepZTUMzj8zbT2/8kV2V/hfSdaN3OMmo8rHqNy395hYt84/jD4rvZFn8Lz1x1AfM3HOW66Da0S1sD//49pO2FHldbQ0br40bySqlGoybdR4+Vfy8izbFKXygXeGHJDg6m5bC+1w/IYeCyP1kfePnAqGlIr+sJWvwI7yS8yQ9bVjNxxwMEF+Txl4LZ8O/vrdnFty+EHlfZ+ncopRqG2hSuzwW6OzsQda7FW47xeWw8/3tRKa22LrJmFTdvf+ZKEX2QKf+Bte9w+bKXGVLyJH7NivE96gGXPmd1FWmVUqVUDdXkmsLXWKONwCqg1xudt1DvjqTl8qdF2xnYqQV3ZP3DKisx4qmKV/b0guFP4HnBtTT7+v8hzZrDmJesexUrpdR5qElL4ZVyr4uBw8aY+HqKRwFFJaU8Nn8zIvDe0Aw8vloOV/3tdDG6yoR1w+cevQ2FUqr2apIUjgCJxph8ABFpJiKRxphD9RpZE/baT7+x9WgG70zsS6vVt0FIJxg0xe6wlFJNQE3up/AZUFrufYljmaoHP+w4znv/3c9tgzpwrVkJSdtg9PNadVQp5RI1SQpexpjCU28cr33qL6Sma976Izz0cSzR7Zrz/JjO8PNL1p3N+txkd2hKqSaiJkkhRURuOPVGRMYCqfUXUtNjjOH1n37jD19u4+Ierfj0/iH4b54FJ+Phir+AR03+MymlVN3V5JrCg8AnIvK24308UOEsZ3X+iktKee6r7czfcJTxA9vzt5suxLsgA1a+Bt2vgs4X2x2iUqoJqcnktf3AEBEJxLp9Z53uz6xOyy0s5rFPN7NsdzKPXtqN/3dlD+u+B7+8AoVZcPmf7Q5RKdXEVNsvISJ/FZEQY0y2MSZLRFqIyEuuCK4xS88p5PaZ61i+J5n/HRfFM1ddYCWE9AOwfgb0uwMietsdplKqialJZ/XVxpiMU28cd2G7pi47FZEQEflcRHaLyC4RGSoiLUXkJxHZ63huUf2WGqaj6bnc/O5qdiWe5N07B3LXkE7WB2n74aOx1n2OL/2jvUEqpZqkmiQFTxEpGw8pIs2Auo6P/Cew1BjTE+gL7AKmAcuMMd2BZY73jc6RtFxu/Ndq0nMK+eS+i7iqj+N2l8e3wYdjoCAbJi2G4Lb2BqqUapJqcqH5Y2CZiMx2vL8HmFvbHYpIMHAxMBnKhrgWOkY1jXKsNhdYATxb2/24o1MzlQuLS/jioWF0jwiyPji8Gj69zSpnPflbaNXD3kCVUk1WTS40/0NE4oDLAQGWAp3qsM8uQAowW0T6ArHAE0CEMSbRsc9EEQmv6MsiMhWYCtCxY8Oq7XNqpvK/7hhwOiHsWQqfTYLmHeCuRRDSoeqNKKVUParpAPjjWLOab8a6n8KuOuzTCxgAvGuM6Y913+cadxUZY2YYY2KMMTGtWrWqQxiutXpfatlM5WsubGMt3LoA5t8O4b3g3qWaEJRStqu0pSAiPYDbgIlAGrAAa0jqpXXcZzwQb4xZ53j/OVZSSBKRNo5WQhsguY77cRvpOYU8tXALXcICeP56x4iite/B0mchciRMnAe+QfYGqZRSVN1S2I3VKrjeGDPCGPMWVt2jOjHGHAeOisgFjkWjgZ3AEmCSY9kkoFGU+zTG8PvP4ziRU8SbE/vj7yXw88tWQuh5HdzxuSYEpZTbqOqaws1YLYXlIrIUmI91TcEZHsOaJe0DHMC6eO0BLBSRKViVWW9x0r5s9fHaw/xnVxIvXdmWPgfnwMIPIOMI9L8LrnvDuheCUkq5iUrPSMaYRcAiEQkAxgFPAREi8i6wyBjzY213aozZAsRU8NHo2m7THe05nsXn337P3LAVXLx6BRTnW91FV74EvW4AcVaOVUop56jJ6KMc4BOsX/YtsX7BTwNqnRQavZIiCrd/RdGSV1nstQuT74/0nQiD74eIPnZHp5RSlTqvvgtjTDrwvuOhKlKYC++PxCdtH4GlEeyL+SPdrpgKzRrtBG2lVCOiHdrOtmsJpO3jd0VTaT5kEs9dH2V3REopVWOaFJysOPbfJNKaneHX8+XVvewORymlzovevcWZ0g/gdWQV84su5uWbovH18rQ7IqWUOi+aFJzIbP6EUoTf2lxHvw4hdoejlFLnTbuPnKW0hIKNH7O2JJqxFw+yOxqllKoVbSk4y4Hl+OUd5yffKxhzqhy2Uko1MNpScJKTa+ZQYgLpOGw8Xp6aa5VSDZOevZwhNx3/A0v5xoxgwpCudkejlFK1pknBCXJi5+NlisjoOYEQfx+7w1FKqVrT7iMnyF03h4OlkYwZfYXdoSilVJ1oS6GOiuK30Cp7D7Etrz19NzWllGqgNCnU0dGfZ1BgvOhy6WS7Q1FKqTrTpFAXxQWEHVzCr95DGH5hd7ujUUqpOtOkUAcHfv2MYJNFSd878PDQeyMopRo+TQp1kLduLomEMuzym+0ORSmlnEKTQi0lxe+jV84G9rW5noBmvnaHo5RSTqFJoZb2/DATDzF0veIBu0NRSimn0aRQC3kFRXQ+uog9fn1p26W33eEopZTTaFKohV+XLaEDSXgNvNvuUJRSyqlsmdEsIoeALKAEKDbGxIhIS2ABEAkcAm41xpywI76qGGMo3fxvcmlGl0sm2h2OUko5lZ0thUuNMf2MMTGO99OAZcaY7sAyx3u3s25LHJcUriSx0w2IT4Dd4SillFO5U/fRWGCu4/VcYJyNsVQq/z9/RQTa3/Anu0NRSimnsyspGOBHEYkVkamOZRHGmEQAx3N4RV8UkakislFENqakpLgoXMuhPVsZkf0jO9uOxze0k0v3rZRSrmBXldThxpgEEQkHfhKR3TX9ojFmBjADICYmxtRXgBXJ/O5FwvGm49jnXLlbpZRyGVtaCsaYBMdzMrAIGAwkiUgbAMdzsh2xVSbz4Gb6Zi5jffh4WkZ0sDscpZSqFy5PCiISICJBp14DVwLbgSXAJMdqk4DFro6tKmlfP89J04wO17nl9W+llHIKO7qPIoBFInJq/58aY5aKyAZgoYhMAY4At9gQW4UKD6+jS/ovfB4ymfGdOtodjlJK1RuXJwVjzAGgbwXL04DRro6nJk58/QJeJojWVz5pdyhKKVWv3GlIqlsyB38hInUNn/ndwvDekXaHo5RS9Urv0VwVY8j67s/kmha0HPUQji4vpZRqtLSlUJW9PxGcEsssj/HcENPV7miUUqreaUuhMqWlFPz0IkmmFf5DJuPn7Wl3REopVe+0pVCZXUvwTdnOWyXjuWNYN7ujUUopl9CWQkVKSyj5+WUOmnaUXngL4cF+dkeklFIuoS2FisQtxDPtN14tGs+9I7WVoJRqOrSlUIHSuAUclXZkdLqKPm2b2x2OUkq5jLYUzmYMxUc3sbqoB1O0laCUamI0KZwt/QA+RZkcadaTy3pWWL1bKaUaLU0KZyk9tgkA346D8PDQyWpKqaZFk8JZTu5fR77xpl2P/naHopRSLqcXms9SdCSW/SaS/p2160gp1fRoS6G8kmKaZ+xkj0c3uoQF2B2NUkq5nCaF8lL34GPyyQ7rq9cTlFJNkiaFcnIPrgcgoPMgmyNRSil76DWFck7sW0exaUaXC865B5BSSjUJ2lIoxzNxM9tNF/p2bGF3KEopZQtNCqcUFxCWs48E/174+2gDSinVNGlScChOiMOLYkra6PwEpVTTpUnBIXn3GgBa9hhicyRKKWUf25KCiHiKyGYR+cbxvrOIrBORvSKyQER8XBlP7qENpJhgevfs7crdKqWUW7GzpfAEsKvc+78DrxtjugMngCmuDCYgNY49Ht1pG9LMlbtVSim3YktSEJH2wLXAB473AlwGfO5YZS4wzmUBFWQRUXiYzBYXYoWilFJNk10thTeA3wOljvehQIYxptjxPh5oV9EXRWSqiGwUkY0pKSlOCSZt3wY8MHh1HOiU7SmlVEPl8qQgItcBycaY2PKLK1jVVPR9Y8wMY0yMMSamVatWTokpefdqANr2Hu6U7SmlVENlx4D84cANInIN4AcEY7UcQkTEy9FaaA8kuCqg0vhNxJswLujS2VW7VEopt+TyloIx5g/GmPbGmEjgNuBnY8wdwHJgvGO1ScBiV8UUmrmDo3498fHSEbpKqabNnc6CzwJPi8g+rGsMs1yx0/yMJFqXHic/vJ8rdqeUUm7N1noOxpgVwArH6wPAYFfHcGT7r/QAgrq6fNdKKeV23KmlYIvMfesoNUKXC/Uis1JKNfmk4JO8haMe7WgZGmZ3KEopZbsmnRRMaSntcneTEtzH7lCUUsotNOmkEH94H2FkQFutjKqUUtDUk8KOXwFo1XOozZEopZR7aNJJoeDIRoqMJx16XWR3KEop5RaadFJonr6NeJ8uePhoZVSllIImnBRO5hXQtWgv2aEX2h2KUkq5jSabFPbs2Eqw5OLXaZDdoSillNtoskkhdY91+822fYbZHIlSSrmPJpsUJHEz+fgS0C7K7lCUUsptNMmkUFJqiMjaSVJAD/C0tfyTUkq5lSaZFH5LPEFPDlIUoZVRlVKqvCaZFPbv2EgzKSSk2xC7Q1FKKbfSJJNCz5K9AIT20KSglFLlNckO9W6dIyHjWiS0q92hKKWUW2mSSYGe11oPpZRSZ2iS3UdKKaUqpklBKaVUGU0KSimlymhSUEopVcblSUFE/ERkvYhsFZEdIvKiY3lnEVknIntFZIGI+Lg6NqWUaursaCkUAJcZY/oC/YAxIjIE+DvwujGmO3ACmGJDbEop1aS5PCkYS7bjrbfjYYDLgM8dy+cC41wdm1JKNXW2XFMQEU8R2QIkAz8B+4EMY0yxY5V4oF0l350qIhtFZGNKSoprAlZKqSbClslrxpgSoJ+IhACLgF4VrVbJd2cAMwBEJEVEDleymzAg1Qnh1hd3jk9jqx2NrXY0ttqpS2ydKvvA1hnNxpgMEVkBDAFCRMTL0VpoDyTU4PutKvtMRDYaY2KcFqyTuXN8GlvtaGy1o7HVTn3FZsfoo1aOFgIi0gy4HNgFLAfGO1abBCx2dWxKKdXU2dFSaAPMFRFPrKS00BjzjYjsBOaLyEvAZmCWDbEppVST5vKkYIyJA/pXsPwAMNiJu5rhxG3VB3eOT2OrHY2tdjS22qmX2MSYCq/nKqWUaoK0zIVSSqkymhSUUkqVaZRJQUTGiMgeEdknItPsjqc8ETkkIttEZIuIbLQ5lg9FJFlEtpdb1lJEfnLUoPpJRFq4UWx/FpFjjmO3RUSusSm2DiKyXER2Oep3PeFYbvuxqyI224+dO9c9qyK2OSJysNxx6+fq2MrF6Ckim0XkG8f7+jluxphG9QA8sWZIdwF8gK1Ab7vjKhffISDM7jgcsVwMDAC2l1v2D2Ca4/U04O9uFNufgWfc4Li1AQY4XgcBvwG93eHYVRGb7ccOECDQ8dobWIc1R2khcJtj+XvAQ24U2xxgvN3/zzniehr4FPjG8b5ejltjbCkMBvYZYw4YYwqB+cBYm2NyS8aYX4D0sxaPxao9BTbWoKokNrdgjEk0xmxyvM7CmmfTDjc4dlXEZjtjccu6Z1XE5hZEpD1wLfCB471QT8etMSaFdsDRcu8rraNkEwP8KCKxIjLV7mAqEGGMSQTrBAOE2xzP2R4VkThH95ItXVvliUgk1hDrdbjZsTsrNnCDY1eXumeujs0Yc+q4vew4bq+LiK8dsQFvAL8HSh3vQ6mn49YYk4JUsMxtMj4w3BgzALgaeERELrY7oAbkXaArVsn1ROBVO4MRkUDgC+BJY8xJO2M5WwWxucWxM8aUGGP6YZWyGcx51D2rb2fHJiJRwB+AnsAgoCXwrKvjEpHrgGRjTGz5xRWs6pTj1hiTQjzQodz7GtVRchVjTILjORmrGKAzJ+w5Q5KItAFwPCfbHE8ZY0yS4x9uKTATG4+diHhjnXQ/McZ86VjsFseuotjc6dg54skAVlCu7pnjI9v/vZaLbYyjO84YYwqA2dhz3IYDN4jIIazu8MuwWg71ctwaY1LYAHR3XJn3AW4DltgcEwAiEiAiQadeA1cC26v+lsstwao9BW5Wg+rUCdfhRmw6do7+3FnALmPMa+U+sv3YVRabOxw7d657Vklsu8slecHqs3f5cTPG/MEY094YE4l1PvvZGHMH9XXc7L6iXh8P4BqsURf7gT/ZHU+5uLpgjYbaCuywOzZgHlZXQhFWC2sKVl/lMmCv47mlG8X2b2AbEId1Am5jU2wjsJrqccAWx+Madzh2VcRm+7EDorHqmsVhnVyfdyzvAqwH9gGfAb5uFNvPjuO2HfgYxwglux7AKE6PPqqX46ZlLpRSSpVpjN1HSimlakmTglJKqTKaFJRSSpXRpKCUUqqMJgWllFJlNCkoVQURKSlXIXOLOLHqrohElq8Cq5Q7sOMezUo1JHnGKn2gVJOgLQWlakGs+2L83VGDf72IdHMs7yQiyxwF1JaJSEfH8ggRWeSo179VRIY5NuUpIjMdNfx/dMymVco2mhSUqlqzs7qPJpT77KQxZjDwNlYtGhyvPzLGRAOfAG86lr8J/NcY0xfrPhE7HMu7A+8YY/oAGcDN9fz3KFUlndGsVBVEJNsYE1jB8kPAZcaYA44CdMeNMaEikopVQqLIsTzRGBMmIilAe2MVVju1jUisEs3dHe+fBbyNMS/V/1+mVMW0paBU7ZlKXle2TkUKyr0uQa/zKZtpUlCq9iaUe17jeL0aq5IlwB3AKsfrZcBDUHYzl2BXBanU+dBfJUpVrZnjblynLDXGnBqW6isi67B+XE10LHsc+FBEfgekAPc4lj8BzBCRKVgtgoewqsAq5Vb0moJSteC4phBjjEm1OxalnEm7j5RSSpXRloJSSqky2lJQSilVRpOCUkqpMpoUlFJKldGkoJRSqowmBaWUUmX+P27nAQuwl9IYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation ACC: 75.25%\n",
      "Test ACC: 73.93%\n"
     ]
    }
   ],
   "source": [
    "with torch.set_grad_enabled(False):\n",
    "    test_acc = compute_acc(model=model,\n",
    "                           data_loader=test_loader,\n",
    "                           device=DEVICE)\n",
    "    \n",
    "    valid_acc = compute_acc(model=model,\n",
    "                            data_loader=valid_loader,\n",
    "                            device=DEVICE)\n",
    "    \n",
    "\n",
    "print(f'Validation ACC: {valid_acc:.2f}%')\n",
    "print(f'Test ACC: {test_acc:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "default_view": {},
   "name": "convnet-vgg16.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "371px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}