14_recurrent_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 14 Recurrent Neural Networks\n",
    "\n",
    "Recurrent neural networks can take in sequences of arbitrary length as input, such as time series data or text. They're also capable of generating novel output. They do this by introducing the idea of networks operating in time, or having memory. \n",
    "\n",
    "## Recurrent Neurons\n",
    "\n",
    "Recurrent neurons can take their output and send it back to themselves as input. We introduce the idea of a time step t and say that a neuron receives both input X<sub>(t)</sub> and output y<sub>(t-1)</sub> from the previous time step. A neuron thus has two sets of weights: one for current input and one for previous output. A *memory cell* or just a *cell*, such as a recurrent neuron, is part of a neural network that is dependent in some way on information from a previous time step. A cell also has an associated state at each time step, dependent on current input and its state from the previous time step. \n",
    "\n",
    "## Input and Output Sequences\n",
    "\n",
    "There are multiple types of RNNs, when considering possible inputs and outputs. We could have a sequence as input and a sequence as output, such as using historical time series data to make forecasts. We could also have a sequence as input and a vector as output, such as classifying a document by its feeding a network its sequence of words. We could also have the reverse of this, with a vector as input and a sequence as output. Finally, we could join together a sequence to vector network with a vector to sequence network. An example of this is an Encoder-Decoder model, where we can use the encoder to convert a sequence of words into a vector which the decoder could translate into a sequence in another language.  \n",
    "\n",
    "## Basic RNN's in TensorFlow\n",
    "\n",
    "Our basic RNN will be made up of a cell of 5 neurons that take in vectors of 3 elements. It will also have 2 time steps, so the output of the first layer (first time step) will also be included in the input to the second layer. Every layer will have the same weights and biases because they contain the same neurons, but at a different time. We will get the output of each instance as it's gone through every neuron at every time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_steps = 2\n",
    "n_inputs= 3\n",
    "n_neurons = 5\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Separate input vector for every time step in one tensor\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_steps, n_inputs])\n",
    "\n",
    "# Cell made up of 5 neurons\n",
    "basic_cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "# States is output at last time step for basic cell\n",
    "outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 0.64979637  0.8083514  -0.73296475 -0.6190548  -0.5570247 ]\n",
      "  [ 0.9607432   0.9887017  -0.9994823  -0.7526373   0.87071806]]\n",
      "\n",
      " [[ 0.926664    0.9872352  -0.9948982  -0.86801    -0.1014521 ]\n",
      "  [-0.10818524 -0.49670416  0.874287   -0.1561937  -0.64163405]]\n",
      "\n",
      " [[ 0.9864417   0.9992215  -0.9999151  -0.958444    0.4010542 ]\n",
      "  [ 0.5496384   0.8406941  -0.95120746 -0.7192609   0.84106195]]\n",
      "\n",
      " [[-0.95322895 -0.89880407 -0.9997649   0.9951918   0.9999122 ]\n",
      "  [-0.7791043   0.10235804 -0.8389093  -0.19287914  0.95108473]]]\n",
      "\n",
      "Output of 4th instance at 5th neuron in 2nd time step: 0.95108473\n"
     ]
    }
   ],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "\n",
    "X_batch = np.array([\n",
    "    # t=0       # t=1\n",
    "    [[0, 1, 2], [9, 8, 7]], # instance 1\n",
    "    [[3, 4, 5], [0, 0, 0]], # ...\n",
    "    [[6, 7, 8], [6, 5, 4]], # ...\n",
    "    [[9, 0, 1], [3, 2, 1]]  # instance 4\n",
    "])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    outputs_val = outputs.eval(feed_dict={X: X_batch})\n",
    "\n",
    "# Output of 5 neurons with 4 instances and 2 time steps\n",
    "print(outputs_val)\n",
    "# Outputs[instance][time step][neuron]\n",
    "print('\\nOutput of 4th instance at 5th neuron in 2nd time step:', \n",
    "      outputs_val[3][1][4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling Variable Input Length Sequences\n",
    "\n",
    "If we have inputs with different lengths (meaning some of them don't span the entire time step), they must be zero-padded to the length of the longest sequence when they're defined, and the sequence_length parameter must be passed when constructing the RNN specifying the length of every instance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=(None, n_steps, n_inputs))\n",
    "basic_cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "\n",
    "seq_length = tf.placeholder(tf.int32, shape=(None))\n",
    "\n",
    "outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32,\n",
    "                                   sequence_length=seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " OUTPUTS\n",
      " [[[ 0.9005094   0.19721422 -0.74749184  0.13447112  0.35255757]\n",
      "  [ 0.9959246  -0.9221021  -0.99922425 -0.00510527 -0.7015953 ]]\n",
      "\n",
      " [[ 0.99999094 -0.9999675  -0.9999993  -0.83891016 -0.99947757]\n",
      "  [ 0.          0.          0.          0.          0.        ]]\n",
      "\n",
      " [[ 0.99997807 -0.9918821  -0.9999742  -0.4349029  -0.93770105]\n",
      "  [ 0.9162537  -0.9960944  -0.6650218   0.7405216  -0.9781005 ]]\n",
      "\n",
      " [[ 0.9999938  -0.9611891  -0.9999759  -0.20734294 -0.7319394 ]\n",
      "  [ 0.99696225 -0.9999669  -0.998784    0.18959737 -0.9995896 ]]]\n",
      "\n",
      " FINAL STATES\n",
      " [[ 0.9959246  -0.9221021  -0.99922425 -0.00510527 -0.7015953 ]\n",
      " [ 0.99999094 -0.9999675  -0.9999993  -0.83891016 -0.99947757]\n",
      " [ 0.9162537  -0.9960944  -0.6650218   0.7405216  -0.9781005 ]\n",
      " [ 0.99696225 -0.9999669  -0.998784    0.18959737 -0.9995896 ]]\n"
     ]
    }
   ],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "\n",
    "X_batch = np.array([\n",
    "    [[0,1,2], [3,4,5]],\n",
    "    [[9,8,7], [0,0,0]], # zero padding\n",
    "    [[5,6,7], [3,2,1]],\n",
    "    [[4,6,8], [7,5,3]]\n",
    "])\n",
    "\n",
    "seq_len_batch = np.array([2,1,2,2])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    outputs_val, states_val = sess.run(\n",
    "        [outputs, states], feed_dict={X: X_batch, seq_length: seq_len_batch})\n",
    "    \n",
    "print(f' OUTPUTS\\n {outputs_val}')\n",
    "print(f'\\n FINAL STATES\\n {states_val}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training to Predict Time Series\n",
    "\n",
    "Let's use RNNs to predict future values in time series data. Since the output of our RNN model at each time step will be a vector the size of the number of neurons, we'll use a fully connected layer to produce an output of the correct size after doing some reshaping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztvXeUZGd55/99K+fQ3dXVOUwOmlEaCckSIgkhkQQYZGCNWWwvZg1O7G+9DqzBnGXNsY292GZhscGwJhkvSQSDQAiEhNJISJNDz0z3dKzUXTlXvb8/7n1vV3Xdqu6Z6apb4fmcM2e6q2v63ts1/a3nPuH7MM45CIIgiO5Hp/UJEARBEK2BBJ8gCKJHIMEnCILoEUjwCYIgegQSfIIgiB6BBJ8gCKJHIMEnCILoEUjwCYIgegQSfIIgiB7BoPUJVDIwMMCnpqa0Pg2CIIiO4tlnnw1zzn2bPa+tBH9qagpHjx7V+jQIgiA6CsbY3FaeRykdgiCIHoEEnyAIokcgwScIgugRSPAJgiB6BBJ8giCIHoEEnyAIokcgwScIgugRSPAJguhZLoVT+N7xZa1Po2WQ4BME0bP81Q/O4Le/+BwePRfS+lRaAgk+QRA9SaFUxs/OhQEAH/jmCXDONT6j5kOCTxBET/Lc3BoSuSJeNN2Hy6tphJN5rU+p6ZDgEwTRkzx6PgSDjuFdd0wBAM4HE9qeUAvYFsFnjH2WMRZkjJ2oeKyPMfZDxth5+W/vdhyLIAhiO5gJJjE1YMeNE17l825nuyL8zwG4d8NjfwTgYc75bgAPy58TBEG0BfOrGYx7rRh0muG0GHA+QIK/JTjnjwJY3fDw/QA+L3/8eQBv2I5jEQRBbAfza2mM99nAGMPuQQeldK4RP+d8GQDkvwebeCyCIIgtE0sXkMgWMe61AQB2Dzopwm8FjLF3M8aOMsaOhkK90QtLEIS2zK+lAQDjfZLgT/TbEEnlkcmXtDytptNMwQ8wxoYBQP47qPYkzvmnOedHOOdHfL5NN3QRBEFcM/OrQvCtAIAhlwUAsBLPanZOraCZgv8ggHfKH78TwLeaeCyCIIgtszHCH3JLgr8cy2h2Tq1gu9oyvwzgCQB7GWMLjLHfAPBRAK9kjJ0H8Er5c4IgCM1ZWMvAZTHAZTECWBf8QJdH+NuyxJxz/rY6X3rFdnx/giCI7SQQzyoiD1SkdGI5rU6pJWhetCUIgmg14WQePqdZ+dxuNsBpMWCFUjoEQRDdRSiRg89hrnpsyGWhoi1BEES3EUrkMLBR8N0WrMQppUMQBNE1pHJFZAqlqpQOIEX4gRhF+ARBEF1DKCFF8TWC77YgmMiiVO5eX3wSfILoYZ64EMG9/+tR/PYXn9X6VFpGKCkJ/saUzqDTjDIHIqnuTetsS1smQRCdyV/+4AzOrCRwZiWBdL4Im6n7JaFehO9zSq2ZwXgOg05Lzb/rBijCJ4gepVgq4/RyHDt8dgDA8/NRjc+oNYST6oI/6JI+F28I3QgJPkH0KBdCKWQLZbzz9ikwBhydXdP6lFpCKJGDXsfgtZmqHhdtmsFE9xZuSfAJokc5vhgDANyxqx97/U48M7txpUV3Ekrk0Gc3Qa9jVY+LiD/Yxa2ZJPgE0aOcWIzBZtJjesCBQ6NunFnp/gUggJTS2Th0BQAWox5uqxFBSukQBNFtnAsksMfvhF7HMN5nQyiRQ7bQ3X7wgDxl66wVfEDq1KEcPkEQXcdSNIMxr+QHL/5eina3lwygPmUrGHSZKYdPEER3US5zLMWyGPUIwZd84efXulvwOec1xmmVDDotlNIhCKK7iKTyyBfLGPFUR/gL8mKQbiWeKSJfKjdM6QQTOXDendO2JPgE0YOIzU7Dsie832WBQcew0OURfigppWsGHCbVr/ucZuSLZcQzxVaeVssgwSeIHkTk6kWEr9cxjHisXS/4wTpTtgKlNbNL8/gk+ATRgyxGJUETOXxAWujd7SmdcDIPQErdqCEsFbq1U4cEnyB6kOVoBhajDh6bUXlsxG3FYpdH+ELIG3XpAOjawi0JPkH0IEuxDEY8VjC2Pm065LYgnMx1tT1wKJGDUc/gthpVvz5IKR2CILqN5VhWKdgKBl0WyR442Z3RLbDeg1/5RleJw2yA1ajvWnsFEnyC6EFCiVoLYL8c3Qa6VOwAKXL3u+pbHzPG4JNbM7sREnyi54kkc4hlClqfRsvgnCOYyNUULoUQBrp4kXcgnoXfpZ6/F3SzvQIJPtHTFEtlvPlTT+A9/9I7G5/imSLyxdrhI0XwuzR/DUh3L40ifKC77RVI8Ime5nsnVnApnMITFyM4H+gNt0ghZhsFf8BhAmPdm9LJFkqIZQqbC34X2yuQ4BM9zed/PouJPhuMeoYvPz2v9em0BCFmG3P4Br0OAw4zgl2a0hGpqno9+IJBlxmJbBHpfPdN25LgEz1LucxxcimGu/f78aLp/p5ZACIi/EGVXLbfZe7aHL64cxlyN47wx4WR3Gr3zSSQ4BM9y1Isg2yhjF2DDuwadOBiKNm1plmViJZDtUjX77R0bUpHvJFtltKZ6JMEfy6Savo5tRoSfKJnmQkmAQA7fXbs9NmRype6VuwqCSZysBr1cJgNNV8bdFm6tmCpCL6zseBP9kuCf3m1+2wmal9xgugRLoSkCG7XoEOZLr0QSm56y9/phBI5DLrUh4/8LjPCyTwKpTKM+u6KB4OJHMwGHVzWxrLnthrhtBgw34WC312vKEFcATPBJDw2I/rsJuzwOQBIgt/tBBNZ1Z2uwHq6oxv70JdjWQy5LXWnbAWMMUz02TBHgk8Q3cOFUBK7fA4wxuB3mWE36XEx1H15240E5QhfDTGU1I2F27lISsnPb8Zkv60rUzok+ETPMr+axoScr2WMYYfP0RMRfihea6sgWJ+27a4In3OOS6EUpgfsW3r+eJ8NC6uZKiO57x5bxrv++emOnsomwSd6klJZsheoNBAb81q7fol3Jl9CIlesuwBECH63FW7DyTwSueKWBX+v34l8qYxz8jDeD08F8N4vPYdHzobwb0c7d16DBJ/oSSKyDfBQRYvesNuK5Vi2q1szlR78OoLfZzPBoGNdl9KZlVssp7Yo+LdO9wEAnrwYAQB84pEZTPXbcOOEB194cg7lDrWQbrrgM8ZmGWPHGWPPM8aONvt4BLEVRMpisELwRzwWpPOljr5l3wxlyrZOL7pOxzDoNHd0SuerR+fxZ986UWWVcUmuzezYouCPeW0Y81rx5MUInp1bxfPzUbzrjmm87dYJzEbSOLPSmTYcrWrLfBnnPNyiYxHEpqzIEexQleBL6/6Woll4bOpLrjudRkNXgkGXpWMj/GKpjP/5vdOIpgv44akAfvpfXwaTQYdLkRQMOla10nEzbtvRjx+dDiAQz2HAYcKbbx5T7hQuhVM4MOJq1mU0DUrpED2JIvjuypSO9PFyrHvz+KFNUjqA1KnTqQtAnpldQzRdwFtvGcdyLIsHX1gCAJxZjmOy3wbDFcwWvOGGUSSzRTw/H8Uf3rsPdrNBqQFcCndmcb8Vgs8BPMQYe5Yx9u6NX2SMvZsxdpQxdjQUCrXgdAgCCMSy0OtY1W5TJcKPdWZ0uxWCiRwMOgZvgzuYIZcFS9FMR9YyfnByBWaDDv/9tQewb8iJT/30ApK5Ih6/EMFde3xX9L3u3D2A7/zunfjw/Qfx5pvGAAA2kwHDbgsuhjuzfbcVgn8H5/wmAPcBeC9j7K7KL3LOP805P8I5P+LzXdkLQhBXy0pcGj7S69aHcAYcZhh0rKs7dYLyij+drv7w0XifDYlcsSNrGU9dWsWt032wmw34zy/diZlgEn/2rRPIF8u458DQFX+/fUMu/NrtU1U/r+kBOy6R4KvDOV+S/w4C+AaAW5t9TILYDLXNR3odw5DbguUuF/x6Q1eCsQZukSuxLI62qaso5xzzq2nslKemX3NoGGNeK77+3CI8NiNumfJuy3FI8OvAGLMzxpziYwD3ADjRzGMSV86Hv30Kn/zJhaohk25HEvzaTpURt7WrUzrL0UxVoVqN8T4ptTW/Vj1p+tj5MG77i4fx5k89gZU2/BmtpQtI5ooYl6dpDXodPveuW/H7d+/G3zxw/RXl7xsxPWBHNF3AWiq/Ld+vlTS7S8cP4Buyd4UBwJc4599v8jGJK2ApmsFnH78EAMgUSnj/K/dofEatYSWWxW07+mseH/ZY8NzlNQ3OqPlwzrGwltk0ly0Ec6N52Gceu6h8/OMzQbz9RRPbf5LXgLBCqLRP2DXowO/fvb3/pyf7pcLt/FoaXntndXM1NcLnnF/knF8v/znIOf9IM49HXDmPnZe6ZR1mAx6f6Y3O2Uy+hHi2qBrhD7utWIllO3awBgD+9ZnL+P6J5ZrHV1N5ZAoljHkbtya6LEZ4bMYqL5l4toDHZyL4zTunMea14sdnAtt+3teKmuA3gwGHJPLhZOd1MlFbZo/z6PkQfE4z3nJkDCeXYiiWylqfUtNR68EXjHosKJR4R/4yA8BaKo//9rXjeM8XnsM/Pnqx6mvza1JOXuToGzHutSnPB4BHzgSRL5Vx36Eh3L3fj8dmwsgX2+v/irgjESmpZiE6u8KJzkvpkOD3MJxzPHEhghfvGsAN4x5kC2WcD3Zmf/GVIPLPar73w+7Obs380Wkp8raZ9PjCU3NVrZULck5+swgfkESzMqXzyJkgBhwm3DjuxXWjbmQL5bbrZrocSWPAYYbN1NxMtfAhCnVgUECC38NE0wVEUnkcGHHh0KgbAHBsIarxWTWf9VV3td0qwx55+KrNxGyr/ODkCkY9VvzJq/djLpLGucD6G/iCEuFvLvi7Bp2Yi6SQzBVRLnM8NhPBHbsGoNMxjHvVi7pac3k1jYkmR/cAYJG3hXXiXSAJfg8zV5HznOq3w2kx4PhiTOOzaj6NdpuK0fvFDhR8zjmeuriKl+714ZUH/ACAh06uKF+fX03DYzPCaTFu+r1unvSizIHnL0dxZiWBcDKHO3cNAKgs6rbXzyiQyCp3aM2m32FCOEkpHaKDEEuaJ/vt0OkYpgfsuNxmv8TNYCWehd2kVxU+t9UIq1GP5TopnROLMXzsobP46bn2mwpfSxeQyBWx0+eA32XB4TF31XkurGW2FN0DwI0THjAGPDu3hkfPS9/jxbul7h6/ywKjnrVdhB9J5pWCarMZcJgR7sCtYCT4PczlSHVXw5jXioUu3PKzkUA8C3+dvbWMMQx7LHX9dP7i30/j7388g/f8y7MotFmBW7yBi9fzzl0D+MV8FIlsAZxznFmJK0NJm+GyGLHX78Qzs6v48tOXcdOER6l56HUMIx5rW+18zRfLiGUK6K+zunG7GXCYKKVDdBazkTT8LjOsJj0AqTNjYS3T0S2JgGR+1qiDZCWWbTh8NOK2YilaG+HHswU8dXEVY14rMoUSTi/Ht+V8twvRljjZvy74pTLH05dWsRTLIhDP4aaJrU+b3rajH4/NhDEXSeM37txR9bWNXTxas5aW0iv9LYzwIx04eEWC38NcXk1hsm/dH3zMa0W+VO7I7gPBaiqPl/31T/COzzyFTL6k+pxAPKeavxeMeCyqHSiPnguhWOb4w3v3AZDSHe3EeluiJPg3TXphNujws/Nh5Vxvnty64L//nj345ZvGcOtUH1510F/1tfG+9robFNF2v71VEb4Za+l8x7Uxk+D3MHOR9Z2uADBWZ8Kyk3jo5AqyhTKeurSKT2/oQwek1YYr8WzVasONDLutCCVzNXcJPzkbgsdmxGsODWPEbWk7wZ+T79gsRumOzWLU42V7B/Gt5xfxxIUIrEY99g05t/z9XBYjPvbA9fjqe26vsSUY89oQSeWRzhe39RqulohcQG1ZDt9pBudSgNFJkOD3KLliCcFErqqINy5Ms9qsGHclfO/ECib6bDg85ladHA4lpNWGIw0WYYx4LOAcNUtAXpiP4qYJL/Q6hhsnvfjF5fZqYZXaEquHqt75S1NYSxfw1aPzuHHCs21+MsJPv12GjyIpKcLva5HVwYB8nE67GybB71HEgouRijY2If7t1m63VbKFEn4+E8a91w3hth39eH4+imyhOq0j2i1HPI1SOtLPobJTJ5UrYiaUxOExaV5h96ADi9EMckX1tJEWzK+mlXSO4LYdfbh50oudPjv+/PUHt+1YYtpUCK3WiAi/VUVbsREtlu4sC2kS/B5lWWXa1GLUw+c0KxOZ9fjwt0/hXf/8dNvczgvmImkUyxwHR1y4bUcf8qVyjRGa6L5p1K+tTNtW5PFPLsXBORTBF28KgVh7CF65zBFI5GpSVYwx/Ou7b8MPfv8u7PZvPZ2zGaI4GmmTXvRwMg+jnsFlac3WVo9NaunttJ0BJPg9yrrwVQvEiNtStwcdAJK5Ir741BweORvCH3/9eFPP8UoRbYlT/XYcmeqDjgFPX6r2bl+Wu29GGgi+iP6XKlozxQTyoVEPgPUBraU2WYcYzRRQKvOqDV4Cg14H2bF22xCpk/aJ8HPot5u3/Trr4bZKgh8lwSe0pFTm+MKTcwgmGnvBCD+Z4Q257GG3taFHysOnA8gVy9jrd+KRM8G2WoM3F1lvS3RZjJgesOPEYnXr5GI0A7tJD5e1fiRoMxngthqVNwcAOLYQw4jbovioiDfKdvGTEV0qaoLfDEQ3TLtMm0ZS+Za1ZAIU4RNtwk/OBvGBb57Ab37+aE3+upLlWBZOswEOc7XwSUNH2bpC/v0TK/C7zPjV2yYQzxbbyoJgbjUFt9Wo5FcPjrhreuWXYxkMe6ybRoIjHmvV8NXxxRgOyekc8XWgjQQ/0VrBt5r0sJv0bZPSiSRzLSvYAoDVqIdRzxClHH7nEc8W8KEHT+JcIKH1qVwz3zm2DJNBh2MLMXzr+cW6z1uJZeu4RVqQlv3i1Ti2EMOLpvtxUDZbO7XUPsNHc5E0piraTA+OuLAYzVRtJlqONW7JFIx7rcqi6limgEvhFA6PeZSvW4x69NtNWFQZ0NKCsHyNPmfrRK/fYcZqm6R01tKFlgo+Ywxuq4ki/E6jVOb4tc88jc/9fBYf/9F5rU/nmsgWSvjhqQDecMMI+uymhn3iy/F6gi9Frmor7BLZAhajGewdcmLfkBOMAaeX2+dNUporWB8kOzDiAgCckqN8sfN0K34yh0bduBhKIZ4t4KRsKHe4IsIHau8CtKTVET4gFW7bZdo0likoefVW4bYaECfB7yzOBRJ4fl4qyD1+Ibzlybn/799ewN893F5vEKeW40jminjFfj+uH3Mr16XGSiyjGumqFSwFwmp3r98Jm8mA6X47Ti23h7tmoVTGYjSzIcKXBPrkknSO4WQea+kCdg1u3q1y/bgUzR9fiOGFBenfCwtpwbBbfSJXC8LJHAw6BtcWnDC3i357ezhGlssc8WzrBd9jMyGa0f76r4SeF3xhB/xfX7UX0XQBz8xuPj15bCGK//fsAv7hkZma4RwtuRSSUhC7Bh24ftyD88EkEtnaCKRQKiOYyGFIpVNFRPjLKqmKsytSNL9XntbcO+Rsm4Up4aQ0UFXZbtlnN2HUY8UL89JrLFJ2W5k2FdH88/NRPHUpgsl+m1IbEIx4rKo/Jy0IJ3Pod5ig07WmSwWQCreRNhg8SuSK4BwaRPhGSuloTbZQwpeeurzlHvHjCzE4zAb82u2T0DHgiYuRTf/NP/3sEuwmPUpljs88dulaT3nbmI2koNcxjHttuGHcA86h6m8fSuTAeW1LJiBNUOqYdAewkbMrcdhNeqUlUQheO3TqiInPjaP1N096cXRuFZxz5Q1rzxb60T02E6YH7Hjo5AoePRfCaw8P1zxnwGFCIldsi+GrcDLf0nQOIKV0VlN5zc32RFql5RG+1UhFW6350lOX8SffOI4PPXhyS88/vhjDwREXnBYjxrw2XAg1jlg55/jpuRBee3gEL5ruw5NbeINoFRfDKYx5rTAZdEr6Qa2oqjZ0JTDodRh0WlRX/J0LJLHb71SiyGG3BZlCCfGM9gNYinnWBtG7edKLQDyHpVgW5wIJ9NlNW/ZbueeAHy8sxFDmwK8cmaj5ep/cmriW0v6XPpzMaSD4ZhTldIqWCNFtteC7KMLXnm8+vwiDjuGrRxfwQoMcNiClNk4tx5Xb9x0+u5IWqUcwkUMsU8DBURf2+J04H0hqHuEIZsMpTMlFy36HGV6bERdUrkfpwa/TrTLqtapO214IJbFrcN1PfX3/q/Z5bCH4PhXBByRny7OBBPb4HVsezvnDe/fhN++cxrvumKoymRP02SWBaYfho3Ci9YLvlXvRtY5yY1pF+DYjEtkiSm3y+78VukrwL4SSOLYQw6/fOQ0ANWP1G1lYk3zT9w5J3RzTA3ZcCqcaCviZirTAHr8TmUKpLXrROee4FE5hemC9S2Wnz6F6x6JM2brUu1XGvVZl/6kgkS0gmMhhh2/9+4s7BLWOnlYTTqr7oe8bcsJhNuBLT83hxGKsqrVyM/Q6hg+89gA++Dp1DxoR4beDY2Ik1bptTwIxfCS86LVCEXxb63P4ADqqU6erBP9ZueD6tlsnMOAw10xZbkTxD5fb9Hb4HMgUSlhpUIg9JwqXfif2+KVo93xQ+9bEUCKHdL5UI/gXVQR/JZaF1Vh/2nTMa8NyLFvVsXRJ7knfUfH9G3X0tJpwMgerUQ/7hkEyg16HX79zGk9eXAUDwzt/aWrbjin6vrUW/Ey+hFyxXFNUbjbieFrbC2gZ4QPaX/+V0FWCPxNKwmTQYaLPhutGXUo7Xj1EFCscBnfKYnaxQVrnbCCBQacZXrtJMaMS7YpaIiyNK+1xd/jsCCfzNY5+y7IffL3UxpjXqvjGC8TPZEfFijyfQxR4tY/wI8kcBuoMHf3WXTuw02fHb71kh1Jw3g7620TwRYTtbXGE65EFVmvHSK0EX/HT0fgO50roLsEPJrFjwA69juG6ETfOB5MN7QXm19Iw6pmy/UiI2cVwfQE/F0gobYluqxF+l7ktJnQDst1x5SYnsb/0wobrqTdlKxhXFqGsR+4Xwyno2Pr6PECKnv0ui+o6wFYTTubrbjuymw146A9egv9yz95tPabbaoSOtY/gaxXht0NKx6TXwSovfmkVYuYhUWcqvR3pOsHfKRcVD464UCpzJeeuxvxqGiMeK/Ry14m0LUinmHBthHOOS6FUVVpjesCuLAPXEjEP4Heti574WVwIXpngi0nUysLtxVASY14bzIbqX6phtwUr8fZI6TQqWuqb0J+u0zF4bdpPm4qiaasj/PUIV+sIPw+X1dgyp0yBS+TwNe5SuhK6RvCzhRLm19LYJUe1u+X8+qUG0frCWkbZ8gRI/hhjXhsW6yxnjmUKSOSKVUsmpPF67SPcYCIHo14SIMG41wqjnlV16pTKHIEtrPhjDFWF25lgsqpgKxjaxE65VYST+Zb6yAj67CasajxtqlWEr9dJ/vNapzQkW4XW+OBXQhG+hlwMpcA5lLbBMVnIL0fqR58La2mM91XndEc9VixE1SN2IYBjFW8SI24rVuJZzVuzAvEsBp2WqklLg16HqX57VeF2cS2DYpnXrMKrxGTQYchlUYra+WIZF0JJ7B921Tx30GlBKKFtW2KpzLGayrVsgXUlfXYTVjUWvDWNInxA2AtoHeG33lYBAJzyshXq0tEApYtEjkItRj2GXBbMraoXYDP5EsLJfJV4A1I6Y2NLokDp6ql4kxjxSAXOzfzng/EsvntsGTNNsiIIxnMYdNUK3g6fvao1c7ZiSUgjdvudSjrsfDCBQomrCv6Aw4REttiwVtJs1tJ5lHnrFlhXIqZNtSSa0ibCB6Q3Ge1TOtoIvs2kh17HKKWjBSLfXJlumeizKSK9kUU5it/YtTHmtSGaLiCZq71Nm1c5htKauEkv/p9/5xTe+6Xn8MD/eaIpg1qBeBZ+Z22aZqfPgblIGgW5xVIIfmX7phoHR1w4F0ggVywpjpgHhmstCcRCkLCGnirCk33A2foI32vTXvDX0gXYTXqYDK3/dXbbTG2S0mm94DMmpbTaYdJ8q3SN4C9GM3BaDFVugRP9troF2OU606aiYKmWx59fzcBtNVYdQyzCaOSLXi5zPD4ThtWox2oqj3NN6NsPxLOqEf5OnwPFMsdl+Y3vUjgFm0mvCHU9rhtxo1jmOB9I4vRyHGaDTvWuQBRKtXRNVGwVNEjp9NtNWEvnNU3pRdN5TaJ7QI7wtU7ppAuaXb/LaqQIXwsW1zI10fpknw3BRA6ZfG26QdltWhPh13aoCObXar3UxRvGcoMI/9RyHNF0Ab/90p0Aaves1mM1lcenfnph0y6gjLywpLIlU7CxU2c2nMJkv33TjoaDspf8icUYTizGsHfICYO+9r+LeOPQMo+v2CpoVLTlXNte7GimAK+99REuoL2BWKnMEc8WlY6ZVuOyGKloqwWL0UyNGAv/k3kV8V6KZcAYakRyVBF8tQg/XdXVAwBOixFOi6FhSufnF8IAgAduGcew24Kntij4//eJWXz038/g7r/9acPhJlE/GFSJ2nf67FWLSuYiaUwP1C/YCib6bHCYDfj+yRU8M7uKl+zxqT5vPcLXUvCFU6YGRVv5mFr2oq+l81XdWa3EbTMhni1odocj7L+1SOkAUuGWirYVMMbuZYydZYzNMMb+qFnHUYvwRSeKWlpnOZrFgMNck/f0OcwwG3Q1ET7nXGrj7Kud1Bz1WBumdJ6bi2Kq3wa/y4JbpvoUC4jNePRcCIDUJSPeNNRQG7oSOC1G7PU7cXRuFfliGfNraUxuUrAFpB7zew748ZOzIXAADxwZV32e8K7ROsJv9fIPgZi21XK3a1TDlIbXZgTn2nWqaDVlK3BZKKWjwBjTA/gEgPsAHADwNsbYge0+juiP39hxI4TtskrhdimWwYhKLzpjDKNea40hWiiRQ65YrirYCjZbdXcxnFS2LO0dcmIlnt3Urz+azuP5+Sh+5+W74LQY8Mxs/buC9aEr9d76W6f78NzcGn5+IYxCiePmCW/DYws++PqDmOq34VUHhlSvGwDMBj3cVqO2EX6i9cs/BCKy1rJwK0X4GqV0NPaT0VzwrVS0reRWADOc84uc8zyArwC4f7sPIgqsoxtSOl6bEQ6zAZcjta2Z0jJrdV+VMa+tJqWc1GcuAAAgAElEQVSjdOh4a4Wv0aq7UpljNpLGTrldVM22QI2fX4igzIGX7vXhyKS3Yd5fbcq2klum+pDKl/CJR2ZgMujwS7v6Gx5b4LYa8YM/uAt/97YbGz7P5zRrGuFLTpGtT+cA63c4Wk3blsocsUxB8bVpNR6rtvYKmgu+xai6Va5dabbgjwKYr/h8QX5sWxHpl40pHcYYJvpsNRE+5xzL0QyGPXX84D21vfhCoNVSOiMeK9bSBdXi8KJswSzmAyaVNFNj3/0zy3HoGHBo1INbp/txIZSqu04umMjBZNDV/U9/63QfGAOemV3Di6b7YDNtfSrRbNi83W/AYdI4h5+rWXzSKkSEv6aR4MczBXCuTQ8+sB7ha2WgJgTfo9EdjtNiRCpf2vIubK1ptuCr3WNXVXcYY+9mjB1ljB0NhUJXdZBRrxW/fsd0lbGXYLLfhrkNgh/PFpHKl+raC4x5rVhN5avSLqKff9SjltKpbxMsjMuEMZuoK6ilmSq5FEljdMP2qrN1TNoC8Sz8LnPdzhu/y4KPv/VGDLks+JVb1HPx14JP42lbafmHNoJnMujgtBg0i/AVp0ytunQUi2Rtrl+rbVcCYTHeKZ06zRb8BQCVCjMGYKnyCZzzT3POj3DOj/h86p0gm3FwxI0/e90B1Shnos+GhdVMVReBkgJSEW9AvRd/YS2DAYcZVlOtI9+I2PykktZRbIXlQSePTerq2UzwZ8MpTA9IbxLi7qCebXMwnlMduqrk9deP4Mk/eQVee3ik4fOuBinC1+YXnnOOcCpfs+mqlfTZtRu+ErYKWhZtAe3WPLZDSgfoHAO1Zgv+MwB2M8amGWMmAG8F8GCTj1nFRL8N+VJZyXMDlROz9XP4QHVr5ryK745A9PIvq3TqXAwl4bYalWUZ9dJMlSjbq+Q7liGXBRajTrGP2Eggka1bsG0FPqcZyVxRNaXVbBK5IvLFcs2mq1aipeBHFS98ba7faTGCMe2KtvFMASaDDpYWWyMLFMfMDincNlXwOedFAO8D8AMApwF8lXO+te3i28RknxQdV7Zmrm+6ahzhV7Zmzq/V9uALhtwWMAbVVYdzkTSm+m1V6ZbJflvDYapwMo9krogp+a5Ap2OYHlDfXgXU99FpFVr24oflVJJWRVtAas3UOsLXqktHL7fDxjQs2moV3QOAyyJSOhThAwA459/jnO/hnO/knH+k2cfbyITSFbMusAtrGTjMhrqFHtGLL94k0vkiFtYyqvbAAGDU6zDoNKu2Zl5eTWNiQ9/7uNwFxLn6sIqa380Onx0XVSL8ZK6IZE59yrZViGnboAZ5fJE711Lw2yHC1yqlA0hvNmsaFm21FHwnpXTaixGPBQYdq3LNnF+VLBLqFTl1OobdfodSJJ0JJsG5tBC7/nGsNZufCqUyFqMZpTNHMOy2IF8q1y30idRN1X7aATvmV9PIFavTJsFNWjJbgW8LEX4gnsUjZ4LbfmwR4Wub0jFjNZWv+wbeTKLpguJLrxVuDS2StWxJBdaLtpTSaRMMeh1GvVZcXt2Yj29sL7B/yKXYEZyVbYL3+BsIvttaU7RdikrF4okN3UPDnvpFXgBYWE1Dx6p9fnb4HChz1KSCxN7ZzYq2zWQzP510voh3fOYpvOtzz2y7PbTio6NxSidfKqs6rDabtXQebg22PVUi+elo16WjaUqnw7Zedb3gA1JaRwxfcc4xv5qpm48X7Bt2IZzMIZTI4VwgAbNB19CSYMRjwVKsOk0jCrMbl42sd/Wo2zEsRDPwuywwVpiVKZ06G9I6olA8vI3Lua8UUZCuF+H/8+OzOB9MQseArz23sK3HFt1B4hy0oE/DZeaSrYJ2ggdo64mvdUrHYTKAManVuxPoGcEXvfiRVB6ZQqlux41gv5y+ObuSwJmVBHb7HQ33og67rcgWylW5TFED2DgfIAa+6tkxLEVrfYFEemdja6b4Ho1WFjYbo16HPrupboT/i8tr2OVz4GV7B/H15xa2vA+Ac45jC9GGxlzhZA5em1HVybNV9DnEG17rBV9L4zSBR0NP/HimoJlTJiClfx3mzjFQ6xnBj6YLiGUKOCOnacQqxHrslQX/2GIUp5biDdM5wHr6pTJNM7+ahsmgq0m39NtNMBl0dXfBLkYzNbbNTosRPqe5plNnKZZFn92kWVuaoNG07cmlOA6OuPCK/X4E4jnVATU1Hj4dxOv/4XE88H+eqLtRa7Pl5a2gX8MIfy1d0KxDR+CxGRHPFls+bVoqcyRyRU0jfKCzDNR6QvBFhD2/msaxxSgA4PCop+G/6XeYsdfvxN8/PINIKo/7rhtu+PxRFcGfjaQw7rXWmHoxxur675TKHMvRbI0vECANb9WmdDKaRveCen46q6k8lmNZHBhxKW+yF+oMkG3kO8ekGb1n59bw/RMrqs+JJPOaFmyBypRO67uUtFx+IvAoeezWpjXiGg9dCVxWIxVt2wlha3BqOY5j8zFM9tvg3kJU9P579iBTKGF6wI5X7Bts+NxhlVWHM8EkdvrU7ySG3RbVCD+UyKFY5jUpHXEdGyP8RiZwrcTnMKu2ZZ5aigOQpqFFHeLCFgq3uWIJD58O4k03jcJpMeCpSxHV57VHhK/d1i8tnTIFir1Ci9M6WvvoCFwWA0X47cTuQQf8LjMeORPEsYUoDo81ju4F9xzw4113TOG/v3b/pta7G9M0hVIZc5F03dTRiNuquiWr3q5dQFpmspYuVBl1LbVJhD/ktiIQz9bk508uxQBIG7T67Sa4rcaqper1eHZuDYlcEfddN4xbp/rw5EV1t9BwUjunTIHVpIfNpG95SiedLyJbKKNPg9WOlQjBbXUvfrRNInxnB2296gnBZ4zh7v1+/PuJFSzFsrh+zL3lf/fB1x3Ey/f5t/RcaRGKJOJzkRSKZV5X8Ic9FgQSuZqC5EIdq2egonArp3VSuSLi2WJd189WMuKxoFDiCG9Ia5xcimPUY4XHZgJjDDt99i0J/vmA9JzDY27ctqMfl8KpKnsMAMgWSkjmipoZp1WixfCVWLrSr2GHErAe4cdabKCmtY+OQPLEpwi/rbj7gCTaeh3Dqw81zsdfLWNeq9KZI/rN6wq+24pSmSvrCQVLdXbtAuupKZHWEXcTI22Q0hmu02p6cimGA/J+XEBaqr6VHP6lcAp2kx6DTjNume4DADw3V70pLNQGtgqCfrup5Y6Z4nha1zBEDr/VrZltI/hUtG0/XrLbh79682E88ccvVxXT7eDAiAtnVxLIF8uK4NfL4SuWyhsEcjGahtsqLW7ZyLjXCqOeKRG+sIvYuMtXC8T1VKap0vkiLoZTykJ0QHrTCiVym3qPXAglscPnAGMMe/1O6BhweqXaHlq84Wk5gyDos5vq7itoFuJ4Wu0CECg7AXpV8K1GJHPFLbcba0nPCL5Ox/CWI+MYbOJE6uFRD/KlMs6sxHEukMSI2wK7inAD6xHxxl58td28AoNeh4k+mxLhn5EFcPcmLaOtQBkmqyhEn1lJgHPgwPC64ItUVb2WVMHFUEpJYVlNekz123F2JV71HPGzU1tV2WqEvUIrUSJ8jVM6TosBOoaWG6iJNIqWffiAVLTlHEhusra0HegZwW8Fh+XawPPzUTw+E8bNU311nysEcqOl8lKdlkzBDp9D8do5F0hg2G3RPMIBpMKdxair6lI6KTp0RtdrJqLAXM9WApBy80uxarO6fcNO5Q1OsNQGU8aCAYeU0mmln46Sw9c4paPTMbitxpb76cQyBViM2lkjCxRP/A7I45PgbyNjXiv67CZ87vFZRFJ5vPJA/WKvy2qAzaSvGkLinGNRZcq2kh0+O2YjaZTKHGdWEsqAmNYwxmqWuZ9aisFjM1ZF4ELwVxpE+LORFDhfr1kAwL4hFy6vppGq8KtZjmXgtBhU01+txuc0I18sK2mGVhBJ5mA16q9oZWWz8NhMre/SkX2EtKaTDNRI8LcRxhhu39mv5Nhfurf+Bi8xfFUZ4cczktVxQ8EfsCNfLGM2ksKFYBJ72yCdIxhxW7FYcT1iwrbS2MvvknYHLDUS/LBUm5iu8C7aO+QE59JdjWA5lm2LgjUg7UQA1s3sKknminjwhaWGFhFXw2pK+6EzgVsDAzWtfXQEIsJv5Zv91UKCv8189E2H8KYbR/Hrd0wr/xHqsTEiFi2djVI6N054AQCfeewS8qVy20T4gDQ7ML+aBucchVIZZ1YSODhS3QJr1Ovgc5ix0sBeQcnNV7Sb7h+S6gBnVxJVzxtqg/w9sH7nolab+NhDZ/G7X/4FPv6jc9t6zHAqr3n+XqCFgVq7CL4Y4mx1W+rVQIK/zTgtRvzNr9yAP3vdgU2fO+y2VEW6iuA3iPD3+J04PObGl566DKOe4ZYGdYJWs3/YidVUHoF4DhdDKeSL5aqCraDelLFgJZaFyaCrcsAc81phM+mr8vjL0WzVm4KWiAU0gQ3XtZrK4ytPz8NhNuDvfjxTtYjnWokkc5p36Ag8NlPLF5nHMtr76ADadSldDST4GjLutSGUyCEtV/cX5ZWKm7WNvuWItBf+P79k56a+/q3kOrk4e2IxVjVhu5Fht7Wh4C/Fshh2W6pSQTodw94hJ04vS4XgbKGESCrfFrYSADDolFJVG6/r2y8sIVMo4cP3HwQgFfSvhOMLsbrGcattFOFLKZ0WR/jpPNxW7a9/XfApwicaoJiJBaWc/0woCafFsOnk6ANHxvBXbz6M9718d9PP8UrYP+wCY1Lu/heXo7Aa9VWFV8Gwx4LlaP0Vj8vRDIZUVjbuG3LibCABzrmyb7jR3VArMRl0GHCYa4rRT8+uYsRtwWsPj8CoZzghvxFuhWMLUbzuHx7Dr33m6ZrlKpxzRJJ5xZpZa7w2ExItdMzknCOcyrfFlLXVpIfZoNNsJ8CVQIKvIbv9khjOhKQ0xbkVqQi72fYis0GPtxwZh8nQXi+f3WzAjgE7ji1E8f2TK3jpXp/qDoFhtwWpfAmJOhuilmNZ1bucfUMuRNMFBOI5nFgULZ+1dxBaMey2VBVtOec4OruKI1N9MBl02DvkxMnFeIPvUM2/HV2ASa/D07Or+OKTc1Vfi6TyyJfKqm+MWuCxtbZwmcwVkS+W22LKGpDe8LTaCXAltJdi9BiT/XYYdAznA0lwznEumMCeNirCXg2Hxzz4ybkQQokcXnt4RPU5w3VmEADJHjoQz6oawomdwieXYji2EIPFqMOuOpPMWuB3Waoi/IW1DALxHG6Zkgrt1424cWIptqVe/VyxhAdfWMJ9h4awb8iJn5wNVX19cW3zek8rEfWWVtlLhNtkBkHg0XCR+5VAgq8hRr0OUwN2nA8mEUrkEE0X2qrN8mr4nZfvwpDLAqfZgJfXsZRe72ip7dSJJCV7aDXBv37cA6tRj5+cDeH4YhTXjbg13XS1EakYvX5Nz8xKDp9H5ML6wVE3oulCw5ZUwenlBGKZAu49OISX7PHh6Nxq1QzCVjq6WolStFZpS20G7WIrIfDYtNvreyW0z29Lj7J70IELwSTOBoRNQvtErFfDDp8D3/vdF+Pbv3MnrCb1CUgxGatWuBViqFaMtRj1uGvPAL5/cgXHF2M4tEXX01Yx6rEini0iJkd6R+fW4DQblG1pO2SriLnI5uZxwotpz5ATd+3xoVDieOLC+k4AMak85mmPor1ILTUaqNtOwm3iFCrwajB4djWQ4GvMviEXZiMp/OCktNGp0yN8QOpLnhqov/B90GlW7WgB1s3X6lk+33NgCKFEDtlCGbe2UUsqIC2+B4CTy1Jh9ujsKm6a9Cp1jPWtaJuL4oVQEkY9w2SfDUemvEouX7CwloHDbFCmPLVm0CVF2q2K8MU6TZ+zXSJ8yuETW+CNN46CA/jCk5dx+47+trlFbSZGvQ6DTrPqApjlBhE+ANxz0I/XHB7GX775MO69bqip53mliBbUU0txRNN5nAsklfw9sD6NK/LvjZgJJjHVb4dBr4PZoMd1oy48W2EPLSw4NivwtwqLUQ+Pzag6adwMhI+Q1gvcBWLwrJVeSlcDCb7GTPTbcPd+yXPn9+5urzbLZjJUpxd/OZaB2aCru7bPaTHiE2+/CQ8cGW8bsRMMOMzwu8w4uRRXxPlIxV2IxaiHz2luaBwnuLBhPeZNE14cX4whX5TaHhfXMm2TvxcMuSxYibXGIjqSysFtNbZNp5rHZkRRXqrezrTHT6vH+cBr9uN/vvEQbtvRr/WptIyRDQVOwZLcktluYr5VDo64cWopju8eX4bdpMcN49XrNCu3otUjXyxjbrV6PebNk17ki2VloG0zkz0tGHRZahb6NIt2WF5fibL1q83z+CT4bcBkvx1vf9GE1qfRUoZke4WNt8ArsWzb9JZfDdeNuDATSuK7x5Zx/42jNda9ox7rphH+5VXJDbXSHvqmSSk19PSlVazEsohlCg3rJFow5KodPGsWoTZYXl9Jp0zbkuATmjDitiKdLyG+YfnzcjTTFjt6r5ZfvX0Sk3025IplvP3W2jfxUa8U4TfK9Yo7n8oI3u+yYN+QEw+fCeLnF8IAgNt2tFfReshlQTiZq5q2jWUK+B/fOYWvP7ewrfntSDLXFlO2gj67lIJs9ZrLK6U9SvxEzyFEfTmWUQywSmWOQCKn2oPfKQw6Lfi399yO08sJxVuokhG3BbliGZFUvm6EKqLkjU6grzzgxycemYHLYoDHZlQcRNuFQZcFZS5F36Lo/rnHZ/FPj10CINVfGu2I2CqccyzHsnjx7vr2461GbNILtqhofbVQhE9ogpqdcCiRQ6nM28YQ7Wrpd5hx5+4B1a+NKK2Z9dM6orXRvyG1dfd+P8oc+NHpIG7f0Q+dim2FloiVlMIbKl8s4wtPzeHOXQOwmfT42flQo3++ZcLJPNL5Eqb622MGAahsS23tXuMrhQSf0AQ1e4UlFR/8bkOIeLCBMCzHsvDajDX5/0OjbvynF09jzGvF/TeMNvU8r4b98hzCKXkO4eHTAYQSOfzGi6dx63QfHp8JX9H345zjb354Do+cCVY9fnlVekOZ7G+fGobZoEef3dSyOYSrhVI6hCYMOs3QMVQtQhHiP+Tq7Ai/EYoFQYNulkA8WxPdA5JF9J++5gD+9DWb71rQgj67CcNui7LL+HsnVtBnN+HFuwYwE0jiI2dPS0X5Labsnrq0ir97+DwYA/7322/CfYeGAQBzEckpdaKNInxA+j9NET5BqGDQ6zDorF4AMydHbu32i7ydDDhMYKxxhL8S37oothsHhl04tRRHtlDCj08HcM8BPwx6HW6ZlgrMV7IP4PM/n4XHZsT0gB2f/tlF5fHZSBqMSUtx2gm/y9L2ET4JPqEZQ+5qd8m5cBoDDlNbLCVvFga9Dv12c8N+9ZWYultoJ3BwxIULoSQefGEJqXwJr5KnocV+4q1u/MrkS3joVABvvmkMDxwZxy8uRzEr74q+HElhxG2F2aDu1aQVQ70s+IyxDzHGFhljz8t/Xt2sYxGdyYjHouTtASnCb6e8bLPwu8x1I/x8sYxwMq+a0ukEbp7qQ5kDH/jGCezxO3CX3EnjthnhthqVu7jNmAkmUSpz3Dzpxf03jIAx4JvPLwIA5lbTmGijTW8Cv8tc05babjQ7wv9bzvkN8p/vNflYRIcx5LJiObo+fHU5ksZkG/4ibzeDTnPdHL6I/Dt1+Oyu3QN4x22TyJfK+MNX7atagDPZb1Py75txTnGPdWLYbcXtO/rxjV8sIlcsYSaQxLSv/QID0Zbazr34lNIhNGPEY0GmUEIsU0C2UMJyPNvV+XuBlOtVj/BFisvfoSkdxhj+/PUH8eP/8hLcvaHnfrzPtuWUzrlgQnILlf8/vOHGUcxF0vj4j84jkStuSz//dtNqi+irodmC/z7G2DHG2GcZY161JzDG3s0YO8oYOxoKbU+fLtEZCHOwsysJLKylwTmUX/BuZtBplha9qNz6C7fJTs3hA1I3kdou48k+GxbWMltKeZwPJLFjwAGjvODmvuuGYDHq8MmfXkCf3YQ7d6nPOWiJmLG4vMU3NS24JsFnjP2IMXZC5c/9AD4JYCeAGwAsA/iY2vfgnH+ac36Ec37E52ufyTmi+Yh9tMcXY5gNy612fe13q77dNLr1V6ZsOzSl04jJfhuKZa7qkrqR88FE1TIgp8WIT/6Hm+E0G/CWI2PKG0E7sWvQAaOe4dTy1vcWt5praofgnN+9lecxxv4RwHeu5VhE9zHotMDvMuPEYgzhZB5GPVP21nYzlcNXG4uzgXgWZoNOsZvoJsbl+szl1bTysRrZQgnzqxn88k1jVY+/bN8gjn7glVV1gXbCZNBh16ATp5baV/Cb2aUzXPHpGwGcaNaxiM7l0KgbxxdjeHwmjBsnvLB3cUumYNBZfzvUstyS2an20I0YkaerN8txiw6mERX7Z5NB17aCD8hzCHKEzznHw6cD+OsfnG2ZbfRmNPO36y8ZYzcA4ABmAfxWE49FdCjXjbrxo9PS6Pz7X7lH47NpDY2mbetN2XYDwm8mmGg8jSrqGJ2Y1jow4sLXnltAMJHF05dW8b4v/QIAkC+V8Sev3q/x2TUxwuecv4Nzfohzfphz/nrO+XKzjkV0Lq89PKJ8XM9wrNtoNG3byVO2m2EzGeAwGzYdTqpnHtcJiDWX3/rFEj787VO4btSFV+wbxNefW2yL/vzuv38m2ppdgw4c/cDdeHZuDTdu2A7VrdSbtuWcIxDLda3gA1KUH9okwg90cIR/ZNKL68c9+Mj3TsNs0OEz77wFy7EMHj4TxM9mwnjZ3kFNz6/9St1EzzHgMONVB4e6Mm9dD7Vp29VUHvlSuSOFbqsMOhvbSgDrhWuXtfPiUYNeh7994HrcOOHBp95xMw6NuXHXHh8YA164Ah+hpp2f1idAEL2I2rRtJ+eut8qg04IXFhoL30pcusvp1ABgh8+Bb/z2HcrnFqMeY14rZoJJDc9KgiJ8gtAAtWnbJdkeWq07pVsYdEp3No3WHQbiWfid3fWmt3vQSYJPEL2K2rSt2ILV1YLvMiNTKCGRK9Z9TjCe7VhriXrsGnTgYjiFUnn79vpeDST4BKEBatO2S9EMTAYd+u3ts5x7u9ls4xfnHCvxLPxO9X2/ncounwP5YhkLa9raLpDgE4QG+FWMtpZiWYy4LW23q3Y78TlFL7564TaeLSJbKHddp9LOQckmQuu0Dgk+QWiA2Na0sLa+D2Apmun4Be6bMSjn5uu1ZoqWzMEuK1zvkBe8z27RHrpZkOAThAZU+soIlqKZrs7fAxXTtnVSOp3cg98Ij80Is0Gn+UYsEnyC0ACH2YB+u0kR/EKpjEA8i1FPdwndRpxmAyzG+sKn7ANwdVcOnzGGIbdlS06hzYQEnyA0onIhSCCeRZl3d4cOIAnfoNNS109HPN6Jtgqb4XdZECDBJ4jeZKLPpkT4l+Xc7pi3+xfA+F31p21XYlm4rUZYjO21oHw7GHJZlOE6rSDBJwiNmOizYTEqbYASlrp7e2AfQKMIPxDPdl3+XjDslgS/0dBZsyHBJwiNmOizoSRvgDqzksCAw6y0LXYzPqcZoXpF20ROKex2G36XBfliGWvpgmbnQIJPEBqxwye16p1ciuP0chz7h7s/ugekTp1Eroh0vnbaNhDr3n0AYrZAyyXnJPgEoRGHxzywm/T4ydkgzgeSODDs0vqUWoLoxd/Ymlkqc4SSua5N6SjDdvHMJs9sHiT4BKERJoMOt+/sx1eemUe+VMb+HhF8f53NV5FkDqUy77qWTMF6hN94H0AzIcEnCA25a48PAGAx6vAS+eNuR4nwN3TqCPfQbk3p+BzSG9lmC2CaCfnhE4SGvP76EZxejuPdd+2Et4tN0yoRS9w3pnRWOni14VYwGXRwWQwIJ0nwCaIn8dhM+Is3Hdb6NFqKx2aESa+rWQCj2Cp0mXFaJQNOMyIpSukQBNEjMMZUWzMD8Sx0DF1tDz3gMCOcyG/+xCZBgk8QRMvxOc01RdtAPAuf0wyDvntlyecwa5rS6d6fLEEQbYuavcJKPNe1+XvBgMOEEAk+QRC9hJq9QjDevUNXggGHGYlsEdlCSZPjk+ATBNFyBp1mRNOFKuELxLNd24MvGJA7lFZT2uTxSfAJgmg5wi9H9KRnCyWspQtdO2UrEAVprfL4JPgEQbSc9eErSfiWopLdQLeveBQRPgk+QRA9w/qqQ6lwOyfvA5ga6O59AGLaVqvWTBJ8giBajtjpOycvgJmNpAAAk/12zc6pFQwIewWK8AmC6BVcFiN8TjMuBJMApAhf7PntZqwmPewmPaV0CILoLXb67LgQkgR/NpLCZL8NjDGNz6r5DDjNiCQppUMQRA+x0+fAhVAKnHPMRdKY6vJ0jmBAw2lbEnyCIDRhp8+BWKaAYCKH+dU0Jvu7u2Ar6LebSPAJgugtxIrHb7+whGKZY9egQ+Mzag0DTjPCnZjSYYy9hTF2kjFWZowd2fC1P2aMzTDGzjLGXnVtp0kQRLexb8gFxoC///EMDDqGl+8b1PqUWsKAw4y1dB7FUrnlx77WCP8EgDcBeLTyQcbYAQBvBXAQwL0A/jdjTH+NxyIIoosYclvw+utHEMsU8OLdA/DYurtDR+BzmMA5sJpufZR/TYLPOT/NOT+r8qX7AXyFc57jnF8CMAPg1ms5FkEQ3ccf3L0HDrMBv3LLhNan0jIGNBy+atbGq1EAT1Z8viA/RhAEoTA1YMcLH7wHel33t2MKtLRX2FTwGWM/AjCk8qU/5Zx/q94/U3mM1/n+7wbwbgCYmOidd3mCICR6SeyBigi/HQWfc373VXzfBQDjFZ+PAViq8/0/DeDTAHDkyBHVNwWCIIhuYcAh1SpCidYLfrPaMh8E8FbGmJkxNg1gN4Cnm3QsgiCIjsFhNsBu0mMlnt38ydvMtbZlvpExtgDgdgDfZYz9AAA45ycBfBXAKQDfB/Bezrk2K14IgiDaCMYYhtwWrMRaL/jXVLTlnLmk238AAAaiSURBVH8DwDfqfO0jAD5yLd+fIAiiGxnxWLGkgeDTpC1BEESLGXJZsBLLtPy4JPgEQRAtZthjRTCRQ6HF07Yk+ARBEC1m2G0B5+srHlsFCT5BEESLGXZLO32Xo61N65DgEwRBtBixrH25xYVbEnyCIIgWM+yRIvxFOcJ/26efxL88Odf045LgEwRBtBiXxQi/y4wzy3EEE1k8cTGCXKH5o0ok+ARBEBpwaNSD44sxHJuPAQCuH/c0/Zgk+ARBEBpwaNSNi+EUnrgYgY4BB0dcTT8mCT5BEIQGHBpzgXPgK09fxu5BJ2ymZrnVr0OCTxAEoQHXjboBAKl8CYfH3C05ZvPfUgiCIIgaBp0W/NF9+3B2JYF33D7ZkmOS4BMEQWjEe16ys6XHo5QOQRBEj0CCTxAE0SOQ4BMEQfQIJPgEQRA9Agk+QRBEj0CCTxAE0SOQ4BMEQfQIJPgEQRA9AuOca30OCoyxEICrNYUeABDextPRErqW9qRbrqVbrgOgaxFMcs59mz2prQT/WmCMHeWcH9H6PLYDupb2pFuupVuuA6BruVIopUMQBNEjkOATBEH0CN0k+J/W+gS2EbqW9qRbrqVbrgOga7kiuiaHTxAEQTSmmyJ8giAIogFdIfiMsXsZY2cZYzOMsT/S+nyuBcbYLGPsOGPsecbYUa3P50pgjH2WMRZkjJ2oeKyPMfZDxth5+W+vlue4Fepcx4cYY4vy6/I8Y+zVWp7jVmGMjTPGHmGMnWaMnWSM/Z78eEe9Lg2uo+NeF8aYhTH2NGPsBfla/lx+fJox9pT8mvwrY8y07cfu9JQOY0wP4ByAVwJYAPAMgLdxzk9pemJXCWNsFsARznnH9RYzxu4CkATwfznn18mP/SWAVc75R+U3Yy/n/L9peZ6bUec6PgQgyTn/ay3P7UphjA0DGOacP8cYcwJ4FsAbAPxHdNDr0uA6HkCHvS6MMQbAzjlPMsaMAB4D8HsA3g/g65zzrzDGPgXgBc75J7fz2N0Q4d8KYIZzfpFzngfwFQD3a3xOPQnn/FEAqxsevh/A5+WPPw/pl7StqXMdHQnnfJlz/pz8cQLAaQCj6LDXpcF1dBxcIil/apT/cAAvB/D/5Meb8pp0g+CPApiv+HwBHfofQYYDeIgx9ixj7N1an8w24OecLwPSLy2AQY3P51p4H2PsmJzyaesUiBqMsSkANwJ4Ch38umy4DqADXxfGmJ4x9jyAIIAfArgAIMo5L8pPaYqOdYPgM5XHOjlPdQfn/CYA9wF4r5xeILTnkwB2ArgBwDKAj2l7OlcGY8wB4GsAfp9zHtf6fK4WlevoyNeFc17inN8AYAxSlmK/2tO2+7jdIPgLAMYrPh8DsKTRuVwznPMl+e8ggG9A+s/QyQTk/KvIwwY1Pp+rgnMekH9JywD+ER30ush54q8B+CLn/Ovywx33uqhdRye/LgDAOY8C+AmA2wB4GGMG+UtN0bFuEPxnAOyWK9wmAG8F8KDG53RVMMbsckEKjDE7gHsAnGj8r9qeBwG8U/74nQC+peG5XDVCHGXeiA55XeQC4WcAnOac/03Flzrqdal3HZ34ujDGfIwxj/yxFcDdkGoSjwB4s/y0prwmHd+lAwByK9b/AqAH8FnO+Uc0PqWrgjG2A1JUDwAGAF/qpGthjH0ZwEshuf4FAHwQwDcBfBXABIDLAN7COW/rgmid63gppLQBBzAL4LdEDrydYYzdCeBnAI4DKMsP/wmk/HfHvC4NruNt6LDXhTF2GFJRVg8p6P4q5/zD8u//VwD0AfgFgF/lnOe29djdIPgEQRDE5nRDSocgCILYAiT4BEEQPQIJPkEQRI9Agk8QBNEjkOATBEH0CIbNn0IQ3QdjrB/Aw/KnQwBKAELy52nO+S9pcmIE0USoLZPoeTrVCZMgrhRK6RDEBhhjSfnvlzLGfsoY+ypj7Bxj7KOMsf8ge5kfZ4ztlJ/nY4x9jTH2jPznDm2vgCDUIcEniMZcD8mr/BCAdwDYwzm/FcA/Afgd+TkfB/C3nPNbAPyy/DWCaDsoh08QjXlGjOozxi4AeEh+/DiAl8kf3w3ggGT3AgBwMcacsm87QbQNJPgE0ZhKL5NyxedlrP/+6ADczjnPtPLECOJKoZQOQVw7DwF4n/iEMXaDhudCEHUhwSeIa+d3ARyRty6dAvAerU+IINSgtkyCIIgegSJ8giCIHoEEnyAIokcgwScIgugRSPAJgiB6BBJ8giCIHoEEnyAIokcgwScIgugRSPAJgiB6hP8fftNwytjfrj0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "t_min, t_max = 0, 30\n",
    "resolution = 0.1\n",
    "\n",
    "def time_series(t):\n",
    "    return t * np.sin(t) / 3 + 2 * np.sin(t*5)\n",
    "\n",
    "def next_batch(batch_size, n_steps):\n",
    "    t0 = np.random.rand(batch_size, 1) * (t_max - t_min - n_steps * resolution)\n",
    "    Ts = t0 + np.arange(0., n_steps + 1) * resolution\n",
    "    ys = time_series(Ts)\n",
    "    return ys[:, :-1].reshape(-1, n_steps, 1), ys[:, 1:].reshape(-1, n_steps, 1)\n",
    "\n",
    "t = np.linspace(t_min, t_max, int((t_max - t_min) / resolution))\n",
    "\n",
    "plt.plot(t, time_series(t), label=r\"$t . \\sin(t) / 3 + 2 . \\sin(5t)$\")\n",
    "plt.xlabel('Time');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "n_steps = 20\n",
    "n_inputs = 1\n",
    "n_neurons = 100\n",
    "n_outputs = 1\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=(None, n_steps, n_inputs))\n",
    "y = tf.placeholder(tf.float32, shape=(None, n_steps, n_inputs))\n",
    "\n",
    "cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu)\n",
    "outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)\n",
    "\n",
    "# Stack all time steps on top of each other to get one output for each neuron\n",
    "stacked_rnn_outputs = tf.reshape(outputs, [-1, n_neurons])\n",
    "# Fully connected layer with proper number of outputs\n",
    "stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)\n",
    "# Unstack to get outputs for every neuron\n",
    "outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.reduce_mean(tf.square(outputs - y))\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0  MSE:  4.5465198\n",
      "100  MSE:  0.12235968\n",
      "200  MSE:  0.084332034\n",
      "300  MSE:  0.06427125\n",
      "400  MSE:  0.048832428\n",
      "500  MSE:  0.051071092\n",
      "600  MSE:  0.045697305\n",
      "700  MSE:  0.04467769\n",
      "800  MSE:  0.04656992\n",
      "900  MSE:  0.049167868\n",
      "1000  MSE:  0.053675525\n",
      "1100  MSE:  0.0419274\n",
      "1200  MSE:  0.03483821\n",
      "1300  MSE:  0.037629236\n",
      "1400  MSE:  0.04460977\n"
     ]
    }
   ],
   "source": [
    "n_iterations = 1500\n",
    "batch_size = 50\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for iteration in np.arange(n_iterations):\n",
    "        X_batch, y_batch = next_batch(batch_size, n_steps)\n",
    "        sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "        if iteration % 100 == 0:\n",
    "            mse = loss.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "            print(iteration, ' MSE: ', mse)\n",
    "    \n",
    "    # Time values\n",
    "    t_instance = np.linspace(12.2, 12.2 + resolution * (n_steps + 1), n_steps + 1)\n",
    "    # Reshape to feed into network\n",
    "    X_new = time_series(np.array(t_instance[:-1].reshape(-1, n_steps, n_inputs)))        \n",
    "    y_pred = sess.run(outputs, feed_dict={X: X_new})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEXCAYAAAC59m+aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xt4VeWZ9/HvzSYakpCiSN9piwpl6gBiCBCMFeUgDhCJWkqspcUXRetYsO1Q6CtTbe0FM9ODrafaDmPVjlVKW3CwknGb1mmhWjQ2SkCrDGNGrKjtICqSA5K9ud8/1k5IQs57J/uQ3+e61rX2Oj9rsbn3k2c9B3N3REQkcwxKdgJERCSxFNhFRDKMAruISIZRYBcRyTAK7CIiGUaBXUQkwyiwS0oxs2+ZWVUSr/9nM7s+WddPBDN73MzW9WD/sWbmZjahL9Ml/UeBXY4T+0/e2fRvCbhGR8HkH4G58Z6/G9e/zsze6uvriCTD4GQnQFLSh1p8LgV+1GZdQ19d2N1rgdq+Or/IQKAcuxzH3f/cNAHvtl3n7gcBzOx0M9toZu+a2QEze8TMRjedx8xGm1m5mb1jZnVm9qKZfdLMsoGXYrs9H8u5PxY7plVRjJn9zMw2mdlXzOzN2HV+ZGYnttgn38x+GrvGm2a2srPiCDObB/wLMLzFXyGrW+ySa2b3mdkhM3vNzL7Y5viTzexeM9tvZu+Z2W/MrLCzZxor4vkHM3vQzGrN7NXYszg5dn+1ZrbbzGa2OW62mf3BzN6P3dt3zCyrxfahsXM23fuqdq6dbWbfM7PXY/tVmtkFnaVX0psCu/SKmQ0FtgLvAOcD5xH8CPy6RdC9GzBgOnAWsAp4z90Px44BmEnw18CiTi73t8AoYBZwBfBpYFmL7XcC5wAXx/adBkzt5Hy/AW4A3o5d+0PA91tsXwU8A0wC7gDuMLPJsfsOAY8Bw4ESYApQBfzGzEZ0ck2AlcA2oBB4BHgAeBD499i1/gCsN7MTYtcaBfwH8DQwEfg8cBXwjRbnvIPgWV4CzIl9PrvNddfH1l0OFAA/B8JmNq6L9Eq6cndNmjqcgLLga3Lc+mXAC23WZQGHgEtiy3uAGzo471jAgQlt1n8LqGqx/DOgBhjUYt0DQHns88lABPhEi+0fiKVjXSf3dR3wVjvr/wz8uM2614BVsc8XEfwgnNBmn93AFzu5XqvzAqfE7v87HT0T4HvAHwFrk+762LNuuveFLbYPIyjKWhdbHg9Egf/TJj2PAbd29m+hKX0nlbFLb00BxppZ2/LwHGBM7PPtBLndS4D/BP7d3at7ca0X3P1oi+U3gL+Jff4YECLIYQPg7gfNbHcvrtNkV5vlN4APxj5PIfjheNvMWu6TzbH77vK87v6WmUWB51ts/0ts3nStccB2j0XfmCeBIcBo4CSCe3+qxXnfNbOXWuw/heAv85o26T0ReL+L9EqaUmCX3hoEVAJL2tn2FoC7/9DMyglyuRcCq83s6+7+rR5eq7HNsnOsGNFarEuUzq43CNgHzG7nuIM9PG/bdU330PLeOrov59i9d2ZQ7BqT2jlXXTeOlzSkMnbpreeAM4C/uPvLbaZ3m3Zy9z+5+zp3LwP+Cbg2tulIbB6KMx17CIoamsuVzSyfoHihM0d6ee3ngA8D77dz3/t7cb7OvAhMs9ZZ7fMIaiXt5di9n9O00cw+QJDTb5neLOCUdtL7ZoLTKylCgV16636CcuyHzez8WA2YGWZ2h5mdDmBmd5nZnNi2yQQvNl+MHf8mQXCdZ2YfjAXjHnP3twleQH7PzGaa2ZnAfcBROs/F7wU+EEvzKWY2pJuXfJQgWD4Su7dRZnaumf2jmRX35h468X2C4p07YvX+LwXWAre5e2Ps3h8guPcLYm0C/o3g3gFw9+eBhwheyi6I/VtMNbMbzOziBKdXUoQCu/SKu79HkHt8g6BWx0vAjwnK2JuKJLIIqhW+RPCy7lXg6tjxDcAK4HqCIP+LOJLzRYIaJY8CjwPbgReAw50c89tYev8d2A98qTsXcvcoQe2T7QRBdA/BC97RBC9IE8bd9wLzgXOBncC/xtL8jRa7fYmgjH0Lwb0/TYv3DTGfBX4K3Ar8F0GNnHOAPyUyvZI6rPV7GZH0F8t97wO+7u4/SHZ6RPqbXp5K2jOzswlyzFUENVZuJPhrYVMy0yWSLArskgmMoMHRGQTl9juA8939L50eJZKhVBQjIpJh9PJURCTDJKUo5pRTTvFRo0Yl49IiImnr2Weffcvdu+qTKDmBfdSoUVRVJW0sBRGRtGRmr3ZnPxXFiIhkGAV2EZEMo8AuIpJhUqYee2NjI/v27ePw4c5agUtPZGdnM3LkSLKysrreWUQyRsoE9n379jF06FBGjRpFm36jpRfcnQMHDrBv3z5Gjx7d9QEikjFSpijm8OHDDB8+XEE9QcyM4cOH6y8gkZiaGlh+XZQlw8t5cdFalgwvZ/l1UWpqunfssmUwciRs3hzMly2jW8cmQ8rk2AEF9QTT8xQJhMPwqYVRfnl4LueGKsn+eR3rQrk8dXcxhT+p4BcPhSgp6fjYsjKIHonyT+eFWfDCDmr+ehI3/aiE++8PsWkTHR6bLCmTY++Jpl/P/HwYNCiYp/Kvp4gkT01NEJhnNISZ6pVkR2rBnSGRWqZ6JTMawpSVtR8/mo49XB+lPDKXL1Uugptv5u+fWUR5ZC6H66MdHptMaRfYw2EoKIB77oFDh8A9mN9zT7A+HO79uc8999xeHffwww/z4osvdr2jiPS7I0egrg7K1+xgqLUeDXCo1VG+tpq6OmhsZ+DCpmOjW8JcmFfJ4IbgR2FwQy0X5lUS3RLu8NhkSqvA3vTrWV9//INsbAzWx/PruX379l4dp8AukrpWrQqCM5MmQW5u6425uVBYSF0drFzZybE7dsQ+tFBXB9XVHR6bTGkV2L/3va5/GRsb4bbbenf+vLw8ALZu3crMmTMpKytj7NixfPazn6WpF8zVq1czfvx4CgoKWLVqFdu3b+eRRx7hK1/5CoWFhdTU1PCjH/2IqVOnMnHiRBYuXEh9fT0AV155JV/84hc599xz+ehHP8qmTce6C//Od77DWWedxcSJE1m9ejUANTU1zJs3jylTpnD++eeze/fu3t2YyAAWDkNpKdRNL4HiYsjLA7NgXlxM3fQS5s+Hxx7r+NjD49r/UTg8trDDY5PK3ft9mjJlirf14osvHreuraFD3YPCl86n/PwuT9Wu3Nxcd3f/7W9/6/n5+f7aa695NBr1c845x5944gk/cOCAn3HGGX706FF3d3/nnXfc3X3JkiW+cePG5vO89dZbzZ9vvPFGv/POO5v3Kysr82g06n/84x99zJgx7u7+6KOP+sc//nGvq6tzd/cDBw64u/sFF1zge/bscXf3p59+2mfNmtXje+rOcxXJZE1xY/589/pDEfctW9zXrnXfssXrD0V8/vyO40bTsRdfFPHIzNnueXnuZu55eR6ZOdsvvigSV8zpKaDKuxFjU6pWTFdqaxO7X2fOPvtsRo4cCUBhYSF79+7lnHPOITs7m2uuuYb58+dTWlra7rEvvPACN910E++++y61tbXMnTu3edsnPvEJBg0axPjx4/nLX4JxIB5//HGuuuoqcnJyADj55JOpra1l+/btXHbZZc3Hvv/++/HfmMgAs3hx8A5u2DCIeIjIvFKOXFDKCSdApCFYn5UFV1zR8bH5J4WoX19BzrYw0eeqCU0upH5GCfnXhzo8NpnSqigmVlKSsP06c+KJJzZ/DoVCRCIRBg8ezDPPPMPChQt5+OGHmTdvXrvHXnnlldx11108//zz3Hzzza3qkrc8r8eKd9z9uKqJR48eZdiwYVRXVzdPL730Uvw3JjLArFwZBO6rr4acHNi1Cy69NJjn5MDSpcH2FSu6OHZoiJ2nljJ/+03sPLWUnKGhTo9NprQK7IsXBw+xM33561lbW8vBgwe56KKLuP3226murgZg6NChHDp0qHm/Q4cO8aEPfYjGxkbWr1/f5XnnzJnDfffd11wW//bbb5Ofn8/o0aPZuHEjEAT/nTt39sFdiWS2MWNg06bgL/kbboCiInj8cZg6FVavDt6BbtoU7JfIY5MprQJ7069nZ/ry1/PQoUOUlpZSUFDAjBkzuC32lvbTn/40t9xyC5MmTaKmpoa1a9dSXFzM3/7t3zJ27Nguzztv3jwuueQSioqKKCws5Lvf/S4A69ev595772XixImceeaZ/PKXv+ybGxPJcCUlMH48HD4MQ4cG7V/y8qChIVjfWQOjeI5NlqSMeVpUVORtB9p46aWXGDduXJfHNrUCa2xsXUMmKyuYUrEVWDJ197mKpIOamqB23COPwPe/D1/4AlxySZDpS7Vc83Gi0SCA7dgRVL0sKYFQqEenMLNn3b2oq/3S6uUpBM9i166gSuMDDwR/IuXlBcUvK1akwT+uiPRKOjbtbxaNwty5UFkZlN/k5gZVLysqehzcuyPtcuzSM3qukglqaoKW5Yfro1Qwl5lDKhl8uI5Idi5bG4qZSwXZOSF27UrRzF15OSxa1LrKXl4ebNgQVJTvpu7m2BNSxm5mw8xsk5ntNrOXzOzjiTiviAikb9P+Zp20XO0LiXp5egfwmLuPBSYCqpcnIgmTrk37m3XSnUFfiDuwm1k+MB24F8Ddj7j7u/GeV0SkSdo27W9S0n53Bn31UiARL08/CuwHfmxmE4FngS+5e6ufVTO7FrgW4LTTTkvAZUVkoMjLg61b4VM5JWwuKiZUdewlZLSomE/9uIRt24IuvFNSKBS8KA2Hg+KXwsJe1YrprkQUxQwGJgP/4u6TgDpgddud3P1udy9y96IRI0Yk4LKJ9e677/LDH/6wz6+zdevWXvciKTJQNTVOzD8pRP3mCqIPbuDI19YQfXAD9ZsryD8pNZv2txIKBX923HRTMO+joA6JCez7gH3uXhlb3kQQ6NNKTwO7u3P06NEeX0eBXaTn0rVpf7LEHdjd/c/Aa2b2N7FVs4G065x89erV1NTUUFhYyIoVK5g9ezaTJ0/mrLPOam7xuXfvXsaNG8eyZcuYPHkyr732Gvfeey9nnHEGM2fO5HOf+xzXX389APv372fhwoVMnTqVqVOn8vvf/569e/eybt06brvtNgoLC3niiSeSecsiaSNdm/YnTXe6gOxqAgqBKmAX8DBwUmf797bb3r70yiuv+Jlnnunu7o2NjX7w4EF3d9+/f7+PGTPGjx496q+88oqbmT/11FPu7v7666/76aef7gcOHPAjR474eeed58uXL3d390WLFvkTTzzh7u6vvvqqjx071t3db775Zr/lllv67b6S/VxFEunll92XLw+6yR00KJgvXx6sHwjoz2573b0a6LLSfLpwd7761a/yu9/9jkGDBvH66683d7F7+umnc8455wDwzDPPMGPGDE4++WQALrvsMvbs2QMEXfG2HFXpvffea9VRmIj03JgxcNddwSQdS7suBfrD+vXr2b9/P88++yxZWVmMGjWquevd3BZVrbyTVrtHjx7lqaeeYsiQIX2eXhGRltKqd8e+1LLr3YMHD/LBD36QrKwsfvvb3/Lqq6+2e8zZZ5/Ntm3beOedd4hEIjz00EPN2+bMmcNdLbIVHXXxKyKSaArsMcOHD2fatGlMmDCB6upqqqqqKCoqYv369R12vfuRj3yEr371qxQXF3PhhRcyfvx4PvCBDwBw5513UlVVRUFBAePHj2fdunUAXHzxxWzevFkvT0Wkz6gTsDjV1taSl5dHJBJhwYIFLF26lAULFiQ7Wc3S9bmKyPH6tROwgewb3/gGhYWFTJgwgdGjR/OJT3wi2UkSkQFOL0/j1DTakYhIqlCOXUQkwyiwi4hkGAV2EelXNTWw/LooS4aX8+KitSwZXs7y66LU1CQ7ZZlDZewi0m/CYfjUwii/PDyXc0OVZP+8jnWhXJ66u5jCn1Twi4dCqTtuaRpRjr0P5eXlAfDGG29QVlbW6b6333479fX1zcsXXXQR776r8Uokc9TUBINRz2gIM9UryY4Ew9sNidQy1SuZ0RCmrAzl3BMgfQN7NBoMELt2bTCPRvvpsj2/zoc//GE2bdrU6T5tA/ujjz7KsGHDenwtkVTVNG5p+ZodDLXWw9sNtTrK11an9rilaSQ9A3s0CnPnBqN+33xzMJ87N+7gvnfvXsaOHcuSJUsoKCigrKyM+vp6Ro0axZo1azjvvPPYuHEjNTU1zJs3jylTpnD++eeze/duAF555RU+/vGPM3XqVL72ta+1Ou+ECRNiSY+yatUqzjrrLAoKCvj+97/PnXfeyRtvvMGsWbOYNWsWAKNGjeKtt94C4NZbb2XChAlMmDCB22+/vfmc48aN43Of+xxnnnkmc+bMoaGhIa77F+lLzeOWdjL+Z0qPW5pOutMFZKKnuLvt3bLFPS/PHY5NeXnB+ji88sorDviTTz7p7u5XXXWV33LLLX766af7t7/97eb9LrjgAt+zZ4+7uz/99NM+a9Ysd3e/+OKL/f7773d397vuustzc3Obz9vUJfAPf/hD/+QnP+mNjY3u7n7gwAF3dz/99NN9//79zddoWq6qqvIJEyZ4bW2tHzp0yMePH+/PPfecv/LKKx4KhXzHjh3u7n7ZZZf5Aw88cNw9qdteSRVm7jNnutcejLjPnh38nzUL5rNne+3BiM+YEXTHK+2jm932pmeOvZORyuN16qmnMm3aNAAWL17Mk08+CcDll18OBF0IbN++ncsuu4zCwkL+7u/+jjfffBOA3//+9yxatAiAKzoYo+vxxx/nuuuuY/Dg4L11U5e/HXnyySdZsGABubm55OXl8clPfrK5j5nRo0dTGBvlfMqUKezduzeOOxfpW03jll7+mRAND1fAhg2wZg1s2EDDwxVc/pkQ27YF+0l80rNWTNOfcrW1x9bF/pSLl5m1u9zUXe/Ro0cZNmxYc2+NXR3flrt3uU/b/Tty4oknNn8OhUIqipGUtngx3HMPDBsGEQ8RmVfKkQtKOeEEiDQE61N+3NI0kZ459pISKC4OftrNgnlxMYmoJ/WnP/2Jp556CoANGzZw3nnntdqen5/P6NGj2bhxIxAE3p07dwIwbdo0fvaznwFBn+7tmTNnDuvWrSMSiQDw9ttvAx135zt9+nQefvhh6uvrqaurY/PmzZx//vlx36dIf2s1bmkO7NoFl14azHNy0LilCZSegT0UgorWf8pRUZGQUb/HjRvH/fffT0FBAW+//Taf//znj9tn/fr13HvvvUycOJEzzzyzeUzUO+64gx/84AdMnTqVgwcPtnv+a665htNOO42CggImTpzIT3/6UwCuvfZaSkpKml+eNpk8eTJXXnklZ599NsXFxVxzzTVMmjQp7vsU6W8at7T/qNveFvbu3UtpaSkvvPBCUtORSKnwXEVaqqmB226DBx4IgnxeXlD8smKFgnpXutttb3qWsYtI2tK4pX0vPYti+sioUaMyKrcuIgNTSgX2ZBQLZTI9T5GBKWUCe3Z2NgcOHFAwShB358CBA2RnZyc7KSLSz1KmjH3kyJHs27eP/fv3JzspGSM7O5uRI0cmOxki0s9SJrBnZWUxevToZCdDRCTtpUxRjIiIJIYCu4hIhlFgFxHJMArsIiIZRoFdRCTDJCywm1nIzHaYWXmizikiIj2XyBz7l4CXEng+ERHphYQEdjMbCcwH7knE+UQkddXUwPLroiwZXs6Li9ayZHg5y6+LUlOT7JRJk0Q1ULod+H/A0I52MLNrgWsBTjvttARdVkT6UzgMn1oY5ZeH53JuqJLsn9exLpTLU3cXU/iTCn7xUCgR491InOLOsZtZKfC/7v5sZ/u5+93uXuTuRSNGjIj3siLSz2pqoKwMZjSEmeqVZEdqwZ0hkVqmeiUzGsKUlaGcewpIRFHMNOASM9sL/Ay4wMweTMB5RSSFHDkSjHJUvmYHQ631YPJDrY7ytdXU1UFjY5ISKM3iDuzu/g/uPtLdRwGfBn7j7ovjTpmIpJRVq4LA3jyYfEuxweTr6oKxTSW5VI9dRLolHIbSUqib3v5g8nXTS5g/Hx57LNkplYT27ujuW4GtiTyniKSGvDzYuhUu/0yIjQ9XMGRrGKqrobCQhpklXP7pENu2QX5+slMqyrFLQqkqXOZavBiysmDYMIh4iMi8Uuq/fBOReaVEPMSwYcH2K65IdkpFgV0SJhyGwrOiLLx7Lv/63iLG//xm1r23iIV3z6XwrCjhcOfH19TAsmUwciRs3hzMly1TLYtUsXJlELivvhpycmDXLrj00mCekwNLlwbbV6xIdkpFgV0SIt6qcOEwFBTAPffArFmwYAHMnBksFxTQ5Y+C9L0xY2DTJqithRtugKIiePxxmDoVVq8OXqxu2hTsJ8mlwC4JEU9VuKYfhfp6iDZG+cq4cli7lq+MKyfaGKW+HtWPThElJTB+PBw+DEOHwqBBQdl7Q0OwXo2TUkPKDI0n6W3VKvjFLyC3qSpcbe2xjW2qwv3Hf7Q+tulHgWgU5s7F/7kS6usoyMklOrsYKiogFGL37n69JenAmDFw113BJKlJOXZJiHiqwjXXjw6HobISqwuKcayuFiorIRxW/WiRHlCOXRIinqpwTT8KFeft4IS61sU41NVx5Jlq5n+3lCee6JdbEUl7yrFLQsRTFa7pR+Gbj03Cc1q3aPScXL4ZLmTbtmA/EemaAru00tt66PFUhWv6UXj5YyVEi4rx3DzcDM/NI1pUzMsfK1H9aJEeMHfv94sWFRV5VVVVv19XOndcl6zROhpCuTwVLebS7K67ZA2HIRKBbdvg1lvBPag18eUvw/TpMHhw+7UmamqCKo3l5TB9WpRXfhjmie9Xc/4XChm9rIRtT4a4+OLgR0JV6WQgM7Nn3b2oq/2UYxcgMV2y9rYqXKv60V8NccaXS1n6PzfxNytLWX1jSPWjRXpIgV2AxHXJ2lQV7uDBoPbiwYPBcldBOd760Wq1KnKMArsAqdEla29/FNRqVaQ1BXYB0rdLVrVaFTme6rELkL5dsqrVqsjxlGMXIH27ZFWr1d7RO4nMpsAuQPp2ydpUhHSkckcswrfQ1Go1BYuQkqnpncR9P4qy4mPlLHhhLX//1+Xc96Oo3klkCNVjl2a9rYeeTPn5cOgQfKOonK+/tCjIqcd4bh5rxm3gG1Wl5OcHL2MHuqY2A4fro1Qwl5lDKhl8uI5Idi5bG4qZSwXZOSG1GUhRqscuPZaOXbKq1WrPNL2TiG4Jc2FeJYMbgqKrwQ21XJhXSXRLuFvVWiW1KbBLK72tcpgsTUVISz8Xwn5VQc0/buDfRq+h5h83YL+q4KprQilZhJQsze8kdrRfdEV1td5JZADVipG01rbV6q23luJeyqCV8OXXgyIktVo9pumdRHj5JLLb6Tf/8NhC5s9HPWmmOeXYJe2lYxFSsjRVa/3Uj4Oiq5btFaJFxXzqxyXqSTMDKMcuGUGj+nTP4sVBi9z8k0LUr68gZ1uY6HPVhCYXUj+jhPzrQ3onkQGUYxcZQFpVax0aYueppczffhM7Ty0lZ2goZau1Ss8osMuAN5Aa67R6J3EDFBXB44/D1KmwejXqSTNDKLDLgDYQOxDTO4nMpwZKMmA1Ndapr4dBRNnxT2EKojvYOWgSk28q4Sih5la4ysFKKlADpQFsIBUtxKOpsY5HokRnz+Wsf14EN99MwTcXEZ09F49E1VhH0pICe4YZiEULvaUOxCRTKbBnEPVN3jPqQEwyVdz12M3sVOAnwF8BR4G73f2OeM8rPae+yXumqbHON2sn8fWc3NYdiOXk8s1wIduqUq8PepGuJCLHHgFWuvs44BxguZmNT8B5pYdUtNAz6d6BmN6lSEfiDuzu/qa7Pxf7fAh4CfhIvOeVnlPRQs+kcwdi6lNdOpPQ6o5mNgr4HTDB3d9rs+1a4FqA0047bcqrr76asOtKQH2T91w69kGvPtUHrn6v7mhmecBDwN+3DeoA7n63uxe5e9GIESMSdVlpId2LFpIhEY11+rtIRH2qS5fcPe4JyAIqgC93Z/8pU6a4JN7LL7vn5Lj/5jfukfcj/t+3bfH7PrrW//u2LR55P+L/+Z/B9pdfTnZKM8ejjwbP9MTBEf/uzC3ua9b4LTO2+ImDI56TE2xPtIsucq+tdfc1a9zN3IM/NILJzH3tWq+tDfaTzAJUeTdibNxFMWZmwP3A2+7+9905Ri1P+046Fi2kq2QViQwaBDNmQHh5OdlXLWrdp3peHod/vIF5d5XyxBNBBSnJHP1ZFDMNuAK4wMyqY9NFCTiv9IL6Aek/iSgS6U0xjvpUl64kolbMk+5u7l7g7oWx6dFEJE56J92Gt0tX8Q4z19uaLU3vUvJPClG/uYLogxs48rU1RB/cQP3mCvJPUp/qA153ymsSPamMXTKBmfvMme4NG7e45+W1LuvOy/OGjVt8xgz3QYOOP7bpfcggIv5rZnvjkDx3M28ckue/ZrYPItLh+5BW71Ii7s8+637hhcE8EnG9S8lgdLOMXV0KiPRSPEUi8RTjqE916YoCu0gvxVMkEm8xjt6lSGcU2EV6KZ5h5ppaCR8eNwlyc1tvzM3l8NjCLlsJ612KdESBXaSX4ikSUc0W6UsK7CJx6G2RiGq2SF/S0HgiSdDUuKm8PGg4tnNnkOv/9rdh4sSggdnFF2tYPmlNQ+OJpDDVbJG+pMAukiSq2SJ9RUUxIiJpQkUxIiIDlAK7iEiGUWAXEckwCuwiIhlGgT1F1dTA8uuiLBlezouL1rJkeDnLr4tqBHoR6dLgZCdAjhcOw6cWRvnl4bmcG6ok++d1rAvl8tTdxRT+pIJfPBRSVTgR6ZBy7CmmpgbKymBGQ5ipXkl2JOjOdUiklqleyYyGMGVlfTdQsoikPwX2FNPUT3f5mh0MtdbduQ61OsrXVmsEehHplAJ7imnup3tS+925UljYaT/dIiIK7CmmqZ/uuuklUNy6O1eKi6mbXtJlP90iMrDp5WmKaeqn+/LPhNj4cAVDtoahuhoKC2mYWcLlnw6xbRvk5yc7pSKSqhTYU8zixXDPPTBsGEQ8RGReKUcuKOWEEyDSEKxXP90i0hkVxaSYVsOt5QT9cV96aTDPyaHltc/JAAALm0lEQVTT4dZERECBPeWon24RiZcCewpSP90iEg/1xy4ikibUH7uIyAClwC4ikmEU2EVEMowCu4hIhklIYDezeWb2X2b2spmtTsQ5RUSkd+IO7GYWAn4AlADjgUVmNj7e84qISO8kIsd+NvCyu/+Pux8BfgZcmoDziohILyQisH8EeK3F8r7YOhERSYJEBHZrZ91xrZ7M7FozqzKzqv379yfgsiIi0p5EBPZ9wKktlkcCb7Tdyd3vdvcidy8aMWJEAi4rIiLtSURg/wPwMTMbbWYnAJ8GHknAeUVEpBfi7o/d3SNmdj1QAYSA+9z9j3GnTEREeiUhA224+6PAo4k4l4iIxEctT0VEMowCu4hIhlFgFxHJMArsIiIZRoG9D9XUwLJlMHIkbN4czJctC9aLiPQVBfY+Eg5DQQHccw/MmgULFsDMmcFyQUGwXUSkLyiw94GaGigrg/p6aGyEpUuD9UuXBsv19cF25dxFpC8kpB67tHbkCNTVHVt+//1gPm0atBw7fPfu/k2XiAwMyrH3gVWrWgf2E09sPYdg+8qV/ZsuERkYFNj7QDgMpaWtg3tLdXUwfz489lj/pktEBgYF9j6Qlwdbt8Lll0NDbRTKy2HtWigvp6E2yuWXw7ZtwX4iIommMvY+sHhxUPvlpPwoWaVz8Wcrg2x6bi5ZU4o56cMVZGWFuOKKZKdURDKRcux9YOVKyMqCr0wIE3q2Equtxdyx2lpCVZWsOjNMVhasWJHslIpIJlJg7wNjxsCmTXDCH3fgta0L2r2ujhNfqmbTpmA/EZFEU2DvIyUl8IGZkziSldtq/ZGsXD4wo5CSkiQlTEQyngJ7H/rQ0hKypxcHb0nNIC+P7OnFfGiporqI9B29PO1LoRBUVAT1H6urobAwyMqHQslOmYhkMAX2vhYKBZXaS0uTnRIRGSBUFCMikmEU2EVEMowCu4hIhlFgFxHJMArsIiIZRoFdRCTDKLCLiGQYBXYRkQyjwC4ikmEU2EVEMowCu4hIhlFgFxHJMArsXaipgWXLYORI2Lw5mC9bFqwXEUlFcQV2M7vFzHab2S4z22xmwxKVsFQQDkNBQTB+6axZsGABzJwZLBcUBNtFRFJNvDn2XwMT3L0A2AP8Q/xJSg01NVBWBvX10NgIS5cG65cuDZbr64PtyrmLSKqJqz92d/9Vi8WngbL4kpM6jhyBuhbDlb7/fjCfNg3cj63fvbt/0yUi0pVElrEvBTosnDCza82sysyq9u/fn8DL9o1Vq1oH9hNPbD2HYPvKlf2bLhGRrnQZ2M3scTN7oZ3p0hb73AhEgPUdncfd73b3IncvGjFiRGJS34fC4WDQo5bBvaW6Opg/Hx57rH/TJSLSlS6LYtz9ws62m9kSoBSY7d6ykCK95eXB1q1w+eWwcSMMGXJsW0NDsH7bNsjPT1oSRUTaFW+tmHnADcAl7l6fmCSlhsWLISsLhg2DSCSY6uuPfR42LNh+xRXJTqmISGvxlrHfBQwFfm1m1Wa2LgFpSgkrVwaB++qrIScHdu2CSy8N5jk5Qe2YrCxYsSLZKRURac2SUXpSVFTkVVVV/X7dngqHg9z5tm1w661BbZhBg+DLX4bp02HwYCgpSXYqRWSgMLNn3b2oq/3U8rQTJSUwfjwcPgxDhwZBPS8vKGMfP15BXURSU1z12AeCMWPgrjui3DUvDDt2wKRJQUQPhZKdNBGRdimwdyUahblzobIyqOOYmwvFxVBRoeAuIilJRTFdCYeDoF5bGxSy19YGy+ooRkRSlAJ7V3bsOL6VUl0dVFcnJz0iIl1QYO/KpElB8UtLublQWJic9IiIdEGBvSslJUGZel4emAXz4mJViRGRlKWXp10JhYIXpeFwUPxSWKhaMSKS0jI+x56QEZBCoaBHsJtuCuYK6iKSwjI6sGsEJBEZiDI2sGsEJBEZqDK2jF0jIInIQJWxOXaNgCQiA1XGBnaNgCQiA1XGBvaWIyA1NLTe1nIEpLy8pCRPRKTPZGxg1whIIjJQpUVgb6qLnp8f9Imen991XXSNgCQiA1XKj6AUDgfVEhsbg6lJVlYwbdrUcev+phGQfvfbKLtvC1PIDnbaJMauKOH8mSGNgCQiaaW7IyildGCvqQkaEtV3Mkx2U258zJgOzrEnSuMFcxn5eiVDqKOBXPZ9pJis31Qw5gy1IBWR9JERQ+N973utc+ntaWyE227rePuYPWHGHqwkj1pCOHnUMvZgJWP2qNmpiGSmlA7sDz7YvcD+wAOd7KD+1EVkgEnpwF5bm4D91J+6iAwwKR3Yu1vHvNP91J+6iAwwKd1XzOLFQU+MnRXHdFkXXf2pi8gAk/G1YkREMkVG1IoZMyaop56TE+TMW8rKCtZv2qSgLiLSUkoHdghKTXbtgmuvbd3y9Nprg/UqKhcRaS2li2JEROSYjCiKERGRnlNgFxHJMArsIiIZJill7Ga2H3i13y+cHKcAbyU7ESlMz6drekadG0jP53R3H9HVTkkJ7AOJmVV152XHQKXn0zU9o87p+RxPRTEiIhlGgV1EJMMosPe9u5OdgBSn59M1PaPO6fm0oTJ2EZEMoxy7iEiGUWAXEckwCuy9ZGb3mdn/mtkLLdbdYma7zWyXmW02s2EdHLvXzJ43s2ozy8hOczp4Pmtjz6bazH5lZh/u4NglZvbfsWlJ/6W6f8X5jKKxfarN7JH+S3X/ae/5tNi2yszczE7p4NgB8R3qiMrYe8nMpgO1wE/cfUJs3RzgN+4eMbNvA7j7De0cuxcocveMbVTRwfPJd/f3Yp+/CIx39+vaHHcyUAUUAQ48C0xx93f6M/39obfPKLat1t27OcZYemrv+cTWnwrcA4wl+G681ea4AfMd6ohy7L3k7r8D3m6z7lfuHoktPg2M7PeEpYgOns97LRZzCf7TtTUX+LW7vx37j/hrYF6fJTSJ4nhGA0J7zyfmNuD/0fGzGTDfoY6k9NB4aW4p8PMOtjnwKzNz4F/dfcBU1zKzfwL+L3AQmNXOLh8BXmuxvC+2bsDoxjMCyI4V40WAb7n7w/2VvmQys0uA1919p5l1tNuA/w4px94HzOxGgv9w6zvYZZq7TwZKgOWxPzkHBHe/0d1PJXg217ezS3v/WwdUrrUbzwjgtFgz+s8At5tZxo8jZmY5wI3A17vatZ11A+o7pMCeYLEXNaXAZ72DFxju/kZs/r/AZuDs/kthyvgpsLCd9fuAU1ssjwTe6JcUpZ6OnlHL79D/AFuBSf2XrKQZA4wGdsbeU40EnjOzv2qz34D/DimwJ5CZzQNuAC5x93aH4DazXDMb2vQZmAMc99Y/E5nZx1osXgLsbme3CmCOmZ1kZicRPJ+K/khfKujOM4o9mxNjn08BpgEv9k8Kk8fdn3f3D7r7KHcfRRDAJ7v7n9vsOqC/QwC4u6ZeTMAG4E2gkeALdjXwMkHZXnVsWhfb98PAo7HPHwV2xqY/Ajcm+1768fk8RPAjtgvYAnwktm8RcE+LY5fGnuXLwFXJvpdUe0bAucDzse/Q88DVyb6X/no+bbbvBU4ZyN+hjiZVdxQRyTAqihERyTAK7CIiGUaBXUQkwyiwi4hkGAV2EZEMoy4FJKOZ2XDgP2OLfwVEgf2x5Xp3PzcpCRPpQ6ruKAOGmX0DqHX37yY7LSJ9SUUxMmCZWW1sPtPMtpnZL8xsj5l9y8w+a2bPxPrNHxPbb4SZPWRmf4hN05J7ByLtU2AXCUwEvgScBVwBnOHuZxP0+/2F2D53ALe5+1SCPlzuSUZCRbqiMnaRwB/c/U0AM6sBfhVb/zzHus69EBjforvYfDMb6u6H+jWlIl1QYBcJvN/i89EWy0c59v9kEPBxd2/oz4SJ9JSKYkS671e06B/dzAqTmBaRDimwi3TfF4Gi2GDTLwLHjUUqkgpU3VFEJMMoxy4ikmEU2EVEMowCu4hIhlFgFxHJMArsIiIZRoFdRCTDKLCLiGSY/w9hTPr7hGVa2gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Testing the model\", fontsize=14)\n",
    "plt.plot(t_instance[:-1], time_series(t_instance[:-1]), \"bo\", markersize=10, label=\"instance\")\n",
    "plt.plot(t_instance[1:], time_series(t_instance[1:]), \"w*\", markersize=10, label=\"target\")\n",
    "plt.plot(t_instance[1:], y_pred[0,:,0], \"r.\", markersize=10, label=\"prediction\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Time\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep RNNs\n",
    "\n",
    "We can also have a stack of cells at each time step instead of a single cell, called a deep RNN. Let's create a a deep RNN with 3 cells stacked on top of each other, each with 100 neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_neurons = 100\n",
    "n_layers = 3\n",
    "\n",
    "layers = [tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "         for layer in np.arange(n_layers)]\n",
    "multi_layer_cell = tf.nn.rnn_cell.MultiRNNCell(layers)\n",
    "outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very simple!\n",
    "\n",
    "### Applying Dropout\n",
    "\n",
    "Let's apply dropout between RNN layers to deal with overfitting. We will use a DropoutWrapper around our cells. It has an input_keep_prob parameter to which we can pass a placeholder which will take the value of 1 during testing, to prevent dropout. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "n_inputs = 1\n",
    "n_neurons = 100\n",
    "n_layers = 3\n",
    "n_steps = 20\n",
    "n_outputs = 1\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=(None, n_steps, n_inputs))\n",
    "y = tf.placeholder(tf.float32, shape=(None, n_steps, n_outputs))\n",
    "\n",
    "# Use default value of 1 to prevent dropout during testing\n",
    "keep_prob = tf.placeholder_with_default(1.0, shape=())\n",
    "cells = [tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "        for layer in np.arange(n_layers)]\n",
    "cells_drop = [tf.nn.rnn_cell.DropoutWrapper(cell, input_keep_prob=keep_prob)\n",
    "             for cell in cells]\n",
    "multi_layer_cell = tf.nn.rnn_cell.MultiRNNCell(cells_drop)\n",
    "rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as before\n",
    "learning_rate = 0.01\n",
    "\n",
    "stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons])\n",
    "stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)\n",
    "outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])\n",
    "\n",
    "loss = tf.reduce_mean(tf.square(outputs - y))\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Training MSE: 17.587788\n",
      "250 Training MSE: 3.7940028\n",
      "500 Training MSE: 3.7322252\n",
      "750 Training MSE: 3.1860335\n",
      "1000 Training MSE: 4.476041\n",
      "1250 Training MSE: 5.648244\n"
     ]
    }
   ],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "n_iterations = 1500\n",
    "batch_size = 50\n",
    "# Change value during training\n",
    "train_keep_prob = 0.5\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for iteration in range(n_iterations):\n",
    "        X_batch, y_batch = next_batch(batch_size, n_steps)\n",
    "        _, mse = sess.run([training_op, loss],\n",
    "                          feed_dict={X: X_batch, y: y_batch,\n",
    "                                     keep_prob: train_keep_prob})\n",
    "        if iteration % 250 == 0:                   \n",
    "            print(iteration, \"Training MSE:\", mse) \n",
    "            \n",
    "    saver.save(sess, \"models/my_dropout_time_series_model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_dropout_time_series_model\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"models/my_dropout_time_series_model\")\n",
    "\n",
    "    X_new = time_series(np.array(t_instance[:-1].reshape(-1, n_steps, n_inputs)))\n",
    "    # Stick with default value preventing dropout\n",
    "    y_pred = sess.run(outputs, feed_dict={X: X_new})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Difficulty of Training Over Many Time Steps\n",
    "\n",
    "If we have a long sequence, RNNs may take a long time to train and are also prone to the vanishing/exploding gradient problem mentioned in earlier chapters. This seems to be particularly problematic when our sequence gets to be over 100 inputs long. Furthermore, earlier inputs also have a much smaller influence on our output than newer inputs as training continues, even if their contribution should be quite significant. Networks with long-term memory cells are made to deal with this problem.\n",
    "\n",
    "### LSTM Cell\n",
    "\n",
    "A Long Short-Term Memory cell has a long-term memory which can store important information for later use, or discard unimportant long-term information. It does this using three inputs: the usual input X<sub>(t)</sub>, the previous \"short-term state\" h<sub>(t-1)</sub>, and the previous \"long-term state\" c<sub>(t-1)</sub>, and three outputs: the normal y<sub>(t)</sub> and state h<sub>(t)</sub>, along with c<sub>(t)</sub>. \n",
    "\n",
    "The input and short term state are given as input to four fully connected layers, three of which are called *gate controllers*. Let's look at these layers:\n",
    " * main layer g<sub>(t)</sub> - analyzes current input x<sub>(t)</sub> and short term memory h<sub>(t-1)</sub> as in a basic cell, but its ouput is further controlled by the input gate \n",
    " * forget gate f<sub>(t)</sub> - controls what information store from long term memory c<sub>(t-1)</sub>\n",
    " * input gate i<sub>(t)</sub> - controls what to store in long term memory c<sub>(t)</sub> from main layer \n",
    " * output gate o<sub>(t)</sub> - controls what long term memory (which is a function of c<sub>(t)</sub>) to output to standard output y<sub>(t)</sub> and short-term state h<sub>(t)</sub>\n",
    "\n",
    "These gates control the flow of information by applying the logistic activation function to a weighted sum of their inputs (current input and short-term state) plus a bias term, which outputs values from 0 to 1. These outputs are then used in element-wise multiplication with other elements of the cell, so values close to 0 will decrease the influence of these elements and values of 1 will increase their influence.\n",
    "\n",
    "We can see that, in a sense, all the outputs are calculated by using long-term states, because these also include information about the short-term state and current inputs after these inputs flow through the cell. Looking at the equations for the outputs will make this more clear, remembering that all gates take in x<sub>(t)</sub> and h<sub>(t-1)</sub> as input. All multiplication is done element-wise.\n",
    "\n",
    " * c<sub>(t)</sub> = f<sub>(t)</sub> * c<sub>(t-1)</sub> + i<sub>(t)</sub> * g<sub>(t)</sub>\n",
    " * y<sub>(t)</sub> = h<sub>(t)</sub> = o<sub>(t)</sub> * tanh(c<sub>(t)</sub>)\n",
    " \n",
    "#### Peephole Connections\n",
    "\n",
    "We previously mentioned that the gates all take the current input and short-term state as inputs. However, we can add *peephole connections* to our cell if we'd like them to recieve more information, which include c<sub>(t-1)</sub> as input to the forget gate and input gate and c<sub>(t)</sub> as input to the output gate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LSTM cell with peephole connections\n",
    "lstm_cell = tf.nn.rnn_cell.LSTMCell(num_units=n_neurons, use_peepholes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gru Cell\n",
    "\n",
    "The Gated Recurrent Unit cell simplifies the LSTM cell but performs similarly well. In particular, the short term and long-term states are pooled into one state h<sub>(t)</sub>. The gate z<sub>(t)</sub> acts as both the input and forget gate, opening one when the closing the other (a 1 opens the input gate). The gate r<sub>(t)</sub> controls which parts of the previous state h<sub>(t-1)</sub> will be shown to the main layer (again, by performing element-wise multiplication with the output of the logistic activation function). There is no output gate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "gru_cell = tf.nn.rnn_cell.GRUCell(num_units=n_neurons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}