experiment_a2c.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This document was used to train and experiment with model from experiment_a2c.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Things that are different in this model \n",
    "- The model will share the same feature extraction layer with different options for head. \n",
    "- The model will carry over all the functionalities from the A2C agent. \n",
    "\n",
    "\n",
    "## Feature to stack previous states and action for input - memory \n",
    "- Change the input of the model by stacking previous plays \n",
    "\n",
    "## Training Schedule\n",
    "- First train the value network on the rewards to make sure that it is predicting the rewards correctly, and then use it to get the policy. \n",
    "\n",
    "## More detailed training schedule (500 episodes each -> 500 * 5 = 2500 episodes for overall model)\n",
    "1) Train the Backbone along with a simple critic head to have the backbone learn better representations. \n",
    "2) Freeze the Backbone and train the complex critic so that it can learn how to use the representations well. \n",
    "3) Train both the backbone and the critic together\n",
    "4) Freeze the backbone and the critic and train the actor (might want to use delayed training for critic)\n",
    "5) Train everything together!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "import matplotlib.pyplot as plt\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "import torch.distributions as distributions\n",
    "\n",
    "plt.style.use(\"tableau-colorblind10\")\n",
    "plt.rcParams.update({'font.size': 14})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pygame 2.5.2 (SDL 2.28.3, Python 3.11.5)\n",
      "Hello from the pygame community. https://www.pygame.org/contribute.html\n"
     ]
    }
   ],
   "source": [
    "# Import resources for the game env \n",
    "from engines.environment import HexChessEnv\n",
    "from hexchess.players import RandomPlayer, GreedyPlayer\n",
    "from engines.train_a2c import A2C, A2CLearning\n",
    "from utils import plot_step_rewards, plot_episode_rewards\n",
    "from engines.experiment_a2c import Critic, Actor, ResidualBackBone, SimpleCritic, AdvancedA2CLearning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assets/a2c figures/a2c\n",
      "mps\n"
     ]
    }
   ],
   "source": [
    "# Constants\n",
    "base_path = os.path.join(\"assets\", \"a2c\")\n",
    "figs_path = os.path.join(\"figures\", \"a2c\")\n",
    "print(base_path, figs_path)\n",
    "\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Backbone with Simple Critic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "Number of trainable parameters in the Critic model: 991297\n",
      "Number of trainable parameters in the BackBone model: 7474272\n",
      "Memory initialized with 80 episodes. Average steps per episode: 11.80, Average reward: 1.86\n",
      "Epoch 1: Average Steps per Episode: 10.28, Average Reward: 1.03, Average Loss: 1.6429\n",
      "Epoch 2: Average Steps per Episode: 11.04, Average Reward: 1.62, Average Loss: 2.1105\n",
      "Epoch 3: Average Steps per Episode: 11.32, Average Reward: 1.37, Average Loss: 1.6583\n",
      "Epoch 4: Average Steps per Episode: 11.63, Average Reward: 1.45, Average Loss: 1.6090\n",
      "Epoch 5: Average Steps per Episode: 11.61, Average Reward: 1.48, Average Loss: 1.7641\n",
      "Epoch 6: Average Steps per Episode: 11.72, Average Reward: 0.97, Average Loss: 2.0333\n",
      "Epoch 7: Average Steps per Episode: 11.67, Average Reward: 1.13, Average Loss: 1.4798\n",
      "Epoch 8: Average Steps per Episode: 11.78, Average Reward: 1.75, Average Loss: 1.6254\n",
      "Epoch 9: Average Steps per Episode: 11.73, Average Reward: 1.60, Average Loss: 3.2512\n",
      "Epoch 10: Average Steps per Episode: 11.63, Average Reward: 0.98, Average Loss: 1.4586\n",
      "Epoch 11: Average Steps per Episode: 11.52, Average Reward: 1.93, Average Loss: 2.2030\n",
      "Epoch 12: Average Steps per Episode: 11.57, Average Reward: 2.28, Average Loss: 4.2333\n",
      "Epoch 13: Average Steps per Episode: 11.56, Average Reward: 1.92, Average Loss: 3.3884\n",
      "Critic saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_simple_critic.pth\n",
      "Backbone saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone.pth\n",
      "Finished training iteration 1/1. Model saved.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "'''\n",
    "We are going to use the SimpleCritic Model to focus training on the backbone!\n",
    "'''\n",
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = None\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_simple_critic.pth\"\n",
    "backbone_load_path = None\n",
    "backbone_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone.pth\"\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = GreedyPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 13\n",
    "episodes_per_epoch = 60\n",
    "batch_size = 40\n",
    "learning_rate = 1e-3\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 80\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = SimpleCritic(input_dim=input_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = SimpleCritic(input_dim=input_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=True\n",
    "                                    )\n",
    "\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    \n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Freeze BackBone and Train Complex Critic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "BackBone loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone.pth\n",
      "Number of trainable parameters in the Critic model: 15927297\n",
      "Number of trainable parameters in the BackBone model: 0\n",
      "Memory initialized with 80 episodes. Average steps per episode: 11.55, Average reward: 1.34\n",
      "Epoch 1: Average Steps per Episode: 11.07, Average Reward: 1.37, Average Loss: 1.4281\n",
      "Epoch 2: Average Steps per Episode: 10.96, Average Reward: 2.35, Average Loss: 3.2006\n",
      "Epoch 3: Average Steps per Episode: 11.32, Average Reward: 2.18, Average Loss: 13.9313\n",
      "Epoch 4: Average Steps per Episode: 11.27, Average Reward: 2.23, Average Loss: 5.5907\n",
      "Epoch 5: Average Steps per Episode: 11.37, Average Reward: 1.48, Average Loss: 3.1552\n",
      "Epoch 6: Average Steps per Episode: 11.68, Average Reward: 1.58, Average Loss: 1.9016\n",
      "Epoch 7: Average Steps per Episode: 11.71, Average Reward: 1.83, Average Loss: 2.0207\n",
      "Epoch 8: Average Steps per Episode: 11.85, Average Reward: 1.52, Average Loss: 1.5330\n",
      "Epoch 9: Average Steps per Episode: 11.87, Average Reward: 1.58, Average Loss: 5.4301\n",
      "Epoch 10: Average Steps per Episode: 11.89, Average Reward: 1.50, Average Loss: 1.6615\n",
      "Epoch 11: Average Steps per Episode: 11.92, Average Reward: 1.93, Average Loss: 3.0270\n",
      "Epoch 12: Average Steps per Episode: 11.97, Average Reward: 1.65, Average Loss: 1.6769\n",
      "Epoch 13: Average Steps per Episode: 11.90, Average Reward: 0.82, Average Loss: 1.3451\n",
      "Critic saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complex_critic.pth\n",
      "Finished training iteration 1/1. Model saved.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "'''\n",
    "Freeze the Backbone and just train the Critic Head\n",
    "'''\n",
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = None\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complex_critic.pth\"\n",
    "backbone_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone.pth\"\n",
    "backbone_save_path = None\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = GreedyPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "hidden_dim = 256\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 13\n",
    "episodes_per_epoch = 60\n",
    "batch_size = 40\n",
    "learning_rate = 1e-3\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "simple_critic = False\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 80\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "\n",
    "    # Freeze backbone parameters\n",
    "    for param in backbone.parameters():\n",
    "        param.requires_grad = False\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=simple_critic\n",
    "                                    )\n",
    "\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train both Backbone and Complex Critic Head together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "Critic loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complex_critic.pth\n",
      "BackBone loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone.pth\n",
      "Number of trainable parameters in the Critic model: 15927297\n",
      "Number of trainable parameters in the BackBone model: 7474272\n",
      "Memory initialized with 80 episodes. Average steps per episode: 11.20, Average reward: 1.16\n",
      "Epoch 1: Average Steps per Episode: 10.75, Average Reward: 0.98, Average Loss: 1.7908\n",
      "Epoch 2: Average Steps per Episode: 10.20, Average Reward: 1.88, Average Loss: 4.3002\n",
      "Epoch 3: Average Steps per Episode: 10.87, Average Reward: 1.47, Average Loss: 1.9138\n",
      "Epoch 4: Average Steps per Episode: 11.04, Average Reward: 2.63, Average Loss: 6.0221\n",
      "Epoch 5: Average Steps per Episode: 11.25, Average Reward: 1.30, Average Loss: 1.5782\n",
      "Epoch 6: Average Steps per Episode: 11.43, Average Reward: 1.45, Average Loss: 1.9187\n",
      "Epoch 7: Average Steps per Episode: 11.37, Average Reward: 1.28, Average Loss: 1.4164\n",
      "Epoch 8: Average Steps per Episode: 11.43, Average Reward: 1.43, Average Loss: 1.7136\n",
      "Epoch 9: Average Steps per Episode: 11.40, Average Reward: 1.60, Average Loss: 5.9084\n",
      "Epoch 10: Average Steps per Episode: 11.39, Average Reward: 1.77, Average Loss: 3.1333\n",
      "Epoch 11: Average Steps per Episode: 11.33, Average Reward: 1.15, Average Loss: 1.4452\n",
      "Epoch 12: Average Steps per Episode: 11.46, Average Reward: 1.37, Average Loss: 1.5152\n",
      "Epoch 13: Average Steps per Episode: 11.43, Average Reward: 1.17, Average Loss: 1.4349\n",
      "Critic saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complex_critic_unfreeze.pth\n",
      "Backbone saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone_unfreeze.pth\n",
      "Finished training iteration 1/1. Model saved.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complex_critic.pth\"\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complex_critic_unfreeze.pth\"\n",
    "backbone_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone.pth\"\n",
    "backbone_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_backbone_unfreeze.pth\"\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = GreedyPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "hidden_dim = 256\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 13\n",
    "episodes_per_epoch = 60\n",
    "batch_size = 40\n",
    "learning_rate = 1e-3\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "simple_critic = False\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 80\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=simple_critic\n",
    "                                    )\n",
    "\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complete Training against Random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[autoreload of engines.ac failed: Traceback (most recent call last):\n",
      "  File \"/Users/tawab/miniconda3/envs/DL/lib/python3.11/site-packages/IPython/extensions/autoreload.py\", line 276, in check\n",
      "    superreload(m, reload, self.old_objects)\n",
      "  File \"/Users/tawab/miniconda3/envs/DL/lib/python3.11/site-packages/IPython/extensions/autoreload.py\", line 500, in superreload\n",
      "    update_generic(old_obj, new_obj)\n",
      "  File \"/Users/tawab/miniconda3/envs/DL/lib/python3.11/site-packages/IPython/extensions/autoreload.py\", line 397, in update_generic\n",
      "    update(a, b)\n",
      "  File \"/Users/tawab/miniconda3/envs/DL/lib/python3.11/site-packages/IPython/extensions/autoreload.py\", line 365, in update_class\n",
      "    update_instances(old, new)\n",
      "  File \"/Users/tawab/miniconda3/envs/DL/lib/python3.11/site-packages/IPython/extensions/autoreload.py\", line 319, in update_instances\n",
      "    refs = gc.get_referrers(old)\n",
      "           ^^^^^^^^^^^^^^^^^^^^^\n",
      "KeyboardInterrupt\n",
      "]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "Number of trainable parameters in the Critic model: 15927297\n",
      "Number of trainable parameters in the BackBone model: 7474272\n",
      "Number of trainable parameters in the Actor model: 18055257\n",
      "action_probs sum: tensor(-74728.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0061800648\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-80730.4609, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0033164863\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-88606.7344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0019246843\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-97439.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013657573\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-108787.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011657523\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112120.9922, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00048485238\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-115330.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00053534546\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-130624.2344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00018874081\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137834.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0002866272\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-144746.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0002482472\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-144395.6250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00029624044\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157916.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00011630535\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-173493.2656, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 8.491117e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-152496.9062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00018081727\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-141259.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00034392835\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-130800.5469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00039511285\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-138497.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00022316293\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134152.9844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00053735456\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-116290.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010784696\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137216.2031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00046667136\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-138790.3438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0005006045\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-135215.4844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00055122277\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-117085.8750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0009444739\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112105.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0018584437\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111440.2969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.001586691\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-95125.6094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0033546556\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-104167.7031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0017760328\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-105531.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.003179165\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-109543.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0032487372\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-103674.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0046360693\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-124681.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0020195325\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-141703.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.001097142\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-132318.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0038592408\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-131580.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0027904476\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-127432.8516, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.002711168\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-129628.7031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0008377106\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112418.3906, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0021594707\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111136.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0025678005\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-116662.4531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0015065433\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-128746.1328, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010593252\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-138299.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00088529\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137751.0156, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.005452022\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134619.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0052206186\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-117629.0859, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0057451027\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-122908.2578, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.006680937\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-124541.5000, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.005821096\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-123198.7344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0014220102\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-126013.3984, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0014103067\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-141166.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0008633429\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-156316.3594, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0004407266\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-155086.6562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00045164666\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-139461.1719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00036604828\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-132370.0938, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00039411985\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-127007.2500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00043304992\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112014.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006180586\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-95529.0781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.001671986\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-81404.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.004574554\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-81837.0547, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00661283\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-80968.1172, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0055177584\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-81870.9062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.006149469\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-95623.3203, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.004308412\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-95445.5859, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.004795408\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-113861.5234, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.004031186\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-119773.4609, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.002249035\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-127010.3359, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0029201815\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-125642.5156, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0021662484\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111370.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0014302091\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-114700.5469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00044811098\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-101857.3047, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00576193\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-109019.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006800075\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-122515.2969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00052752474\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-144893.0469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00033916108\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170210.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00016495801\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-189038.6406, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.09260895\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-205791.9844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.6206973e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-192846.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.431237e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-188785.6719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.271931e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-177574.5000, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.028772e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-152527.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.573354e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-155787.5312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 4.5974175e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133567.7031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.6792212e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133567.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.2542872e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-142316.8750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 7.2414514e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134075.3438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.7252245e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-159122.6094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.4449392e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-140586.4062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.306339e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-166027.1094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.817975e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-189989.8594, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.909248e-09\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-190404.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 6.994385e-08\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-213908.5625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.324694e-09\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-196431.5469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.8270066e-07\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-211329.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 7.817445e-08\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-209454.6719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.0298454e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-187472.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 2.5151265e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-162191.8281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 6.607447e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-160239.2188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.6599324e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-156406.8750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.8683513e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-74728.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0061800648\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-92423.6094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00091386866\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111209.5781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00019300127\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-117211.7969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0002500225\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-123369.5938, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00025998423\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-135250.3125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00014010002\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-136496.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00015961431\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-128878.0547, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0002525808\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-117245.6328, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0008832583\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-114577.5781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011283519\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111849.8516, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006244645\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-109722.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011151773\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111999.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011990587\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-121128.1719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00067556225\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-121504.9844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00074335764\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134066.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00042819817\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-145628.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010755912\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-149869.5000, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00087269687\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-161380.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 8.047793e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-74728.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0061800648\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-76721.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0047582965\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-91807.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013489209\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111587.9219, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00049174245\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-126469.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00030837514\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153623.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.262565e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170426.1719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.329081e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170324.2500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.424765e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170324.2812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.252562e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157269.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.012391e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-145262.4844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.3894266e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-118718.5781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00035187858\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112912.4531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0014826797\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-115506.5469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0065288604\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-124505.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.005342\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-128510.7344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0051607443\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-135713.5000, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.005532798\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-135842.5625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.005278689\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-136055., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0056813266\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137908.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0059337174\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-122719.5625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.007422872\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-115441.1953, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0070662606\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-118689.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.008473746\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-131033.9297, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.011372347\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-130281.0781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.012191635\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133552.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.011937882\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-139868.3594, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.011055174\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-156751.2812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.009920211\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153531.2031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.012646962\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-143635.5000, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.01220044\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-149233.6250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.004172958\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-150619.8750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0042176647\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157653.6094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0035271773\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137929.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0033710415\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137151.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00027888015\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153449.3750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00019377809\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-161913.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.000121598954\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153058.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00014514441\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137194.0781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00025734832\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157660.2188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.6124416e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-149680.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00010563465\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-131674., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0001923989\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-129355.4297, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00021684743\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133655.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00076028623\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-139440.3750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0004948627\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-119711.3828, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00068889296\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-118491.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00093286496\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-127189.7578, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0007260236\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-116337.1172, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0014577338\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-74728.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0061800648\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-79332.2812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0038733121\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-84526.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.002635248\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-91786.3438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0017703371\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-93779.0234, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0021625932\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-104917.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0012532104\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-116166.8047, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00048722117\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-120479.6719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00042258558\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-121551.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00044134958\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-128238.5625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00019217287\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-150313.1406, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00012353549\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-160251.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00026184862\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170507.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 4.858082e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-172586.5312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00011330306\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-172659., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.877781e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-160020.1875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00016477158\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-147202.2031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00026075984\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-132057.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0007783224\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-131630.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0009760477\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-148652.7031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00033896515\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-163763.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00020274361\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-186342.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.2405903e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-192205.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 2.0167725e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-203967.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.1059391e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-206315.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.0244692e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-202383.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.2872608e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-193231.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 2.0629475e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-182523.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 7.541091e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-181341.9375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 6.4194974e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-169414.8281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00050192885\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-173119.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00042899436\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-162334.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0005945609\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-175469.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00023968688\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-174234.9531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00023471618\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-182823., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00015386572\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-186553.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.000106823776\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-181377.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.000104211365\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-187840.9219, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00010826856\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-190194.2656, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.1964524e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-185269.4531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.0104376e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-164247.9375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 7.585188e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157670.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00010074019\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-139022.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00026203383\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-120553.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0007221304\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-111355.4609, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013380724\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-119573.2656, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006218314\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-116836.7969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006825392\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-124860.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0004808979\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-149346.1406, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00014790647\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-175616.4375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.015085e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-167172.4844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.466713e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-163938.1875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00027426408\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-185206.4062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0002325452\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-184382.9531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00019357832\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-169770.6719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00034897996\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157631.3750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0063271266\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-158241.5781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0022699768\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-74728.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0061800648\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-85724.5703, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0021705893\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-95801.0938, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011121291\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-107148.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0005965987\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-110357.6875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0005975092\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112350.4531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0008994953\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-136436.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00036471363\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134003.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00023402016\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-129120.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0003043846\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-141847.0781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00015973324\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-159933.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 6.0139573e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-158192.7812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.5192977e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-157945.3438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.010591e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-152716.1875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.8858498e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-166307.9375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 4.5416127e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-164166.7031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.1318282e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-151787.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.0070762e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153358.5938, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.103995e-06\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-148631.2188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.940665e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-152995.8750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 2.004492e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-145641.5938, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00023739801\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137702.9531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0004686191\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-144277.9375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.002435203\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-148388.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0026576729\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-150923.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.07080116\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-163871.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.06899355\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-159619.6250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.056828983\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-171021.5312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00012266605\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-174315.2500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 5.9472448e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-191176.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.1513204e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-187466.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 9.917128e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170386.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00015135056\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-179921.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00010979104\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-160637.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00022166516\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-148770.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00038784102\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-126709.6797, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00071709004\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-121020.6562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0008896794\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-136829.9062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 8.4919666e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133902.4531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00028143259\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-129319.1484, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0003707384\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-123339.1016, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00038827155\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-139065.3750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006120776\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-149180., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00039435393\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-146802.4219, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0007483934\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-150613.9531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0007124375\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170480.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00057281536\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-190188.5000, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00045295595\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-191635.5781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00036770146\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-180676.2656, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010016704\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-170353.8750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013299994\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-166768.3125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010407642\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-152081.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0041050524\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-156789.9844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0028805288\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-147114.9375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00031537848\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-151050.9531, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00028804212\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-151507.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00030465637\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-155965.9375, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00046136565\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-158215.3438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0016255886\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134995.4062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0024383077\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133975.0156, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011257139\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-141676.8438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0007346887\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153129., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0004281176\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-138693.2969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.000118701806\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-140008.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00015757697\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-154180.1250, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 3.5914174e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-154795.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010613622\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-134780.4219, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013449936\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-130243.3594, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0001786094\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-127117.0859, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0002376846\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-128512.9219, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00025345618\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-126007.7344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0008110138\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-135044.1719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00050218945\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-148231.7344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 7.664767e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-152734., device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00044023275\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-166889.7188, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00040106237\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-174598.3906, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 4.81642e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-180661.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 1.0946944e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-183172.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00018362158\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-192841.2812, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00012318404\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-195187.9844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 8.554877e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-195363.1875, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 8.766889e-05\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-176956.6719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00023646271\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-179882.1562, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00024354026\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-173050.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0015758128\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-153545.3438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0004124765\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133893.9688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0005859937\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-124586.9062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0009070929\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-123777.0703, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.00021387171\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-118921.7344, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.002399469\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-107750.3125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.006093983\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-107383.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.003749409\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-108129.3906, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.003977528\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-121589.9844, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0034808214\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-122045.3750, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0038165017\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112887.2031, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0046091867\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-118541.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0057273363\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-114355.8906, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0051469537\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-132159.6719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.005463911\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-132139.0156, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.004990143\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-133379.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.006650131\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-74728.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0061800648\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-79922.4062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0037866002\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-81230.3125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.003731439\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-83087.4062, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0033536176\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-94295.8438, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0021729437\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-109131.3281, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0016153837\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-123973.0469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.002263045\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-132984.6094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0017647283\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-136830.0781, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0017888742\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-149964.7500, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011784654\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-138912.8906, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011050437\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-124559.1094, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011022423\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-109969.1953, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013160685\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-109580.7969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0012923712\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-112126.1719, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0006554189\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-121201.4297, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.001147225\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-130135.8594, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013576238\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-137804.8125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0013974587\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-138327.0312, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0017794503\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-125770.8984, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0017313446\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-139545.0469, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0010397516\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-128121.2969, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0015172411\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-119120.0625, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0012519767\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-114733.7578, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0011461184\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-118022.3125, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.0012781397\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n",
      "action_probs sum: tensor(-141067.4688, device='mps:0', grad_fn=<SumBackward0>)\n",
      "Sum of legal_action_probs: 0.000748066\n",
      "<class 'torch.Tensor'> Shape:  torch.Size([91, 91])\n"
     ]
    }
   ],
   "source": [
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = None\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random.pth\"\n",
    "backbone_load_path = None\n",
    "backbone_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random.pth\"\n",
    "actor_load_path = None\n",
    "actor_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random.pth\"\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = RandomPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "hidden_dim = 256\n",
    "\n",
    "# Actor parameters\n",
    "input_dim = 256\n",
    "hidden_dim = 256\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 120\n",
    "episodes_per_epoch = 2\n",
    "batch_size = 4\n",
    "learning_rate = 1e-3\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "simple_critic = False\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 12\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "    \n",
    "    if actor_load_path is not None:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim, model_path=actor_load_path)\n",
    "    else:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "    actor.to(device)\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "    \n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in actor.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone,\n",
    "                                    actor=actor, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=simple_critic\n",
    "                                    )\n",
    "\n",
    "\n",
    "    \n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    actor_num_parameters = count_parameters(actor)\n",
    "    print(f\"Number of trainable parameters in the Actor model: {actor_num_parameters}\")\n",
    "\n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path, actor_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwoAAAGsCAYAAAB94lD7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAADGk0lEQVR4nOydd3gbVdb/v6NiWe417olLnN4bpJFGAoTey7L0LMv77m+XBAhkd2kbWAgEso3l3aUunUBYaiCVJJBenN5c4sS9N1my6vz+GN2RbMu2NBoVy+fzPHme2BqNrjXSnXvu+Z7v4Xie50EQBEEQBEEQBOGEItADIAiCIAiCIAgi+KBAgSAIgiAIgiCIblCgQBAEQRAEQRBENyhQIAiCIAiCIAiiGxQoEARBEARBEATRDQoUCIIgCIIgCILoBgUKBEEQBEEQBEF0gwIFgiAIgiAIgiC6QYECQRAEQRAEQRDdoECBIAiCIAiCIIhuDMhA4b///S8WLlyIxMREhIeHIycnB7fffjvKyso6Hdfa2oply5ZhyJAh0Gg0yM7OxmOPPQadThegkRMEQRAEQRCEf+B4nucDPQh/wfM8fv3rX+Pf//438vLycNlllyE6OhqVlZXYvn07PvzwQ8yaNQsA0N7ejlmzZuHw4cNYtGgRJk6ciIKCAmzcuBFTp07Fjh07EB4eHuC/iCAIgiAIgiB8gyrQA/Anf/vb3/Dvf/8b//M//4O//e1vUCqVnR63WCzi/1966SUcPnwYjz/+OF588UXx90888QRWrVqFNWvWYMWKFX4bO0EQBEEQBEH4kwGTUTAYDMjIyEB8fDzOnDkDlarnGInneWRmZqK1tRXV1dWIjIwUH2tvb0dqaioGDRqE4uJifwydIAiCIAiCIPzOgMkobNy4EU1NTbj33nthtVrx9ddf4+zZs4iLi8Oll16KoUOHiscWFhaisrISl112WacgAQAiIyMxc+ZMbNiwAWVlZcjKyvL3n0IQBEEQBEEQPmfABAoHDx4EACiVSowbNw5nz54VH1MoFFi6dClWr14NQAgUACA/P9/lufLz87FhwwYUFhb2GCgYjUYYjUbxZ5vNhsbGRiQmJoLjOFn+JoIgCIIgCILwBJ7n0dbWhvT0dCgUvfsaDZhAoba2FgDw6quvYtKkSdi3bx9GjhyJgoIC/OpXv8Irr7yCvLw8PPTQQ2hpaQEAxMbGujxXTEwMAIjHueKFF17As88+K/NfQRAEQRAEQRDeU1ZWhszMzF6PGTCBgs1mAwCEhYXhyy+/RHp6OgBg9uzZ+OyzzzB+/Hi88soreOihh2R5vRUrVmDZsmXizy0tLRg8eDDOnTuH6OhoWV7DXcxmM3788UfMmzcParXar69N9A1dn+CGrk/wQ9couKHrE9zQ9Ql+5L5GbW1tyMnJcWs9OmACBZYdmDJlihgkMMaMGYPc3FwUFRWhublZPLanjEFra2unc7pCo9FAo9F0+31CQoKYkfAXZrMZERERSExMpEkgCKHrE9zQ9Ql+6BoFN3R9ghu6PsGP3NeIncMdKfyAabg2fPhwAEBcXJzLx9nvDQaDWJvAahW60lcNA0EQBEEQBEH0dwZMRmHevHkAgFOnTnV7zGw2o6ioCJGRkUhOTkZqairS09Oxc+dOtLe3d7NH3blzJ3JycsjxiCAIgiAIgghZBkxGIS8vD4sWLUJRURHefPPNTo+9+OKLaG5uxvXXXw+VSgWO4/DAAw9Ap9Nh5cqVnY5duXIldDodlixZ4s/hEwRBEARBEIRfGTAZBQD45z//iRkzZmDJkiX48ssvMWLECBQUFGDr1q0YMmQIXn75ZfHY5cuX46uvvsKqVatQUFCASZMm4dChQ9i4cSOmTp2Khx9+OHB/CEEQBEEQBEH4mAGTUQCErMKBAwdwzz334ODBg/jb3/6GwsJC/O///i/27duH1NRU8djIyEhs374dDz/8ME6dOoVXXnkFp0+fxiOPPIItW7ZAq9UG8C8hCIIgCIIgCN8yoDIKAJCVlYV33nnHrWNjY2OxZs0arFmzxsejIgiCIAiCIIjgYkBlFAiCIAiCIAiCcA8KFAiCIAiCIAiC6AYFCgRBEARBEARBdIMCBYIgCIIgCIIgukGBAkEQBEEQBEEQ3aBAgSAIgiAIgiCIblCgQBAEQRAEQRBENyhQIAiCIAiCIAiiGxQoEARBEARBEATRDQoUCIIgCIIgCILoBgUKBEEQBEEQBEF0gwIFgiAIgiAIgiC6QYECQRAEQRAEQRDdoECBIAiCIAiCIIhuUKBAEARBEARBEEQ3KFAgCIIgCIIgCKIbFCgQBEEQBEEQBNENChQIgiAIgiAIgugGBQoEQRAEQRAEQXSDAgWCIAiCIAiCILpBgQJBEARBEARBEN2gQIEgCIIgCIIgiG5QoEAQBEEQBEEQRDcoUCAIgiAIgiAIohsUKBAEQRAEQRAE0Q0KFAiCIAiCIAiC6AYFCgRBEARBEARBdIMCBUJkzdZTmPnqBjTrTYEeCkEQBEEQBBFgKFAgRF7afBK7Suqx+Ux1oIdCEARBEARBBBgKFAgAQFWLAdWtHQCAkvq2AI+GIAiCIAiCCDQUKBAAgILyRvH/xfW6AI6EIAiCIAiCCAYoUCAAAAVlTeL/KVAgCIIgCIIgKFAgAACHyhwZhRIKFAiCIAiCIAY8FCgQAICCckdG4UJTO8xWWwBHQxAEQRAEQQQaChQINOmNONcgZBHUSgWsNh4XGtsDPCqCIAiCIAgikFCgQOCwPZuQnRiJ/ORoAFSnQBAEQRAEMdChQIEQC5knZSYgNykKAFBMFqkEQRAEQRADGlWgB0AEnkN2a9SJWfGo1xkBUEEzQRAEQRDEQIcyCoQjo5CVgLwkkh4RBEEQBEEQlFEY8OhNFpyuaQUATMxMAM8LvyfpEUEQBEEQxMCGAoUBztGKZth4HinR4UiL1SLPINQolNTrwPM8OI4L8AgJgiAIgiCIQEDSowFOgb0+YVJWAgAgOzEKHAfojBbU2esVCIIgCIIgiIEHBQoDHNaReWJmPAAgXK1ERmwEACpoJgiCIAiCGMgMqEAhOzsbHMe5/Dd37txuxxuNRvzpT39Cfn4+wsPDkZ6ejl/96leora31/+B9hHMhMyOPLFIJgiAIgiAGPAOuRiE2NhYPP/xwt99nZ2d3+tlms+Haa6/Fhg0bcPHFF+PGG29EYWEh3nzzTWzZsgV79uxBcnKyfwbtI8xWG45VNQMQrFEZecnR2F5US85HBEEQBEEQA5gBFyjExcXhmWee6fO4//znP9iwYQNuv/12fPjhh2JR7//93//hoYcewh//+Ef861//8vFofcvJqhaYLDbEatXISYwSf5+b6ChoJgiCIAiCIAYmA0p65AlvvPEGAOCFF17o5Pzz4IMPIjc3Fx9++CEMBkOghicLjvqEhE5/I0mPCIIgCIIgiAEXKBiNRrz77rv485//jH/84x/Yu3dvt2M6Ojqwd+9eDB8+HEOGDOn0GMdxWLhwIdrb23HgwAF/DdsnFJQL9QmskJmRywKFOsooEARBEARBDFQGnPSouroa9957b6ffTZ06FR9//DHy8vIAAMXFxbDZbMjPz3d5Dvb7wsJCzJ492+UxRqMRRqPDXrS1VWhqZjabYTabvf47PIG9XtfXPXShAQAwLj2202OD48IBAFWtBrS0GxARNuA+Jn6lp+tDBAd0fYIfukbBDV2f4IauT/Aj9zXy5DwDagV47733Yvbs2RgzZgyioqJw9uxZvPrqq3j//fexYMECHDt2DNHR0WhpaQEgFD67IiYmBgDE41zxwgsv4Nlnn+32+40bNyIiIkKGv8ZzNm3aJP7fxvM4eKEdANBWchTra0+Ij/E8jwgVoLcA7335PQZHKf0+1oGI8/Uhgg+6PsEPXaPghq5PcEPXJ/iR6xrp9Xq3jx1QgcLTTz/d6ecJEybgvffeAwC8//77eOONN7Bs2TJZXmvFihWdztXa2oqsrCwsWrRIDDT8hdlsxqZNm7Bw4UKo1WoAwNnaNnRs/h7haiUeuPFKqJSdVWjDTm3E4YpmZI6ajMVj0v063oGGq+tDBA90fYIfukbBDV2f4IauT/Aj9zViKhd3GFCBQk88+OCDeP/997Fz504sW7ZMzCT0lDFgb3BPGQcA0Gg00Gg03X6vVqsD9kV0fu1jVcLfMD4jDtrw7uMcOigGhyuacb7ZQBOHnwjkZ4PoG7o+wQ9do+CGrk9wQ9cn+JHrGnlyjgFXzOyKpKQkAEB7uyDFyc3NhUKhQGFhocvj2e97qmHoDzgKmRNcPi46H9WR8xFBEARBEMRAhAIFQHQ+Yk3XtFotpk2bhjNnzuD8+fOdjuV5Hps2bUJkZCSmTJni76HKRkG5YI3q3JHZGbGXQgM5HxEEQRAEQQxEBkygcPr0aZfFG6dPn8bjjz8OALjjjjvE3//qV78CINQa8Dwv/v5f//oXSkpK8Itf/AJardbHo/YNNhuPQ2WurVEZecnRAEDdmQmCIEIQnufxwId78NvP+rfNN0EQvmXA1Ch88sknePXVV3HJJZdgyJAhiIyMxNmzZ7F+/XqYzWasWLECl1xyiXj83XffjU8//RQff/wxzp07hzlz5qCoqAhffPEFcnJy8NxzzwXwr/GOjw6UoqHdiFitGmPS41wewzIK5xp0sNpsUCoGTExJEAQR8pQ36/HW7mIAwP9eMgzDU/xrskEQRP9gwAQK8+bNw6lTp1BQUICffvoJer0eSUlJWLx4Mf7nf/4HixYt6nS8QqHAV199hRdffBHvv/8+1qxZg4SEBNx///147rnnkJycHKC/xDuMZiv++O0RAMATC0cjXO3a+jQrPgIqBQeTxYbKFgOy4iP9OUyCIAjCh9S2dYj//+ZYOYanjArgaAiCCFYGTKAwZ84czJkzx6PnaDQaPP30091sVfszr/9ciPON7UiP1eK3c4f3eJxKqUB2YhSK6tpQXKejQIEgCCKEqNM5GoJ+c7wCj15KgQJBEN0hPckAosVgwnM/HAcAPHvluD47LlNBM0EQRGjinFHYWVKHxnZjL0cTBDFQoUBhALF66xk0tBsxMjUG91yU2+fxeclkkUoQBBGKOAcKVhuP709WBnA0BEEEKxQoDBAaOmz42/azAIAXrpnQrROzK/KSyPmIIAgiFGHSI6WCAwB8c6wikMMhCCJIoUBhgPBJiQkGsxUzc5NxzdhMt56Tm0TSI4IgiFCEZRSuHJ0BAPjhVCXMVlsgh0QQRBBCgcIA4FRNK7ZUWAAAL103ERzHufU86s5MEAQRmtTphEDhqjEZSI7SoMVgxk9FtQEeFUEQwQYFCgOAJ787BhuAa8ZmYEau+7auLKPQqDehWW/y0egIgiAIf1Nrlx6lxoRjsT2r8M1xkh8RBNEZChRCnJ3Ftfj6WAUUAFZeOdaj50Zp1BgUHQ6A5EcEQRChBJMeDYoOx9Vj7IHCsXLwPB/IYREEEWRQoBDicByHMWmxWJChwkgJnTdJfkQQBBF6MOlRcpQGi0amIUylQHG9DmdqWgM8MoIgggkKFEKcGbnJ2P/oQtw3TCPp+VTQTBAEEVq0Gy3Qm6wAhIxCdLgac4emACD5EUEQnaFAYQCgVCigVblXwNyV1GgtAKCujZrxEARBhAJMdhSuViLS3njz6rEO+RFBEASDAgWiVxIjwwAAjXoKFAiCIEIBJjsaFKURXfBYoLCzpB4NOprvCYIQoECB6JWESEGy1EiuR4QLDpU1YldJXaCHQRCEBzgXMjOGJERhbHocbDx1aSYIwgEFCkSvJETYMwrttMNEdMZsteHSv2/B3L9uxvlGqmEhiP4Cs0ZNjgrv9HvR/eg4yY8IghCgQIHoFcooED1RXNeGJr0JZqsNnx26EOjhEAThJqL0KLpLoDA2EwDww8kqmCxWv4+LCAznGnRYX2ZCh5muOdEdChSIXmEZhQbKKBBdOO1ko/jpofMBHAlBEJ7gkB51dsObNiQRKdHhaO0wY/mXBdRTYYDw5HfH8O/TJnxxlDJJRHcoUCB6JdEpo0A3DcKZU9Ut4v8PXGhEST312iCI/kBdD9IjhYLDX2+aDAD467YzeGnzSb+PjfA/1a1C4FjepA/wSIhghAIFolcS7K5HJotN9N0mCAA41aUx01qSHxFEv0DMKHQJFADg1snZePWGSQCAJ746jP/sKfHr2Aj/09phBgDUk3KAcAEFCkSvRIapoFYKHxOySCWcYRmFy0elAQDWkvyIIPoFLFBIjnbdiHPp/JF4dMFIAMD9H+3B9yeoCVso02a0AADqyRaXcAEFCkSvcBzn5HxEBc2EAM/zYo3C7xeNgVLBoaC8CYW1rX08kyCIQMOkR64yCoxV107EnVOzYbXxuOmtn7C3tN5fwyP8TItByCg00D2ecAEFCkSfMPkRFTQTjIpmA3RGC5QKDhdlJ+LS4akASH5EEMEOz/Oo7cH1yBmFgsPbd07HZSPToDdZceXr23Chsd1fwyT8CJMe1dE9nnABBQpEnySSRSrRBSY7GpocjTCVErdOGgKA3I8IIthp7TDDZLEBAJKjXEuPGGqlAp8/MBtj0+PQ0G7E5wW0ERBqGM1WmKzC54E6chOuoECB6JOECHugQLsNhB0mOxqREgMAuG58JtRKBY5VNndyQyIIIrhgsqMojQraMFWfx0dp1Jg3LMX+3A6fjo3wPyybAFBGgXANBQpEn4g1CpRRIOycqhGCgZH2QCE+QoNFI5j8iLIKBBGsOHoo9Cw76grLKpMrTujhHCjojBZqukZ0gwIFok9YjQK5HhGMU9VCRmFkaqz4u1vs8iOqUyCI4EV0POpDduRMoth4kzaLQg3nQAGgWkSiOxQoEH3CdpPoJkEwmLzIOVC4dlwmwlQKnKxuwfHK5gCNjCCI3nDH8agrjnsALSJDja6BAsnLiK5QoED0icMelW4SBNCkN6LGvis5fFCM+PtYbRguH5kOgORHBBGsOHoouB8oJEVRoBCqdA0UqJcC0RUKFIg+SSDXI8KJ03bZUUacFjFadafHbp00GIDgfsTzvN/HRhBE77Ad40GeSI8ooxCysB4KDAoUvCfU7n0UKBB9QhkFwplTdsejkSmx3R67emwmwtVKnK1twzGSHxFE0OFVMbPOGHKLoIFOd+kR3ee94YvDF5Dxh//id58fCPRQZIMCBaJPEqhGgXDitN3xiFmjOhMdrsbUwQkAQDapBBGE1NoXgskSahQsNh5tHRafjIsIDN2kR7QhKIkOsxW/WbsfN775E6paDfj6WHmghyQbfZsoEwOeRCfXI57nwXFcgEdEBBJXjkfOsEVFE0nVCCLoqHOjK3NXIsKUCFcr0WG2oqHd2E1ySPRfqJjZe87WtOLWd37G4fIm8XeNIbSxShkFok9YwzWjxQYDeSwPeByOR90zCgAQpxUCSwoUCCL4cEiP3K9R4DhO3DCiOoXQggUKEfZtY6pR8IwP9p3DpFXf43B5E5KiNHjvrukAhPfVarMFeHTyQIEC0SdRGhVUCiGLEEpRMuE5HWYrzjW0A+g5oxBvr2lpMtBnhSCCCZuNFzXonkiPACpoDlVYoJCqFZaDJD1yn/UnKvDL93ah3WTB3PwUHHliMW6bnC0+3qw39/zkfgQFCkSfcBzn5HxEk8hAprC2FTaeR6xWjZQepAtioEAZBYIIKpoNJlhtQjGyJw3XAOrOHKo4AgVhM7CujaRH7rKtsAYAcOOELGz+f/ORHhcBtVKB6HAhPRMq6yUKFAi3SIigtDPR2fGop1oVChQIIjhhsqNYrRphKqVHz02ijEJIIgYKEZRR8JSyJj0AYHpOMpQKx3I6PsTktxQoEG7BdpNIejSw6as+AXAECs2G0Ei7EkSoUCv2UPBMdgQ4S4/oHhBKsD4KovSILHDdhgUKWfERnX4fHxFahh4UKBBuIfZSCJEPPiGNvhyPAMooEESwUtcm7BZ74njEoGLm0IRlFFLs0iOLje/WhI1wTVmTUK+XFdc1UAiteyAFCoRbUI0CAQCn7dIjVz0UGKGWdiWIUIFlFDytTwA6N10jQgcWKMSGcYjSCNr6YJAfFZQ1igvxYMRqs6GixQAAyIqP7PRYqG2sUqBAuIWjO3NofPAJz7HabDhTSxkFguivSOnKzCDXo9CEBQpaFYfkIAkGS+rbcNHqDbjy9W0BHUdv1LR2wGrjoVRwSIvt/H0KtXsgBQqEWyRQ2nnAc76xHR1mK8JUCuQkRvZ4HJskWzpMsNlI60oQwQKzRpUSKCRFUaAQapgsVnTYeyNFqDgk2q9xoJuu7T5XD7PVhmOVzWgM0s9bWbNQn5Aeq+1UyAxQoEAMUMRi5hD54BOew2RHw5Kju02MzrCGazwvBAsEQQQHLKPgjfSIAoXQoa3DIv5fqwSS7BuCgc4oHK1oFv9/rLK5x+MCSU+FzICz9Cg0visUKBBu4ZAehcYHn/AcdwqZAUCjVkKrFqwXQ2VHhSBCATGjEAKuR2sPncdPRbWBHka/RpQdqZVQKTjRAjfQGYWjlU2O/zsFDcEEq5/IjOseKFBGgRiQJERQRmGg47BG7T1QAEJvoiSIUEDMKHjhetRusohylUBxobEdt779M65/YwdZeXqBWMgcrgbgkJcFupjZOThwDhqCCTGjENddhhtq9z8KFAi3YDUKoZJKIzznVI09UOjF8YhBvRQIIvhw9FHwXHoUGx4GpUKw0Ay0/KikXieOo4Y6CUuGSUOj7YFCMDhb1es6UGl3EwKCOKPQ3Jv0KLQ2Vgd8oLBq1SpwHAeO47Bnz55uj7e2tmLZsmUYMmQINBoNsrOz8dhjj0Gn0wVgtIGDffCDJe1M+Bee50Xp0Yhemq0xQm1HhSD6O1abTVzgSylmVig4UYIa6EChskUv/r+ori2AI+nftNo3cmLCBVvUZLGYOXDXl9UkhNvlq8ermmG12QI2np4Qeyi4CBRC7f7nVqCgUCigVCo9/qdSqXw9fq84fvw4nn76aURGunZwaW9vx5w5c7BmzRqMGDECS5cuxfDhw7F69WrMnz8fHR0DZyeD7TR0mK0wmCx9HE2EGnU6I5r0JnAcMHyQG4EC9VIgiKCiod0EptJh87mnBEtBc4XTjjMFCtJh0qOYIMoosAzCwuGp0KqV0JusKK4Lvo3ZcjGjEPrSI7dW8pdccgk4juv0u6amJhw9ehRKpRJZWVlISUlBTU0NysrKYLVaMW7cOMTHx/tk0HJgNptx9913Y8KECcjPz8cHH3zQ7ZiXXnoJhw8fxuOPP44XX3xR/P0TTzyBVatWYc2aNVixYoU/hx0wosNVUCo4WG08GvUmZIQFdxBIyMvJKkF2lJ0QBa0b194xUZJUjSCCAVafkBipgUopTUwQLAXNztKUwloKFKTCAgUmPRJdjwIYCB6pEGoSJmYloLqtA/vPN+BoZTOGuSF59RcWqw1VLcL3qbeMQrvJApPFijCV0q/jkxu3Zott27bhxx9/FP+9//77aGpqwh133IHi4mKUlJRg9+7dKCkpQXFxMW6//XY0Nzfj/fff9/X4JfP888/jxIkTePvtt6FUdr+IPM/jzTffRFRUFJ588slOjz355JOIiorCm2++6a/hBhyO48j5aADDCsrGZcS5dXyo7agQRH+HOdkMipaWTQCcil0D7IpT0ewkPaqnQEEqYkbB3pE5KQj6KLCMwrj0OIxLj7P/LrgKmitbDLDxPNRKhUsHsVitGmxvPRTugZK2FR599FGkpaXhgw8+wODBgzs9NnjwYHz44YdITU3FY489Jssg5ebQoUN4/vnn8fTTT2PUqFEujyksLERlZSVmzpzZTZoUGRmJmTNnoqSkBGVlZf4YclCQECS7SYT/cZ683SGOpEcEEVQ4eih4Xp/ACJaMQoVTjQJlFKTTVXrE7FFbDGaYrf6vC7BYbThhd9cblxEnbkwdDbJeCqw+ISNOC4WC6/a4UqFAbHjo3AMl6Uc2b96MBx98sNdj5s+fjzfeeEPSoHyJ0WjEXXfdhQkTJmD58uU9HldYWAgAyM/Pd/l4fn4+NmzYgMLCQmRlZbl8HaPRsfPe2ioUgprNZpjN/nWCYa/n7evGa4XJpLZN7/e/IZSR6/r4EpYOHpUS7dY4YzRClq6h3RjUf5c79IfrM9Cha9Q3VfZd+KTIMMnvU5y96LWuzeDROeS+PpXNnWsUTCZTN3k00TfNdmloZJiwZxyl4qDgONh4HlVNOqTFav06ntM1regwWxERpkRWrAajUqIBAEfLm4Lqu33OnsXKjNX2OK54rRrNBhPq2vTIS+wuT/IUub9DnpxHUqDQ0dGBqqqqXo+prKyEwWDo9RhXFBUVYefOnSgvL0d9fT0iIiKQnJyMsWPHYsaMGdBqvfvgPvXUUygsLMTBgwddSo4YLS1CVBsb69ozPiYmptNxXXnhhRfw7LPPdvv9xo0bERHh/YdGCps2bfLq+RadcD137DkITflROYZEOOHt9fEVVp7H0XJhB6Wx8AjWVx7r8zkXKoVJqPBCJdavX+/T8fmLYL0+hAO6Rj2zu0hYFOobqiV/J+vLhN3Ro2dLsH59pcfPl+P68DyPcvuOLgC0GS34+KvvEBc24E0cPeZkkZBlqikrBbLDsGXLZkSrgBYz8N8fNiM72r/a+p+rhftGppbHhh9+QJtZqL4/19iOz7/+DhGq4AgGN5cK3wOFvrnH75LCLKyXNm7fhcaT8tV0yjXH6fX6vg+yI2n0kydPxieffIIlS5Zg+vTp3R7ftWsXPv30U1x88cVuna+srAxvvvkm3n33XZSXlwNAtyYqHMdBpVJh0aJFePDBB3HllVd6vIOwe/durF69Gs888wzGjBnj0XM9ZcWKFVi2bJn4c2trK7KysrBo0SIxyPAXZrMZmzZtwsKFC6FWqyWf5/OmvThQfx4ZecOxeMEIGUc4sJHr+viKM7WtMG3+AVq1EvfduBhKRd83ZNvxSvz1xM9QRcZg8eKFXo/hhU0ncaSiGe/cMc2tYmo5CfbrQ9A1cofv1h4AzpVgyuhhWHy5tPtf9Z4SvFd0AOFxyVi8eLbbz5Pz+tTpOmDZ/DUAIC0mHFWtHcgedzFm5CZ5dd6ByDvVO4GqCkweMwrQFWHhwoVIO7oFLTWtGDXpIszNH+TX8exZfww4dgqzR2Zj8eIpAIAVh79BebMB6WMuCpprvPmLAqCwENNGDcXixeNcHvP389tR3FaDvNHjsXjKEK9fU+45jqlc3EHSHff555/HggULMHv2bFx99dWYNWsWBg0ahNraWvz000/49ttvoVKp8Nxzz/V6nrq6Ojz99NN48803YbFYMHToUNx5552YMmUKUlJSkJCQAIPBgMbGRpw5cwZ79uzB1q1bsX79euTn5+Oll17CNddc49aYLRYL7r77bowbNw5PPPFEn8ezTEJPGQP2JveUcdBoNNBouheNqdXqgN3IvH3tpGghm9NitNDN2AcE8rPRG6dqBGu6selxCHfxmXZFcoz9s9Lh/WfFZuPxwqZT6DBbcc3YLNx9ca5X55NKsF4fwgFdo55p0Au7tWmxkZLfo5QYIRveaDBJOocc16euXZiPBkWHY1RaHKpaq1HaZMAcuu4e02YUrM7jIjWATrg+yVEanK4BmmSYuz3lRJWwrpqQlSC+9riMeJQ3G3Cytg1zhqf5dTw9UdEqZGKyE6N6fI9YTWeb0Srr+yjXHOfJOSQFCrNmzcL69evxq1/9Cl999RW++uorcBwnZgFycnLw73//GzNnzuz1PHl5eVAoFPjtb3+LO++8ExMmTOjztdvb2/H555/jzTffxPXXX4/Vq1dj6dKlfT5Pp9OJdQdhYWEuj2HZkf/+979ikTN7Tlf6qmEIRYKl2Q7hX1ghmbuOR4C8rkeVLQZ0mK0AgLd2FwUsUCCI/gzrypwsoSszIxiKmVkhc3qsFvnJ0dhyphqFde7vjhIOuhYzA45i90D0UmC1cM73mnHpcVh/ojKoOjQ7mq257sEFONZLjSFgES45h79gwQIUFRXh559/xpEjR9DS0oLY2FiMHz8es2bNcksW9Lvf/Q6PPPII4uLi3H7dyMhI3H333bj77ruxdevWHnf8u6LRaHD//fe7fGzHjh0oLCzENddcg+TkZGRnZyM/Px/p6enYuXMn2tvbOzkftbe3Y+fOncjJyXFZyByqsJtEI7keDSgcjkfu90VhDdeaDSbwPO9VoWFJg8PV5KfiOpytaQ0qT22C6A8w1yMpXZkZwdBwjfVQyIjVYmiyUOxaFIQNufoDzoECm2VFC1w/X+NmvQkXmoQgcKyTu964DOG+wyy6g4Ey+zgzXfRQYISSRbikQOG+++7D2LFjsXTpUsyePRuzZ7uvVXRm5cqVkp7HmD9/vtvHarXaHvse3HPPPSgsLMSKFSs61VU88MAD+NOf/oSVK1d2ari2cuVK6HQ6/P73v5c++H6II0Lu/x98wn087aEAOCZJq41HW4cFMVrpqdKuXTnf2VOMF66dKPl8BDEQqbPvEHsXKDg2ACxWm+TGbd7AeihkxEVgaHIUAKCQujNLwtFwTSUGCizjVNfm314Kx+yZ66z4CMRHOLJe7L5zrLIZNhvv0o7UnxjNVtTY35usuIERKEj6ln/00Ueora2VeyxBx/LlyzF+/HisWrUKl112GVasWIHLLrsMq1atwtSpU/Hwww8Heoh+hWnuQiGVRrhHi8GE0gYhzTrWzR4KAKANU0GjEqaXJoN3n5eSBiFQYB0w/7P3HCwB8PgmiP6KyWIVFyzeSI/YPYDnA7cAqrBbowrSIyGzWFTX1s0AhegbR8M1x0ZOEpMe+TmjwJqqde3VM2xQDMJUCrR1WHC+sd3FM/0Lk76Fq5Vi9sUVLNgZsIFCXl5en/aoUmhra0NJSUk3f9dPP/0Uv/jFL3D//ffj0KFDsr9uT0RGRmL79u14+OGHcerUKbzyyis4ffo0HnnkEWzZssVrq9b+hqMzc///4BPucdy+y5MZFyEuEtxFrqZrxfbdwgdn5iM5SoOqVgO+P+m5NSNBDFSY3lzBcUiIkB4oqJUKxNqzg4GSH1W2ODIKuUlR4DhhwRsITX1/xmK1QW8Sar+caxRY07U6P7+frBZufEZniataqcCo1Fj7MYGXH4myo7iIXiW1oaTAkBQo3Hffffjuu+9QUVEh62DYDr5zoPD666/jjjvuwMcff4x33nkHs2fPxunTp2V93XfffRc8z7u0c42NjcWaNWtw4cIFmEwmnD9/HqtXr0Z0dLSsY+gPJERSMfNA44i9PmG8B7Ijhlyp1+J6IaMwIiUGd00TCpnf2l3s1TkJYiDBFn1JURqvpRuBrlOoaHFkFMLVSlH+QfIjz2gzOtZZMeEOFTrLOPk78BJr4Vzca1iW4UgQFDSX26VvWb3UJwAkPcKNN96Iiy66CDNmzMBrr72Gffv24fz587hw4UK3f56wfft2XHrppZ0akr344ovIyMjAjh07sHbtWvA8j5dfflnKsAkvYTcIg9kKg8kS4NEQ/kBMB2e4X8jMkGuiZNKjvOQo3Dc9DwDw7fEKVLd63tCRIAYipY3Cd8gb2REj0M5HYo1CrLBOcBQ0U6DgCUx2FK5WIkzlaKzmKGb2X42CzcaLNQqu7jUsy8DuR4GEZRR6q08AQitQkFTMnJubK9qh/va3v+3xOI7jYLG4v6CsqqrC5ZdfLv586tQplJWV4aWXXsKsWbMAAJ9//jl27NghZdiEl8SEq6FUcLDaeDTpTX5vfBWs8DyPcw065CRGeeXuE4yI1qge1Ccw2ETZbJDecr7V4JAU5CZGI0arxsXZSdhTWo/39pZg+cLRks9NEAMBnuexatNJAMCcod430EoKYEbBZLGK2ZGMOEH6m58cja1nayij4CEthu7WqIAjUKjTGb12rHOXcw06tJss0KgUyE/urtZgWYZgsEhl1qiZfQQKTIERCjWdklZ6d911l08+PEajsVOPg+3bt4PjOCxatEj8XW5uLr7++mvZX5voG47jEB8RhnqdEY16E9L7+KIMFF7ZcgqPfVmAd+68GPdcnBfo4chGX7s8fSHHjkpxvXDzT4rSiM5J98/Iw57Sery1uxiPXToq5IIzgpCTr46WY/e5emjVSvzhMmkdmZ1hzkf+LnYFgCp7FjFMpRAzG5RRkIarHgqAo4+CyWKDzmhBdLjvm66x/gmj0+JcOmmx+09RfRvajRZEagK3SSlmFPqSHmmFz6fRYoPBZOnXG6uSRv7uu+/KPAyBzMxMHD16VPz522+/RUJCAsaNc7TIbmhoQFRUlE9en+ibBDFQ6P9RslzsO98AIDh2O+SktFEHndGCMJUCwwZ5XpMTLxYzS/+slNjrE/KSHN/5WycNwcOfH8TZ2jbsKqnDzDzvd0kJIhSxWG1Y8fVhAMDSeSNk2dwJZI1CpZPjEdsgYIFCYW3oBQptHWZUNOsxwl7MKyc9BQoRYSpo1UoYzFbU64x+CRQcvXriXD4+KDocKdHhqGnrwImqZkzLTvL5mHqiTKxR6LnZGiBYzoaKAsP/Jsi9cMUVV2Djxo149NFH8cc//hE//PADrr766k7HnD17FoMHDw7QCAnmmBHIzpzBBrNLCwUtojNs8h6dGivJL13OjEJekiNQiQ5X45ZJwhxARc0E0TPv7CnB6ZpWJEZqsHzhKFnOGchAoUJstuYIePLtmxhF9aFnkbrko70Y9fy32HOuXvZzt/YgPQKc5Uf+qVMQJa69mGaI8iP7sYHC3YwCU2AA/d/5KKgChRUrVmDw4MF49dVX8ec//xkpKSn405/+JD5eW1uLnTt34pJLLgngKAc2LO3cSM5HIuX2iaPJ0L8ng64c9UJ2BMgTKLCMQm5S5yzi/fai5rWHLqCtQ3oNBEGEKu1GC57+TsjQ//HyMYjVhvXxDPcIZDEzK2ROj3VYk+cmCnNDi8Hsk+DlaEUTTlW3yH5ed9hWWAOeB9afkNdhEug5owA45Ef+kpexIuWu1qjOjEsXHjtSHriCZoPJIn7G+goUAOesev9eG0jOhbS1teEf//gHNm/ejMrKShiN3T9QHMehuNj9Hb/U1FScOHECW7ZsAQBccskliImJER+vr6/Hyy+/jMsuu0zqsAkvcTRd698ffLmw2XhU2ne5Qi14ckzecZKeL/ZR8CKAKnYhPQKAGbnJGJ4SgzM1rVh/ogK3Ts6W/BoEEYr8ddtpVLUakJ0YiYdm5ct2XtEVx0+7zc6wuTbDSUKlDVMhKz4CZU16FNa2iQ3D5KDdaMGMVzZCrVSg+s83QKNW9v0kmWjSG8UOwLt8kVEQA4Xuy0BWsO4Pi1Sd0SzO87019QyGjAKzRo0MU4n3t94IFecjSYFCXV0dZsyYgeLiYsTExKC1tRWxsbEwmUwwGOwawvR0qNWea9u0Wi2uuuoql4+NGjUKo0bJkz4lpOFouhZai2Kp1Oo6YLEJ6e7+Phl05Yjoax24jEJxDxkFjuMwKTMeZ2paxU6tBEEI1Os6RKej564aL+sCN6AZhZbuGQVAqFMoa9KjqL4N03OTZXu9843taLdbgZ+uacX4TGlzoRROVbeK/99bWg+rzQalQj4RCAsUXGWanJ2PfM3xSiFbkxajRXJ0z0GewyK12W9uTF1xlh258/qhEihI+tQ988wzKC4uxnvvvYemJmHXcenSpWhvb8fevXsxbdo0ZGdn48SJE5IGZTKZsH79erz66qtYuXKl+PuOjg7U1tbCZrNJOi/hPZRR6AyTHQGhJT0SdnmE+gAp1qiA95Ok2WrDBbsVnXONAiNR/CxS0EoQzjy/4QRaO8yYkBmP22XOtiUGsPEm2xTI6BIo5PuooJlJnQD/dwU+XeMIFHRGi7iglovepUf+yxoxZ73esgmA0HBTpeDQbDCJO/v+xrkrszs4ujP373uUpEBh/fr1WLBgAe68885uUdXUqVPx/fffo7S0FM8++6zH5/76668xePBgXH311Xj00UfxzDPPiI8dPXoUaWlp+OSTT6QMm5AB9sGn7swCzhNWYwgVeJ+oagHPA6kx4b3u8vSGt4HChcZ2WG08wtVKpMVouz0e6MZPBBGMnKvX4bUdZwEAq66d6HUn5q44FzP7u3i40p5RyOiyUPOVRarz/H6kvFnWc/dF17qIXefqZD1/T30UAP9mFArrhIBoRGpMr8dp1ErxOjtnW/xJWbOwceVOfQIwwDMKVVVVmDhxovizUqkUJUcAEB8fjyuuuAJr16716Lw7d+7ETTfdBI1Gg7/+9a+44447Oj0+bdo0DB06FOvWrZMybEIGEimj0AnnHSeD2Qqj2RrA0ciHw65OeqrdueGalAUFK2TOSYx0udhJCODOJkEEK69sPQWz1YYFw1OxcESq7Odn9wCLjUdbh/sNVb2F53kxo9BVesQyCkX2OUMuygOYUWCBAtsk2VUib6DQW0ZBrFHww9zKskBDXWSNuzIiRQgmTtcEprjcIT3q3RqVMaADhdjYWJjNDqeR+Ph4lJeXdzomJiYGNTU1Hp135cqViIuLw8GDB/Gb3/wG+fndC7CmTJmCI0eOSBk2IQOOGoX+/cGXi64p0FCRH7FC5t7s6vqCTZJmqw16k+cBlCtrVGcCadNIEMGIwWTBh/tLAQDLfdSMkPnsA0B9u/8Kmts6LGK9QE8ZhcJaeS1SO0mP/Nwnh0mP7r4oBwCwW+aCZrdcj/yQUWBZoHw3evWMtPeTCFhGwU1rVAazk+/vG6uSAoXc3FyUlpaKP0+cOBGbNm1CQ4PQeMpgMOCbb77xuN/B3r17ce211yIpqedmGllZWaiurpYybEIGEkgX3omugUKoBFDu+Fr3RWSYCip7JkDKjkpPjkcMym4RRGf+e6QMzQYTsuIjsGB4is9eJxCyP1bIHKcNQ0SX5lXM7KDZYJJ1Dnae32vaOlDT6h/jhA6zFecaBJnLPRfngeOE+VDO1+81o+CnPgo2Gy9mgViw1xsj7fKkQNnVss9Dlps1CgM6o7Bo0SJs2bIFer3wpj344IOora3F+PHjcfPNN2PMmDEoLi7GPffc49F5jUZjJztUVzQ3N0MhY+U/4RlUo9AZ1gCI4U0X4mCB53lx96w3X+u+cG44I+V96amHAoMyCtKobjXg/63dj53FtYEeCuEBnxdcwOrNJ2Gz9bxjzhoQ3ntxnqwOOV0JxHfPVQ8FRkSYSiwwLZSxTqG8i6PaMT9Zc56tbYWN5xGnDcOwQdEYbd9JlzOrEAwZhcoWAzrMVqgUHLIT+pbzjEgR3gfnQm9/QtIjD/j1r3+NN954QwwUbrjhBrz88stob2/HunXrUF1djWXLluGxxx7z6Ly5ubnYv39/r8fs3r0bI0aMkDJsQgaYLlxvsqIjRPT43uDsegT0/wkBEHZNmg0mqBScqAmVijhRSpBkOTIKrneaKGiVxgsbT+AfO87ikr9sxnM/HIOVXOSCng6zFb98bxce+7IAaw+dd3lMSX0btp6tAccB916c69PxsB1nf373HD0UugcKgG8KmlkWY5hdFuMv+RHbMR+ZGgOO4zA9R7B89UmgoO05o9BkMMFi9d38wAqZsxOjoFL2vRxl96Oatg6/b8rpjGY02+9jbrsesQa1/XwDUVKgkJaWhltvvbWTROiRRx5BfX09qqqqoNPp8PLLL0Op9My7+cYbb8TOnTvxzjvvuHx89erVOH78OG699VYpwyZkIDY8DApOupwklOB5XkxFsl3vUKhRYJ0vR6bGIkzlnf96nMTOlDzPo6SB1Sj0nlGgoNV9rDYbPjt0AQBg43k8+e1RXPbaj6j2k6SCkMaec/XiZ/wP3x6BydL98/7unhIAwIJhqchOdP2dkYtAWKQ6MgquF2miRapMgUKH2SruqC8enQHAf82+2I45WxjPyBXWWnI6H4l9FFxkFNgmDM/7VtpZVCdsBuW7ITsCgOhwtbhI93edAssmxISrXQZXrgiVzsyy5iaVSiVSUlIkF1A99thjGDlyJB544AEsXLhQ7NC8fPlyzJ49G48//jgmTJiA3/zmN3IOm/AAhcIhJ+nvUbK3NOlNMNhv3mPS4gCERo3CGbsLxei0WK/PJTX1Wq8zoq3DAo5Dj4ueWK0aSnsNBDUAdI+dJXWoajUgVqvGG3dchIgwJbacqcb4F9Zj06mqQA+P6IHtRQ5jkJJ6Hf69s6jT41abDe/YA4X7p+f5fDyJfuzcy2Ayz649FBhDk4V5Qq6MAgtMwtVKzBk6CABwpMI/zkdsEcyKd2fYMwr7zze4DBI9xWqzQWcUCsNdSY9USoUYLPjyGrNr5U59AoPVKfhbfuRpITMAxNuLmZv0Jr9bCcuJpM7MCxYswPz58zF37lxcdNFFUKkknaYbUVFR+Omnn/Cb3/wGa9euhdUqfCFWr14NjuNwyy234J///Cc0Go0sr0dIIyEiDA3txpBYFHsDu5EkRmqQFiNoOvv7zgHg+Ls8mRB7QmqgwGRHGbERCO+hqyyrgajXGdHQbkS6m+nggcynBwXZyvXjsvDAjKGYlZuMW9/5GUcrmnHZP7diYmYCujrRzh+WilXXTXRxNsJf7CgS6kmmDUnEvvMN+NP3x3DXtFxxZ3PT6WqUN+sRHxGG68Zn+Xw8gahRcEiPesgoDBIWkLIFCi2O5lqsVutkdQvMVhvUbshkvOGU3f5zpD2jkD8oGomRGjS0G3G4vAnTsns2fHEHZ1tbIVDoLi9KitKgUW+yFzR7v2nkikIxUHA/AzYiJRabTlf7vaC5rMmzHgqA4/5nsfFoN1kQpXEvExFsSPq079y5E08++SQuueQSxMXFYdGiRXjhhRewZ88ecXEvlfj4eHz44Yeorq7G+vXr8cEHH+Drr79GZWUlPv74Y8TH+6+FOuEa8q8XYLKjzLgI0Q0qFKRH7AbZU4rfE5x7KXhCid0atadCZgY1XXMfq82Gzw+XAQBumSQ40o1IjcWeRy7Dr2flg+eBQ2WNOHCh87+XNp9EcwgEwP0Vk8UqatPfuOMiDBsUjTqdEa9sPSUew4qY75ya02NgLScBcT3qpZgZAIba5wq5pEflTl14hyREIjpcBZPFhrO1vt3JttpsOGvP6o6wZxSEOgUmP/K+ToHJjsJUCmh6+LwkRfk+ayRao0rIKPg/UGCOR+4VMgNARJgSYSphmd2fN1YlpQJaWlqwe/du/Pjjj9i6dSt27NiBzZs3g+M4REZGYubMmZg3bx7mzp2LadOmSRpYYmIiLr/8cknPJXxLqHgDewtzxMiM0zrkWCEQPFX2keL3BKkazb6sURmJIVIs5g+2F9aitq0D8RFhuHREmvh7bZgKr982Db+elS8GiYyb3/oJepMV9e1GxNk/44R/2X++AQazFclRGoxNj8Ofr56Am976Ca9sOYWHZudDyXH46qjQx8gfsiPA0ZArmDIKefbFZpPehMZ2o7h5IxW2EZQRp4VCwWFsehx2ldTjaEUzRtulpr7gfGM7OsxWaFQK5CQ6FqUzcpLw7fEK7Cqpw8PzvDN06c3xiCE6H/noGttsvDTpkd356JSfpUflEjLtHMchXhtmL742YbAbzk7BiKRAQaPRYO7cuZg7dy6effZZGAwG7Ny5E9u2bcOPP/6ILVu2YOPGjeA4DhaL/zo3Ev5BXJyFwKLYGyrEG0lEyNigARC7n/Z0Q/YEqfaofTkeMRIjKKPgLswt54bxWS6lE+Mz4zE+s3PGNilSgwsmvfBd9+BmTsjHdrvs6JKhg8BxHG6YkIWLshOxt1SQIOUnR8NstWFSVkK36+cr2D3AH517AWGXvaq19w2MiDAV0mO1qGwxoLhe53WgwGoiWPHsuPR4e6DQhNunZHt17t5g9QnDBsV0srhlzkdyFDS7EyiwYNBXvRSqWg0wmK1QKjiPiu9ZRuFcgw4dZqtfMmiAI1DtKaPVE/ERjkChvyJLcYFWq0VmZiYyMjKQlpaGqKgoNDc391m8MX/+fEmvx3GcWOhM+J8EanQFoIv0yAsb0GCC53lUtvSe4vcEqQGUu9KjBOql4BYWqw3rjgiyo1snDXH7eYmRGlxo0tP7G0C2FwqBwpyhQgM1juOw6tqJmPvXzfj3ziJx4eyvbALg/xqF2jYjrDYeCo7DoOjwHo/LSYxCZYsB5xp0mDok0avXdJYeAcB4e/NJXzsfsSJdtiBmTB2SCKWCQ0WzAWVN7W57+bvCrUBBbLrmm2tcaJdXZSdEelTzMSg6HHHaMDQbTDhb24pxXvT68QQWMPX2+XNFKFikSg4UiouLRenRtm3bUFNTA57nkZeXh5tuugnz5s3DvHnzej3Htm3bXP6e4ziXQQb7vS/a0hPukxBCMhtvcA4UHNKj/h0oNLabYLQIhW1pMTIGCh4GUCQ9kpcfz9agXmdEUpQG84a537E3gWpAAorZasPOEmEHeU7+IPH3c/JTcOXodHx3ohIXmvQIVytxhw93ubvi70CBbV6kxoT36refkxiJnSV1ONeg8/o1y50yxgDEBamveykw7T1rLsaI1KgwISMeB8sasaukDrdO9m2gkGPf5T9c7hunpyL7ZlD+IM8ylRzHYWRqDHafq8ep6hY/BgrCZ51JstwlFCxSJQUKgwcPRkVFBQAgKysLl112mRgYZGW577hg69Lox2g04uabb0ZhYSH++Mc/Yvbs2UhJSUFNTQ127NiB559/HsOGDcPatWulDJuQCVaj4K+0c7ASitIjplFPitL0WOTmCVL6KBhMFjHN26f0iBaybvGpXXZ04/gstxobMQLhl084OFTWiHaTBQkRYaIFM+PFaydi/clK8Dxw44Qsv9aQsN1m1sPE1/IPJofsK8vJFrfn6r0PFJxdjwCHBXZ5s16WGoie6Op45Mz0nCQhUDhXj1snZ0t+jd56KDAWDE8FIDR50xnNsjv2iPUJfczxrhiZGovd5+r9apHKMgrJUZ5d91BYG0hyPSovLwfP81iwYAFeeuklvPzyy7jrrrs8ChJc8fTTT+PYsWPYv38/fvGLX2Dw4MHQaDQYPHgw7rzzTuzduxdHjhzB008/7dXrEN4x3D6B/VxcN6C7ujqKmSM67Zz3Z79kMfiRwfEIkDZJnmsQbOhiwtVi2rYnAmHT2N8wW2344ghzO3JfdgQ43t+BLjMMFNsLhf4Js4cOgqKLb+2Y9DgsmzcSWrUSy+aP9Ou4YsLVUNnH44/vHlu091U3JQYK9jlEKharDVUtwsKQBQoxWrV4fl9lFXie79ZDwZkZufJ0aG419NyVmZGXFIXsxEiYrTZR/iYnTHrkaUYBcDSi81fTNb3JAr1JcPT0NKMQClJtSYHCq6++imuuuQYHDhzA7bffjpSUFIwbNw6/+93v8NVXX6GlRZpt1UcffYQbb7wRUVGu5QYxMTG48cYb8fHHH0s6PyEP84alIMFeoMP8vQca7UZLp3buLMtistjEJmz9EYeziPeyI0BaoFBc7+jI3JfMkGRwfbP5dBWa9CYMig7vJF9xB/b+UiAWGFghM2v41ZWXr5+I9ldvxaSsBH8OCxzHiQsgfzRdc9eJzREoeJdRqG7tgI3noVRwGBTt2EEeJ9Yp+EaOU6czoklvAscBw1wsoFmH5oKyRuhN0o1iWtyQHnEch0V2d7RNp+VvxsikR544HjFYtoVlX3wNyyaEqRSIDvdMiBMK0iNJgcLDDz+ML7/8Eg0NDdi3bx9WrVqFrKws/Oc//8H111+PpKQkTJkyBY8//rhH562rq4PZ3LvfusViQW3twFycBgtqpQI3TBCyR6yB00CD7bxHaVSI0aoRpVE5dQnuvxNChURnh55ggYLRYkOHmwFUiV020FchM0DSI3dYe+gCAOCmCVmdXFTcgTI2gcNqs+HnYlaf4LquhOO4gNXs+VOW5uih0EdGIUnQ7Z9vaofNJj2zW+Fk6OD8nRmXHgfAdxkFVp+QnRAFbVj3Beng+EikxWhhsfE4cKFB8uu4U6MAAAvFQKFa8mu5gud5r6VHAHC2ts0vqoa6NlafoPH4+zZgpUcMjuMwefJkPProo/juu+9QXV2N1atXIyEhAYcOHcLq1as9Ol9eXh4+++wzNDS4/gLU1dVh7dq1GDp0qDfDJmTglomChGHdkTJYrANPfuRcyAw4ugQDnluBBhOVMkuPojVqKOwTq7sTpbvWqAAtZPvCaLbiv0ftbkeTPZMdAU7SIwrE/M7h8ia0dpgRq1WLjjvBhD97Kbib6cyMi4BKwcFksYnPkUJXxyMG69DsK+cjFih0dTxicBwnZhX2eCE/cjdQmD8sBRwndKRmwZocVLUYoDcxa1TPi7KzEyOhUSnQYbbifKN3MjN3cNQneCY7AhyBQn823PAqULDZbGJG4fLLL0dycjIee+wx1NXVQa1WY/r06R6d7+GHH0Z1dTUmTZqEv/71rzh48CDKyspw8OBB/OUvf8HkyZNRW1uLpUuXejNsQgbmDUtBUpQG9TojfjxbE+jh+J2ugQKAkLBIreijqZGnKBQc4uw6WHcDhRIn6VFfOHcJ78+1Ib5i0+kqtBjMSIvRYqZd3+wJFIgFDqYLn5Wb7HEmyB/487PhbkZBqVCITa28kR+5mt8Bh/ToeGWzT3ayRWvUlO71CYzhg4QggnUKloK7gUJCpAZTBgs2s3LKj1j37CEJkQhTeV4Ir1QoMGyQ/+oUmONRkoQC9oQQyChIcj169dVX8eOPP+Knn35CW1sbeJ6HUqnE5MmTRfejWbNmISLCs8XGAw88gKqqKqxcuRLLli3r9Bh7jWeeeQb33XeflGETMqJSKnDj+Cz8a2cR1hacx8KRaX0/KYSoaO5eXBcKFqmOG7I80iNAeF8a9Sa3My3FEqRHFhsPndGC6D5ufAON/9o79t48abCkxaZYo9CPd8P6K2J9Qg+yo0DjT9lfhQe1UzmJUSip1+Fcgw6ze6jt6Pv1XBdP5yZFISJMCb3JiuI6HYa5cCbyBrboHdFDRgFwFMd6syElFjO7MV8uHJ6K/ecbsOl0Ne65WJ5eHUx2lO9FE8eRqTE4VtmM0zWtuHJMhizj6gmHNarngUIoSI8kBQqPPvooFAoFJkyYIAYGs2fPRnS09507n3zySdxxxx348MMPcfToUbS0tCA2Nhbjx4/HHXfcgbw8/zWVIXrn1slD8K+dRfjicBn+ees0j5qm9HccO06OG1coTAhyFzMDvb8v+0rruzX0YTuB7kiPIsJUCFcr0WG2oqHdSIFCF1i9x8XZSZKe769d45pWA3RGC/Ko+zMAwGbj8VNx74XMgYZ9Nnxtk20wWcS5wx1JZLYcGYUepEdKhQJj0uKw73wDjlQ0yR8o9GKNykiQYUPK3YwCACwamYY/bzyBzWeqYbPx3dy3pCDWJ3gTKNizLkyu5UtE6ZGHzdYAIN5udNKf1wWSAoUvv/wSl1xyCeLi4mQejkBeXh6eeuopn5ybkI9Lhg7CoOhw1LZ1YMuZalw+Kj3QQ/IbztaojIR+PiGYrTbU2ifEvlL8ntBT07UNJytx+T9/dPkclYJDVrx7Y0iMDENFswEN7UZkJ/adhRhI1LZJ6ybKYIvBtg4LzFabzzYDZr66EVWtBpQ/d714Yx3IHKtsRpPehCiNyu+ORu7CeinU2+cMX8E2L7RqJWJ7sfNkyGGR6mojiDEuQwgUjlY242YP7YZ7Q2c0i3IiV9aoDDk6/Yp9FNx4P6fnJCEyTIXatg4cq2zG+EzvG5wVypRRAPzjfORNRoFdryaDSbZAy99ImvWvueYanwUJRP9BqVDgpgHqftSb9Ki/1ihUtRjA84KrlRQtZk/01HTtE/tnJjMuAlMGJ3T699zV491uDMYCNDkkX5XNevxcHDquamwnbJCEGxwAxEWowUw+fGVB26w3obheB73JihNV/rE7DHa2Fwl1XzNzkz1qkOdPWOf2qlbfBgrOMiB3HGdyEr3PKPRWqzUu3Tcdms/Y6xMGRYf32sxNDomrJxmFMJVStFWWq06hqE64Nt5kFEaIGYVWn9en1bV5Ucxsv//xvON97294NQMVFBRg+fLluOaaa3DppZeKvz9//jzWrl2LxsZGSef98MMPsXDhQiQnJ0Oj0SA5ORkLFy7ERx995M1wCR/AnFT+e7QMJkv/7R/gKa6K3eL7uad/pZM1qpy7Hq6kR1abDd+dELq7/+eX07F/+RWd/j2+cLTb55dTHnPrOz9j9ppNOOiF9WCwYLXZRFmIlJQ5IGwGsEDPV1p0Z9eSEhk66oYCrJA5WGVHgEOeWC6jG44rSu2Zgb56KDC87aXA83yPxcyA73opiPUJfciZxMy1NzUKHgQKgLw2qc7WqN5kFIYNigbHCfeVrvJVuRHnUQkbLhq1Elp75/L+qjaQJD0CgOXLl+OVV14RIznnSJ/nedxxxx145ZVX8Lvf/c7tc1qtVtxyyy348ssvwfM8wsPDkZ6ejpqaGmzZsgVbt27FunXr8Nlnn0ERhC4QA5GZuclIi9GiqtWAjaeqcNXYzEAPyeeYLFbUtHXu2gn0f3cDXxQyA64DhX2lDajTGRGrVUsuOGTI6efOXEc2nKrCZLvbR3+lod0EttHmTYYoMVKDJr3JZ/Z+zoFC8QALFKw2Gz46UNptd3ibvSNzsBYyA465r7xJD57nfdbPgb0XU4e4933MsZsglDfrYbJYPXbVqdcZYbIIjkau5kIWKJQ2tKPFYEKstvfu8e4iWqP2GSg4MgpSpCw2G482o2eBAmu8tqO4Fh1mK8LVnjsVMapbO9BuskDBSbNGZWjDVGLh+qnqFsnySndwSI+kvUaCXR7bqDciB/1PHitptf3OO+9g9erVuOqqq3D06FGsWLGi0+PZ2dmYNm0avv76a4/O+7e//Q3//e9/MXPmTOzcuRN6vR7nzp2DXq/Hrl27MGvWLHz55Zf4+9//LmXYhA9QKhS4edJgAMDaggsBHo1/qGoVdt7DVApRpwv0f+lRT04f3sLel2aDI+36zXEhm3D5yHSvde9sh63BywDNYrWJwQbb0e3PMNlRYqTGK/mKrxtrlTY6ggNmjTtQeG/vOdz13m48vO5gp3+NehMiwpSYMjg46xMAxzzRbrL4TFLB87y4i812tfsiJTocWrUSPA9ckGAhyubBQdHhLoOM+AiNGCSdlFEqJ1qj9lKfILy+8H208Y4FvyfojBZxA8HdQGFkagzSY7XoMFu9lmYW1gl/Z3aiNGtUZ1j2xdcFzY4+CtI2XPp7d2ZJd49//vOfGDlyJNatW4cxY8YgLKx7RD1ixAgUFhZ6dN7//Oc/GDZsGLZs2dKtB8PFF1+MzZs3Y9iwYXjnnXekDJvwEbdMFAKFL4+Wud19tz/DHDEyYjtrZvu7ParoeCR3RsHFJPmtPVC4eqz3tnZsIeut5KtOZxRvoDtL6mDu540Ea9u8u7kxxEDMD9KjgZZR+PBAKQDBler2yUM6/Xv3zuleL6R8SUSYSpzzfCU/OlPTivJmPTQqBWbnudcHhHPaqT4n4fPUk+ORM+z8cv7douNRH4GC1u70Bki717CgTqXg3M4McBwnm/xIrE+Q0JG5K8z5iAVZvsBksaLFvsklNaPQ3x0RJQUKJ0+exMKFC6FS9axcSklJQW2tZ5Hn2bNncc0110Ctdh3lqtVqXH311Th79qxH5yV8y/ScZGTGRaCtw4IfTlYGejg+pydHjP7ueuR76ZGwkD/fqMOxymYoFRyukMEpS64ahZo2RyfXdpMFh8qk1VgFC946HjHkCsR6orRhYAYKNa0GsVnlh/fMwEf3zur0T05HHV/BFtMVzdK7IPfGRnvx7Oy8QdCGua+U9qZOobf6BAazaa3wovuzMzYbLy6gh6f0vYD2prmnc32CJ3KxhSNSAXhf0OywRvVegiM6H/mw6Vq9XXakVHDivcxTmJNbYz9dG0gKFFQqFUym3v/gyspKREV59kEICwtDe3vvlmbt7e0uMxhE4FAoONxszyqsPRT67kfspthVotPfpUeVMndlZnTdTfnmmJBNmJmb3Ku7h7vI1fippot7y/bC/t1x3BtLP2d83UvBOaNQ29YBnQQ5RX/kiyNlsPE8pgxOQK4Mu6uBQKxT8FFGwSE7SvXoefIECj1vmLDNlAqZ/u6GdqOYwXTHmjrBC+MMMVBwwxrVmUuHC9egoLxJdAGSgmiNOsj7HhQs++JLi1Q2jyZGaiSbfPT3+kVJgcLYsWOxdetWWK2uZSZ6vR6bN2/G5MmTPTrvxIkTsXbtWlRWut6Vrqqqwtq1azFp0iSPx0z4llvsu1/fnaj0SWv7YKKnHSfnBbGv7dp8QYWT65GcdA2gWKBwtUzdNOXwFQcgFqgzWGfc/opcGYUEH3fgLW3svDk0UJyPmKX0rf0gc9ATvnQ+MlttYiGzu/UJDG96KfS0EeQMe6xSpoxCtb3uLSlK41bNFvtOStmhFnsohHu24ZoSo8X4DMEadvMZ6fIjMaOQ5H1GgdUolDXpfbbB4G19AjBApUf33Xcfzp49i1//+tcwGjvfnFtbW3HPPfeguroaS5Ys8ei8y5YtQ0NDA6ZMmYJXXnkFBw4cQFlZGQ4cOIDVq1dj8uTJaGxsxLJly6QMm/AhUwYnIEqjQmuHWdYCr2Ckr0DBauPR1mHx+7i8RewNIWOzNaDzJNnWYcY2u0f81TI5ZMmVUWA369Fpwi7Vz8V1/TroZTthgyTqahmJEb4rZtYZzeJ5h9tv+gNBflTVYsAOe1EoM4PojzikR/IHCnvO1UNntCA5SiMuUN3Fm14K7kiP5M4osE2KVDeDem+suB3SI89NL72VH/E8j8Ja+TIKCZEacSPkjI/qFLx1PAIGcKBw22234a233kJycjLeeustAMC0adOQkZGBzz//HHfffTduuukmj8579dVXY/Xq1aivr8fy5ctx0UUXITs7GxdddBGWL1+O+vp60W2JCC5USgWm2e3rdp2rD/BofEtP7kBatRIalfCVajL0r14KbR1m6IxCcJPeS8pdCsyHX2+y4tvjFTBZbMhPjhYXht4iX42CcLNeNCINMeFqtHaYcbhcXq90fyJXMXOiF7uXfcFkR3HaMEy0d3wdCBmFzwsugOeFIuYhCf3PLpHhS+kRq0+4dHiqx5IPn9co2OdIuWoUqu2yx9QY9+Zeb6QsnvZQcGb+MMGud2+ptD4zNW0Oa9QcL6xRnRk2SJDt+WqDQY6MgigV85HFtK+R7Jn30Ucf4V//+hdycnJQUVEBnudx4MABDB48GK+//jrefvttSeddtmwZTp8+jWeeeQbXXXcd5s+fj+uuuw5/+tOfcPr0aSxdulTqkAkfMyNXcKXYVVIny/naOsz47WcH8O6e4qCS8vTkisFxXL/dOWA7YzHhakRpPL+B9Easkxb2/X3nAMjjdsRIEO1XTV5lAFigkB6rxSy7w8qOfiw/qtXJVczsuxoFFigMSYhErn1xVzwALFLXFgiyo1v6cTYBcGyW+CJQYLvWnsqOAEegUKczeixJcccmmmVdK1v0stybWDYzNca972qCF8WxLXYJqJRAIdv+vkqVXDHZ0ZAE761RGWx+avbRPVeOWq/+ui5gSG64BgBLlizBkiVLYDAY0NTUhJiYGI8LmF2Rm5uLJ5980uvzONPR0YHf//73OHDgAIqKitDY2Ii4uDjk5eXhgQcewJ133tnNbam1tRXPPPMM1q1bh+rqaqSlpeHmm2/G008/LcvfGWrMyEkCAOyWKaPwp++P4e/bzwAQ7DTfvONixEl0HZALm40XJ0lXO07xEWGobu3odxapFWIhs7zZBEDotRGrVaPFYMaGU8LN/+ox8jXmY3pdngea9WYkSpzQWTFzSkw45gwdhPUnKrG9qBa/mT1UtrH6E1F65HWNgu+kR8zxKDsxEnn2Lq2hnlEob9Lj52JhM+Xmif23PgHo3HRNTpr0Ruw/L7iOSQkU4iLCEKcNQ7PBhNKGdoxJj3Prea0Gsygb7W0uZNIjvUmwzvT2vuRxRkGsy/JvRiHNPr5mg0lS4zUmO5LD8YjBMtbNPjIRqRMzsyQ98gqtVov09PROi+effvoJc+fOleP0gq6tsBBlZWWSz6HT6fD666+D4zhceeWVWLZsGa6//npUVFTgvvvuw1VXXQWb025ke3s75syZgzVr1mDEiBFYunQphg8fjtWrV2P+/Pno6JBe9R+qXGwPFArr2rxyRQCAC43tYpCg4DisO1yGiavWY19pYGVNtboOWGw8FBzncvenv1qkVrYwa1R56xMYzk2CYrVqzHTTE90d1EqFeNPzZjHLMgop0VqxI+5PRbWw2YInm+UJtTLc4AD/SI+GJEQiL4llFEI7UPj8sNCYcmZuMjLjffN98xcsUGjUm2AwyVeX9ePZGth4HiNSYiS/R1LqFFhmJE4b1mtmVevUQ0KOguZquzVzij9rFDx0PQKE7DCT17IsiCecrRXqCPKT5XP5YhnrFoOvipm9zyh4U3weDEgKFBobG9Ha6rpwZM+ePVi4cCHmzp2Ln376yaPzfvHFF7jrrrvQ1OTQBZeWlmLcuHEYMWIEsrOzcdttt/XottQbCQkJaGlpwfbt2/HGG2/gz3/+M15//XUUFRVh7ty52LhxI77//nvx+JdeegmHDx/G448/jg0bNuDFF1/Ehg0b8Pjjj2P//v1Ys2aNx2MIdeIjNKKvsbdZhae+OwqjxYa5+SnY8+hlyEmMQmlDO2a+uhGvbDkVsMUb2zlLjQl32e22v1qkik4fMjseMVjTNQC4YpT33Zi7Ioc8xhEohGNSVgIiw1Ro1Jtw3MddP32B2WoTg9VB0fLUKHSYrdDLuBgEHF2ZsxMikWsPFEobdLD082Z3vREKbkeMWK0akfb+BnLp9QFgoz3zuGik59kEhpQ6BdHQwY3MaoaMBc39pUaB4zik2f/uKg+vd21bB97cXQwAmJApX8dxn2cUWI2CF5nZAdWZ+dNPP0Vubi6Sk5MRHx+P8ePHY9euXQAE69IbbrgBM2fOxJYtWzBp0iR8/fXXHg3m9ddfx+HDhxEf73A4WLp0KU6cOIF58+Zh3Lhx+OyzzyTVPygUCpf9F1QqFa6//noAQFFREQAhg/Hmm28iKiqqmwTqySefRFRUFN58802PxzAQmJEj7BTvPie9TuFYRRPe21cCAFh17QRMHZKIgieuwC2TBsNi4/Hofw/hgY/2yDJeT+mr0M2bnZ5A4qseCgznRjVy2aI64ygWkzYRW202sbFOSkw41EoFZtprbn4qkqfmxp+wv0XBcWKWSypRGpUY2MktP3LOKGTERiBMpYDFxveoeT9S3oS5f9mEgxekFVMGmvONOuwprQfHATdN7N/1CYCwcBQtUmWUH22y228uHO5Z/wRnpFikulPIzEgXm67JESj4r0bBm0ABcMiPqjzIKPA8j4c+2Yd6nRHjMuJw90U5kl7bFXH2jEJzEGcU2P2vtcPcLzdB3A4UtmzZgttvvx2lpaWIiIhAREQEjh07hsWLF2Pfvn2YMmUKvvzyS0ycOBFfffUV9u/fjyuvvNKjwZw8eRLTpk0Tf25ra8N3332HW2+9FZs3b8a+ffswcuRIyYXSrrDZbPjhhx8AAGPGjAEAFBYWorKyEjNnzkRkZOfK/MjISMycORMlJSVeSaFClel2+ZE3zkcrvj4MngdunjgY07KF88Vqw/DJvbPwz1unAgDe2VOCDrPnmSVvqejjRtJfG6v4qiszg02USgWHK0Z73425K95mFOp1Rth4HhwHJNnPNSd/EABgR3H/CxSY7CgpSnqTIAbHceLnWu5AwVGjEAWFghMXdz3Jj17degrbi2rxj+1nZR2Hv/jskCA7uiRvkLgz298RLVJlWDADQEl9G0rqdVApOFECKIVsu/SoVIL0yJ1AgQVIckiPRHtUj2sUpEuPPO2jwGBjrG51X1780YFSfHGkDCoFh//8crpshcyAPzIK8tijhtklW2Uy1/P4A7eLmf/yl79AoVDgo48+wi233AIA+Pjjj3HnnXdi4cKFMJvNePPNN3HfffdJHkxjYyNSUx07CD///DMsFgtuv/12AIBarcbChQvx4YcfSn4Nk8mEP//5z+B5Hg0NDdiyZQtOnz6Ne++9FwsWLAAgBAoAkJ+f7/Ic+fn52LBhAwoLC5GVleXyGKPR2KnHBJNqmc1mmM3+7TzKXs8frzt1sJAN2n++AfoOo8cSkx1FtfjuRCVUCg7PXDG625jvvygbS9cdhNFiQ3ljG4YkyGOx5i7n7VKJtBiNy/czRiN8pRp0HW6/3/68Pj1R3iws2FKiwnwyjli7Z/fMnCREqRWyv0Z8hLCrVNuql3Tu8ibhuiZFasDbrDDbrJiZLdj9/lRci7sGqQN6fTylyn49k6Ncf049JSEiDDVtHfb3Vx59scFkcThNRQufu9yESJypacXZ6mZckpvY7TnMUe14VXOnvysYvkPO/G37Wby79xwuG5mKX07Nxih7B9lPDpYCAG6akBk0Y/WWdPsu+PmGth7/Jk+uzw8nhIaMF2cnIlwp/ZpmxQrjKqnXuX2OMvv8nhrd9/cmxS7pK2t0//yuMFsd2cxErcqtc0WHCffVxnaTx6/dYt/EilBz3a6LO+dKiRIW5uVN7v3dlS0G/GbtfgDAHy4bhdEp0bJ+9iPt70Wz3vP3oi+sNpsYjMWFK706/8iUGBypaEZBWT0yYz3PTsg9x3lyHrcDBZYhYEECANx+++34+OOP8d133+Gdd97BXXfd5dlIuxATE4OGBkdK+ccff4RCocDs2bPF36nVarS3e95tkWEymfDss8+KP3Mch0cffRQvvPCC+LuWFkGTHBsb2+M4nY9zxQsvvNDpdRgbN25ERERgCtg2bdrk89ew8TyiVIDObMXra7/D0Fj3dw54nsfyfcLuzMJ0FQr3/4RCF8fFqHjUWYD/btiKYR6cXw72nRQWNm3VF7B+fffulFUXhEn4ZMkFrF/vmbWmP65PT5TUCN+p0hMFWF9+VPbzR7cJk9LUiFasX79e9vO31QmT+b6jJ7FeX+zx8w83CNp7Lczi+Mw2HmEKoL7dhPJ2VUCvj6dsrxLeb4VRJ8v7zRmFXbDNP++FodArszyRinYhBR+uBPZs2wyO46CwL5g27z+K9OYznY5vNtlQVC+M41h5I7797jsouM7ZkmC5Rq/ubEelnsfxqha8svUM8qIVmDZIhYNlJigARNeexvr1/TMr0hVDvXDNdh45jdHGc70e6871ef+IcA8YzHk3V5TphM9XUW0zvvvuO3Bc35m1gkLhtZvKirF+/YVej20qE75jh86WYv36GsnjbOgQxqnkgD3bN3f7TLtCZxZq9AxmK7785juEKd3PGlbYv0OnjhxCWJe53p3r01ot3OP2nyjEevSuquB5Hs8d7kCzwYq8aAXGmkqxfv15t8fqDmcahbm7vLZR9ntLs8kG5n67b/sWKL3IzibYhLXDum37obxwRPJ55Jrj9Hr3Mxtuz/gNDQ0YPXp0t9+PHj0a3333Ha677jq3X7QnRowYgW+++QbPPfcclEolPvroI0yePLlTzcL58+eRkiI9HRkVFQWe52Gz2VBZWYlvvvkGv//977F7926sX79eDAK8ZcWKFZ06SLe2tiIrKwuLFi2S7TXcxWw2Y9OmTVi4cGE3C1hfMKtiB344VQ1FxnAsvmSY289bd7gMha27ERmmwr9/dUWPDhDZZzaj7kIjcsdMwmIZ/fjdYc25HwHUYcFFE7F4cvdixKYD5/Hmmb0Ij03E4sVzOj2mN1kw6y9bMCYtFu/98mLx9/6+Pl2x2Xg0b/kcAHDTFQvcSrt7ymIAzxgtiNTIs8jsyn7uONaXnURC+mAsXjzZ4+c3HTgPHNqLoWlJWLx4rvj7mRe24cfCWhxvsuK+6y8LyPWRQtH2s8DxwxgxJB2LF0/3+nxvVf2Mk82VGDJsFBbPlMcudtPpamDXDuQmx+DKKy8HABRFnsV3ZYeB2BQsXjyj0/HfHK8Atu8EABhtwOiL54hSpUB/h7rSuuMLABZckpeMXefqUdxmQ3GbsMCamz8Id1w/N5DDk5ULPxdhXekhqOMGYfHimS6Pcff6WG023PPzVwCs+J+rZ4tNPKWgN1nw/3Z/Ab0FmD73UtF5pjeeOrERQDMunz0Nl/dRSG09XoF/nd4JmzYWixdfKnmch8oagZ82IzVGi6vclGvbbDzu2v45bDyPiy6Z75GM7f/t/xaAHgvnzMKUwUJRsSffn5o9Jfio+ADC4pKxePHsXo99d28JDtYfQJhSgc8fWih2vZeTtLImPHVwE6wqDRYvXizruU9UtQDbNyAhIgxXX+WZlL4rZ7Rn8OPXR9ARNajb3OYOcs9xPRkSucLtu7bVakV4ePeFG/udHIvf3/72t7j55puRmZkpZg6ee+65Tsfs2bMHkyZN8vq1FAoFMjMz8dBDDyEpKQm33HILnn/+eaxatUrMJPSUMWBvcE8ZBwDQaDTQaLpPTGq1OmA3Mn+99sy8QfjhVDX2nm/CUjdfz2y14an1xwEAjy4YicyEnuUNabERABpRrzf7/b0ssWuq85JjXL52cjTzme4+toJzjThe1YKT1a147+6Z3WRZgfpsVLcaYLEJ+vyshGiXbk5yEOfDv22QXTfb5OJ9d4d6e9+LtLiITs+fOywVPxbW4kSTNaDfXU9p1Au7nakxEbKMmX2uW4xW2d6D8hZhhy07MUo857AUYU4tbdR3e519Fzp3yT5b145hqfGdfhcM10hndHQ5/+bX82C0WPHJwfP4z94SFJQ34eH5IwM+RjkZkijM1ZWtHX3+XX1dn0Ol9Wg2mBGnDcPFuYOgVEifi2LVaqREh6OmrQPlrUakxPXt3V9pL9DNTozu828ZbL9HVbYYvLqe9Xrhs5ISo/XoPPERYWhoN6LNbMNgN59nsljFIuTMhKhur+fO9ycjXngfq9uMvR57obEdj3x5GACw8qpxmDA4ya0xekpiL/dcb2nuEOogk6PDvT73xCwh6D1e1eLVueSa4zw5h29WBBK58cYb8dprr2H06NEYNmwYVq1ahXvuuUd8fPv27WhtbcXll18u6+suWrQIALBt2zYAjtoEVqvQlb5qGAY6UpyP3tldjMK6NgyKDscjC0b2eixzhpDi4+wNOqNZLEQameo6SOzNHvVopbDQsfG87A2KvIEV46VEu7Z87Q84ipmlFbQ5W6M6c0meUNB8oskaVN3B+4J1ZfbGqcOZRB80XWOOR9lOdUa5Yi+Ftm7vN7NcZk2ejlcFp20tK/KMCFMiOlyF5Ohw/L+5w3Hg8Stg+dvtuHqsfM0GgwE5XY82nxbknPOHpXgVJDA8cT7qMFvFWgF33N/YMdWtHV452XjqeMQQnd48mPPO1rbBYuMRE66WnDl21x512RcH0dZhwfScpD7v6d7AipnbTRbZHYXqZJxHx2XEARD6TMltM+1rPNIB/OMf/8Ann3zS6Xf19cLkPWrUqG7HcxyHEydOeDSghx56CA899JDLx+bMmdOpx4JcVFZWAnBEWPn5+UhPT8fOnTvR3t7eyfmovb0dO3fuRE5OTo+FzAOdadmJUHAcLjTpUd6k77NhjtVmw+otpwAAv180GtF92LZJcV2Qg9PVQiZpUHR4j2ns3ibvIxXN4v/PNeiQkxQc3b1F73AfNVvzBwle2tL2FChclJ2IMKUCTSYbiup1GJUun/+3L5GrKzPD20DMFaVO1qiMXPvCrsVgRmO7SeyybbbasO+8UL9204QsfLC/FCeqmmUbi5ywBVRajLabLt4dnXx/gy04q9sMMFttXvVIKaoTOvcySYy35CRGYk9pvVu9FNg8qFUrO9k598SgaA2UCg5WG4/atg6kS1x4e9pDgSGl2y/7zoxOi5X8WWT2qDVtHbDabD0GdKz7+MvXTZQl6OuJWKfGcS0GszhnyIEcjkeMlBgtBkWHo7atAyeqWjDVC1mdv/Ho6tXX1+P06dOd/tXX14Pn+W6/P336NE6dOuWrcXvMyZMnXRZv6PV6sZaA6ds4jsMDDzwAnU6HlStXdjp+5cqV0Ol0WLJkie8H3U+J0qjF6NmdrMI3xypQWNeG+Igw3D8jr8/jA5VROF0jBAojUnqW2bHJu6XD1K0p3NEKR5DrSRMgX+NoMtR/AwVv7VFrWh1dmZ3RhqkwbYiwaNl6VnrBor9h9qhyBQrMt90nGYVER8CsDVOJFr0lTt+Rw+VN6DBbkRARhhsmCP0HTgRtRsEeKISI/WlfJEcJfUd43vs5mWXCUjxcNPeEJ03XnK1R3VlEKxUK8V7kTbM5MaPg4XdVikUq+854UyuQHKUBxwmZcZaB6Yqzo1lP2Xe5UCkViLLXvsltkSpnRgEAxqXHAei8FugPuJ1RsNnkbxJx4YLgKpCRkQGlUin+7A6DB3vWrGbt2rV49dVXMWvWLGRnZyMmJgYVFRX4/vvv0dDQgNmzZ2Pp0qXi8cuXL8dXX32FVatWoaCgAJMmTcKhQ4ewceNGTJ06FQ8//LBHrz/QmJGThMPlTdh9rh4399GB9JWtQkD561n5iNL0rZtLddrR8Cen7B16R7oRKPC8ECzE2xdYVputk1QimAIFJj3yVQ8FfyCb9MhF+v/qMRn4uaQer/9chP+dM6Jf7AqzQEFu6ZEU3/aeOO8iowAI8qPKFgOK69rEXTe24XBxThLG2m+2p2pae93RDBRVopSk/36fPEGh4JAeq8X5xnaUN+mRFS/dslqO5lbOsKytO/Mt693B+i+4Q0ZsBCqaDahs0QOQtkPsaQ8Fhth0zYM5j92DxqTFefRazqiUCgyKEmo/qloNLoO6C3YZWpRG5VZ2xltitWrojBYfBAryfh7HZ8Rj85lqHK1sluV8/sI3FiRukp2dDY7jcOrUKQwbNkz8uS84joPF4pnG66qrrkJlZSV27dqF3bt3Q6fTITY2FuPGjcNtt92G++67DyqV4+2IjIzE9u3b8cwzz2DdunX48ccfkZaWhkceeQRPP/00tNqBcROQyvScZPzzp0Ls6iOjsLe0Hj8X10GtVOA3bjokMXmIv6VHp+wZhd52SMJUSkSEKaE3WdHY7ggUiup0nRrEedIt1NdUiF2Z++9nmu2utZssMJqt0Kg9s82taXPUaXTlvotz8Mx3R3CyuhUbTlXh8lHyN4yTG1F6JEPKHJBfemSyWMUGXdldAoW8pGj8XFzXKaOwq0SQuM7ISUZOYiTC1Up0mK0oqdchf5B/XeT6wll6NFDIjIvA+cZ2r3bWAfkzYZ7UKJytFeb34R58ntjmSkWzDBkFqTUKHmUUmgF4l1EAhGxZTVsHqloMmOCi5IY1uRuSEOmXjZU4bRgqmg1okbk7syOjIM/nkSktjjrJkPsDAQ0U7rrrLnAcJ7oHsZ99wZQpUzBlyhSPnhMbG4s1a9ZgzZo1PhlTKDMjV3A4OFTWBIPJAm2Y64/aK/bahDumZLut8XSWHvE877cdXjGj0EcqNSFCA71J30k72jXVGFwZhf5foxAbHgYFx8HG82jUmzySfdhsvLiwdhUoxGrDsDBDja8vmPHKllNBHyh0mK1i91X5pEfyFjOXN+vB80JhctcxOgqanQIF+4bDjNwkKBUKjEyJQUF5E05UtQRfoGDfwEjzcOHXn2F1Cky+IxW5M2E5Tt2ZbTa+1y7lZ+wbQcM8+DwxuWaFF3+3v2oUOsxWFNUJ36nRXmQUACEIPowm8bPeFVdGBb7EV92Z5c4osEDhSEWTX9cu3hLQQOHdd9/t9Wei/5KTGCVa0x0sa8Qsu3uMM+fqdVh3WGjY8sj8EW6fm+nIDWYr2josiNH63mrQbLWJhXa91SgAwgRe3twlULCnGidmxqOgvCmoAgW2G9afpUcKBYeEyDDU64xoaDd6FCg0tBthtdeTJPewsL5qsBrflVuw+Uw1Dpc3YkJm8BY1s10wtVLRqdDPG1hGobHd1OeCyx1KGxyyo643yzwWKNi/b+VNepQ16aHgOFGKNCY9TgwUrhsfXKYS1QNMegQ4BQpeOB+1Gy0w2LOucgW4WfGRUHAcjBYbqlsNvW5Gna0VPm/D+5jfnWFzZqU3NQq9ZDN7gxlqNLoZKJyuaYGN5xEfEeZx9qIrfdUJujIq8CVx9nmuWe6MQpu8GYWRKbFQKjg06U2oaDb0afQSLASXuJMIGTiOw/QcIavAbA278tdtp2HjeSwakYaxGfEuj3FFpEaF6HAhxvVXQXNxnWArFxmmQlYfX25XFqks1XjtOCFPW93aAUOQWKSFQjEzIL2gmWmEEyM1PTq2DNIqcON44dq9uvW0F6P0Pc67YHLtWLH31sbzYrbCG8T6BBd69rwkwZ+eSY9YfcL4jDixholJJ4LR+ahqgBUzAw7ZIpOTSYFlE7RqJSJ7yEB7ilqpEOfr3jZnrDYbiuqFQGHYoJ57+HSFZWGl/t3tRgvaOoT7gOc1Cp7Zo54Q6xOkOx4xmKyuJ4tUV0YFviSWZRQ8cIByB3EujZYno6BRK8WNRmaX3h8I2kChoKAAb7/9NtasWYO3334bBQUFgR4S4SEzcoV+CrtKutcpNOmNeHNXMQBI8lhOtWcV2G6Mrznl5HjU1yTryqqTTQpzhqYgxm7/ynZdAkmH2SruSPXnjALgrNn17GbRkzVqV5bOHQ4A+PhAaVD1weiK3DpvQLjBscWbHPKjUnEh0T1QYNKj8mY9jGYrdtk3GtjGAwCMtsv/grGXwkCtUQC8kx459/6QU5LB6hScpWxdOd/YDpPFBo1KgcEeFGOLAZLEGgVWG6VVK8XNL3fxtEbB4XgU59HruELspdBjRsFRo+APWEahRYZNDIbNxqO+XT57VIbD+ahZtnP6mqALFPbv348JEyZgypQpWLJkCR555BEsWbIEU6ZMwYQJE3DgwIFAD5Fwkxn2G/uuc/XdGoz8e2cR2k0WjE2Pw8IRqR6f25H69E9B82l7fcKI1L7T0l21oy0Gkyi1GJcR55Fln69h9QnhbnqHBzOSMwr2m50rxyNnJg9OwCVDB8Fi4/H37WekDdIPyG3px3DYMXq/a9eT4xEgjDtKowLPC9+R3WJ9QrJ4zBj7zfZMbavsTZa8wWy1iYuLgZRRkCNQqPNBgAsAo+xzdm92uqw+IT852iNZXbo9oyBVeuRcn+BpcORpjYIc1qgMlv3oKVAIhRqFZoNJlKQm9dA3SQrj7OqJ/mSRGlSBwqFDhzBv3jwcPXoU8+fPx9NPP43XX38dTz/9tPj7efPm4fDhw4EeKuEGkwcnIkylQG1bB1J/vw4PfLgHPxXVwmi24m/bhIXWI/NHSto9Ei1S/RQoiI5HKX1Psl2lR8fs9QmZcRFIiNSIBXbnetnh8hdsJywj1vMbVbDhrfTIHY3wo/bs1792FqJNxt0rOaltk7fZGsPbXhXOOLuidIXjODGrcKKqBYfKhBsq6/gOAIPjIxEZpoLJ4qgdCgZq2zrA84BSwcm6uAh2HEW9hm79Y9ylVuYmgQy2MDvSy8KM1Sd4UsgMODIKzQaTpG67Uh2PAM9rFI7L5HgEOAr1XW3UmSxWMXDyV0aB1WLJKT1isqOYcLXHLnq9ITof9SOL1IAWM3dlxYoVMJvN2LBhAxYuXNjt8Q0bNuDqq6/GE088gR9++CEAIyQ8IVytxD9vmYrnNhxHaUM73tpdjLd2FyMpSoN6nRFpMVrcPqX3Hgs94ejO7Cfpkeh41PeNpKu/NUsxjrdPEJ5Y9vkallHo77IjwHPNLqO61f1A4crRGRieEoMzNa14a3cxHp7nfhG+v6iVuQCPIWegcL6p9x3HvKRoHK1oxmcFF2C22pASHd5JpqRQcBiVFov95xtwvKoFeYnBUV/DZEcp0eFeF3z3J9JiteA4R0ZFymJfbitKhjtSjzPMGtWDQmZAWEQyO+zKFgOGJrtf3wBI76EAOOa7ZkPfBgN6k0XMYHvTQ4EhSo9aujsPljX17GjmK1hGQU7pka8ys+PtgevpmlZ0mK0IlzEI8RVeZRQsFgtOnDiBnTt3YseOHS7/ecKuXbtw8803uwwSAOCyyy7DTTfdhF27dnkzbMKP3D9jKIqfvhbbfncp7pueh+hwldjN8f/NGYYwlbQvidhLwQ9N13ieF7syu9NlsmtKmKUY2c5WMEmPxIxCPy9kBvyTUVAoOCy1Bwd/+fF0UMleGOwGN0hu6ZFMFqkWqw1l9hqPIQmuix2Z89FXx8oBCLKjrhkvVqcQTAXNYlfmAVSfAAhFw6xuTKr8yFFbI+/nlsnUqloNorypK46MgmcLfY7jHAXNEv5uh/TI8wV11+aevXGqugU8Lyx6e3J28wQW2BicrJgZzrJCf2Wp4yLkL2Z2mELIG+ykx2qREBEGq40XNyCDHUkZBZ7n8dRTT+Hvf/872tp6T/tardZeH+80GJUKQ4b0vsOck5MDpTL4IzDCgULBYU5+Cubkp+DvN0/Bf4+UoaxJLy64pNCXPZucVDQboDNaoFRw4gKmN7pKj1iKke1sidIjHwYKXxy+AI1KiSvHZPR6XCh0ZWaw7sGeBwqsRsG99+CuaTn447dHcL6xHesOX8Ctk7M9ej1f44tiZqCzRao3VLYYYLXxUCsVSIt1PUYmPWJNCmc4FTIz2AKwN+25vxF7KITA98lTMuK0qGo1oLxJj0lZntsH+yoTFh2uRm5SFErqdThW2Yz5w7vXxLEaBU+arTEy4iJQWNcmMVCQZo0KCM09I8NUaDdZOjX3dIWchcwAEBGmQky4Gq0dZlS3doiuQ4CTUYGfZEeAc42CfBmFepl7KDA4jsO4jHhsK6zB0YpmTJTwXfE3kgKFlStX4vnnn0dcXBzuuusuZGZmdupqLJUZM2Zg7969vR6zZ88ezJo1y+vXIgJDRJgKv5ia4/V5HNIj32cUWNQ/NDnarQyIswTGZuPFGgUxo5DkW+lRTasBN7/1MwDg/J+u69WruaC8EQA8cvoIVqQXM7ufUQAAbZgKD83Kx8ofjuOTg+eDL1Dw0U6Y1ECsK8wRJSs+AkqF66Q2s0hlTHcRKDgsUoMpUJCuOe/vZMZF4MCFRslWoXJ3E3dmfEY8Sup1OOoiUGg3WsQsiKc1CoB3vRSkNltjJESGCYGC3oS8Xo6Tsz6BkRarRWuHGVUthk6Srd6MCnyFWKMgYzGzKD3ygXxqXEacECj0E4tUSav7t99+G0OGDMGBAweQmJgo22BefvllzJw5E3/84x/x+9//HhERjgWOXq/Hc889h4KCAuzcuVO21yT6J/7MKJyqsdcnuKlfdUiPjCht1EFntCBMpRDT2mwCbTaY0Kw3IVItb3r2SEUzbLxQUPju3mL88fKxLo8rbdBh69kaAMA1YzNlHUMg8LS4j+GJ9IjBHHiCqZCWIUqPgrSY2R1HlFynzJ1aqcDkwd3vM2zRc7a2FSaL+5lrXzIQrVEZ3jZd85X0CBCyuf89UubSaaawTsgmJEZqkChh99jRQ0JKoOBdYJkQoUFZk76TFbcrnHsoyEVajBZnalq7OR/1ZlTgK8QaBRkzCixw9YUpQX+zSJUUKFRXV+Ohhx6SNUgAgNWrV2PcuHF44YUX8Prrr2PixIlISUlBTU0NCgoK0NzcjNmzZ+Pll1/u9DyO4/DWW2/JOhYiuBFdj9o6ZOkU2xunqt2vTwA6S4+O2CeC0amxUNmbeUVp1EiO0qBOZ8S5Bh3GpHqmi+0LZ3ePt3eX4PeLxrh8f97ZUwIAmD8sRcxy9Gek7HjbbLy4QOnLHtWZXHudSUmDrlsxX6Dx1YJLLNL3Ugfs3JW5J4YkREKp4GC18ZicleCy4C8zLkKUP5ytC3y9D+BUozAgpUfeWaTW+qiYGXBymnGxMJNan8BI96pGwbsu3syyuC+LVLmlR4DD+ahr0zV/W6MCzn0U5OkcD/iumBlwskjtJ85HkgKFnJwctLa2yj0WvPvuu+L/m5qasHXr1m7HuCqSpkBh4MF2S602Hg3tRp+kBxmnnZqtuQNbULV1WHDwQgMAx8TAyEmM8lmg4Lxrdq5Bh22FNd3S7VabDe/sERre3T+9t6R1/8Gx421ye/HepDfBYrdz9ETykJ0YCQXHQW+yoqatQ/KNXm7ajRboTcLuerBKj9zp2qpWCo2vzjXoXMqOAGHeH50Wi93n6nGyugXBEOpWebnw68+wjIKUnXWe5x3SI59IPYT590R1CyxWm7hpA3hXnwAI1tKA59IjnudFMw6pGYV4bd+9Tdo6zOJ3Tk7pkSj/7dL0tDQg0iNHYXeb0dypZkIqdW2+kXACwnXgOGFTp6bV4HZ9XKCQFCg89NBDeP7551FbW4tBgwbJNphz587Jdi4itFErFaLNak1bh08DBYc1qnuTbFyEWvz/jqJaAA5rVEZOYhT2nW/wSZ0C26XISYzCuQYd3t5T3C1Q2HKmBmVNesRpw3D9+CzZxxAIWIBmttrQbrIgSqPu4xkO2VF8RJhHXtlhKiWy4iNwvrEdxXVtQbMwZLtg4WolojTyul87B2LeIC4k+qiLmTYkEecadLh8VHqPx7BA4URVCy7yalTyQNIjaRmF1g4zTBbBQcwXO7i5iVGijWlRXRtGOM3lZ+3WqFIzCmIPCQ9rM1oMjr85Jdq7jEJv3ZlP2u9faTFaUZ4pB84WqQyL1SZe/942AuQmXK2ERqWA0WJDs8EkT6Ag1ijI/3mMCFMhPzkaZ2vbcKSiGYuCfL6QdCe59tpr8dNPP2HGjBl46qmnMGnSJMTEuI7GBw8e7PZ5+3I8IghnUqLDUa8zorrVIDqgyE2T3iguJt3NKCgVClESsfd8TxkF3zgfmSxWUSr10nUTcfNbP2Hd4TL842aTaCEHAG/tLgIA/GJqNrRhQdVORTIRYY6bRUO70c1AQbrrSG5SlBAo1OswM0++DRNvEGVHURrZ5VCy1ygk9h4ovHbLVPx27vBOHZm7wqQUJ6tbcZHnDd5lxXmHuCc3p1CGmSaUN+k9luOxz22URuWT+Uih4DA2PQ57SxtwpKKpS6AgSI887aHAcC5m9uTvZrKjOG2YZC/9rj17XCFnR2Zn0sTuzA5DkU6OZn5e/MZpw1DT1iFbnYKv7FEZ4zPicba2DUcrmrBoZJpPXkMuJPVRyMnJwRdffIGSkhLce++9GD9+PHJycrr9y83N9WpwjY2NKCsr8+ocROjiD+ej0/ZFd0acFtHhfS88GWynh+0YjesSyPiql8KZmlaYrTbEhKtx44QsjEmLRYfZio8OlIrHNOiM+PKo4E8fKrIjQJCieLrr7anjkTPMKrckCDpsM3x5c2Of6dYOM8wS+0fYbDwuNLknTUiM0vQaJACO4syT1fJLYT2lSW8Sv++pEneI+zNMgtNusnTz1u8LX8qOGKzRlXOdAs/zYrM1KY5HgCNQMNk3KNxFbPTohUOWOzUKJ3zgeAQ45FLOGQXmaDY4PsLvDQfjnBrQeYsghfNdjQLQvzo0Swrd77rrLp8V77W0tOCpp57CJ598gvr6enAcB4tFaI2+d+9ePPvss1i5ciUmT57sk9cn+g/+cD46xRqtpXg2ycZrw1AKYUGUGhPeTRrlq0BB7NmQEQeO43D/jKFYuu4g3tpdjP+5ZBgA4IP952Cy2DAxM75feDh7QmKkBpUthk4uIAcvNOCxLwtw3bhM/HZu594dNRIKmRmsoLm4Pnicj3zVQwEQ5FkcJ+iAm/QmSa9R3WqAyWKDguNkafLHMgrF9TqYrIFtGsjqEzyVsYUK2jAVEiLC0Kg3obxZ75H8w9FDwTeLMsDJacZpYVanM6LFYAbHwa0eOa4IUylFc4qKZgOS3AzSxUJmL76r7tQo+KKQGXBIj5zvv4GwRmXE2jfymvXeZxR0RguMohTON8HruPTugWuwIilQcC46lpPGxkbMmDEDZ8+exaRJk5CcnIxTp06Jj48bNw47d+7Ehx9+SIECIe7a+TKj4Gl9AiPeSebDJgRnmMtQaUM7eLuVqRywSYftnt05NRvLvyzAobJGHC5vxPiMeLy1O7SKmJ1x7h7M8zz+vv0MHv1vAcxWGw6VNeKh2cOgdipkdFijer4DnJcsaJqLgyijUOvDXTClQoE4bRia9CY0tBslBQrMGGBIQmSn6yCV1JhwxEcIYyrXB7ZLNpuHBmJ9AiMzPkIIFJr0Hi1MfdlDgSE6zTiZPZxx+jx6I3lKj41Anc6IyhY9xmd2n+9d4Shklv55cVhC95zJYD0U5LRGBRyf80a9CUazFRq1UnQ082d9AsPRdM37jALLJmjVSkTKXOvFYBmFk9UtMFttssyHviKoRvbMM8/g7Nmz+OSTT3DgwAHcfPPNnR7XarWYM2eOSzckYuAhZhTafJhREAMFz9LSnQKFLoXMgJCa5TjAYLaKDbLkgDVwYbtnSVHhuG6c0CPhrV3FOHihEccqm6FRKXDH1GzZXjdYYNKjwro2XP/GDvzu84OiTKbFYMbPxbWdjvemM2pQSo/afCvhcDQTlPaZPVQmNPib6OZiqi+Y8xEAlOl8Gyi0GsyY/7fNeGHDcZePi4XMA9AalZERK835yJeZMMZY+5x4oUmPZvsOvFjInCxNdsSQ0kvB2x4KQOfmnq5o1ptQ0Sy8ziiZA4X4iDCEqYQlJLsHBzKjwExE5KhRqPNRV2ZnhiREIjpcBbPVJgaswUpQBQpff/01rrrqKtxyyy09HpOdnY3y8nI/jooIVvxSo+ChNSqDFZkBjt19Z8JUStElpFRG5yOWUXAunmaZgw8PlOK1HWcBADeMz0J8hO8mwUDBAoUnvz2Kr46WI0ylwN9umoK7LxLqpb45VtHpeK+kR/ZAoaatAzqjfI1+vKHWR83WGN46HxWUC4HsxCx5AgUAGGPfub7g40Dh+5MV+PFsDVZtPukyCziQuzIzpDof+TITxoiLCMNge8H1Mbv8iNUnSC1kZrA6BU96KXjblRnou0aBOR5lxkXI4gTkDMdxomyK/S2lAeihwJA1oyBK4Xz3XeY4zkl+FNwdmiUHCm1tbXjhhRewYMECjBw5Erm5ud3+5eV5Jm2oqqrCqFGjej1Go9GgvV1+S0mi/8FuyDU+qlHoMFtF+1KvpEcuMgqAU51Cozyf53pdh+jlPSbdMd5LR6QiKz4CTXoT3t0rNFm7f8ZQWV4z2GBe/wAwNDkaux+5DP9v7nBcMzYDAPDN8YpOizxvipnjIzTidT4XJFkFXxfgeet8VFAuZBQmyVgbwzIKO6ot+L+fiyRnO/pin93BrMVgdllbNJCtURnOzkeeUOeHjALQXX7kbbM1BsukeNJLQY6MApt/GvVGl8HrcXtAJHchM6OrRWpAaxTsTdfkkR7ZMwo+sEZ15p+3TsW5Z6/F7VOyffo63iIpUKirq8OkSZPwhz/8AQcPHsSZM2fQ1NSEmpoalJaWorS0FCaTCTabZzs8iYmJfbocnT59GmlpwW0lRfgHtrhjWk+5OVvbChvPI04b5vFCkk3gKgXXYzaCBQqlMhU0s2xCXlJUJ2tQpUKBey92BO05iVGYl58iy2sGG5cMHYQwlQJ3Ts3GocevEBekC0ekIUylQFFdW6c0r6NGQdrN2lHQ7Pk1lLM2hVHrY+mRN03X2o0WMUM3MVO+QGH+sFRo1UrUdfD47bpDSPvDF7jpzR345li5ZHcmV+y3BwoAcKis+w6g2JV5AAcKzPnI054CtX5wPQIc/WxYQbO3zdYYYi8FDzIKNXLUKNizwkaLDQaztdvjJ+wZhTEyFzIzHBapBo8czXyBI6Mgh/TI9xkFQAhcsxOjfGYOJBeSAoVnnnkGxcXFeO+999DUJEyYS5cuRXt7O/bu3Ytp06YhOzsbJ06c8Oi8l1xyCb766qsepUUnT57EDz/8gEsvvVTKsIkQg02w9TqjrAsChnN9gqdfZKYdHZkaizCVawcU1kuhVKaMgsPxqLus496Lc8H+hHsvzvW7dZ2/uGJ0BtpW34L3757Zyc42OlyNuUOF4Oib44L8iOd5UfIg1aIwL1laoPDEVwVIfPxzHJM55ezoo+CjGgWxeNLzXbujFU3geWEHVU4d/6i0WJx98krcNywM49LjYLLYsO5wGa7513ZcsmaTLK9htdk6BQcF9loLZ5if/ECuUWDSI0/d3Gr9IPUAHHPjkYomWKw28Xs7LMW7jEK6hO7McmQUojQqqOxzuas6BV/1UGCwe3BVi0F0NFMqOPFz4E9YoNAiZ0bBh1K4/oSkQGH9+vVYsGAB7rzzzm4LqKlTp+L7779HaWkpnn32WY/O+4c//AFWqxUzZ87Ehx9+iPr6egDAqVOn8NZbb2H+/PnQaDR47LHHpAybCDESIzVQ2ifJWh9kFaTWJwDCznZKdDh+OS2nx2McGQWZAoWKzoXMzmQnRuFXM4YiPzkaS2aGpuyI0VNgdrVdfvStPVBoNpi87oyalyQsMEo8sEi12Xi8uasYTXoTnuuhMFYKnby/fZQy90Z65Chklt+SNyU6HNcMCcOBxxbh8BOLsXSeYIO7p7ReLFz1hlPVrWg3WcSfWa2FM1SjAEzIjIdKweFUdSsOOGVg+qJOrK3x7cKMzY3HKptRUq+D2WpDuFqJrDjvdsA9LWa22mxi9k/q3AMIOncWvDe5cD7yVQ8FhsMitUOUHWXEaqEKgINPnCg9krOYeeB+l52RdDWrqqowceJE8WelUgmDwfEFiY+PxxVXXIG1a9d6dN6xY8fi008/RXNzM+666y7885//BM/zGDNmDJYsWQKDwYC1a9ciPz9fyrCJEEOh4BzyIx/UKbAOx57WJwDAiNRYVP35Bjx2ac81N2KgIFdGQSxkjnP5+P/dfhHOPn2NV6nu/gwLFHaW1KGx3SjWJ8SEqyV3RmUFzZ5kFE5Wt4gL7c8LymSTnrV1+N77OzFCuvSILa4nyVjI7IrxmfF49cbJ4m6gHL1KWH0C27U85CKjQNIjICVGK+qtX9l6qveD7dhsvN8WZkOToxGuVkJvsuKHU5UAgPzkaK8zrOn2GoXatg6YLN0lQF2p1xlh43lwnPe71o46hc4Bcb2uQywyltvxiOEsPRILmQNgjQpALNaWY2PA17Ve/Q1JgUJsbCzMZkfUFh8f300uFBMTg5qaGo/Pfc011+DcuXNYvXo1br75Zlx66aW4/vrrsWrVKhQXF2Px4sVShkyEKL50PjpVY5ceSXTE6EuuxAKFC016WL3Uq1usNlGP6spliQCGJERhbHocrDYe35+s9Lo+AZBmkbqjyGHRauN5/OXH05Jf3xl2c4vSqBDhhSd8b3gjPfJlRsEVcjY1ZPUJd0wZAgXHoaato1NHWoPJItoyDmTpEQA8Mn8kAOCzggu44MYmSJPeBKtNmP98vTBTKRUYbd/4+azgAgDvHY8AIClSI/rgu3MvYsckR4V7vfue0EOgsKtEUGQMT4npVLMmJ2lO3ZkDWcgMODIKLR52BXcFZRQ6I+kTmpubi9LSUvHniRMnYtOmTWhoECZTg8GAb775BoMHD5Y0qISEBCxduhSffPIJNm7ciM8//xyPPfYYkpKSJJ2PCF181Z3ZZuNFR4wREjIK7pAWGw61UgGLjUdDh3eBQlFdGzrMVkSGqcQFEtGdq8fY3Y+OVXhljcpgGYXSxnZY3TRv2F4kbKBcMnQQAOCt3cWy7IL5Q+ctVXpkslhx3K6XltPxqDccgYL3GTsWKMzJTxGliM5ZBSY70qqViAn3zaKsvzA+Mx4LhqfCauPx1219B8EswI3ThvUoG5QTlnHdWVIHABiW7F19AiBktz2xSJWjPoHRU28TNs/Msc8zviDVOaNgD8gDYY0KCPa3gLwN1yijICApUFi0aBG2bNkCvV74Qjz44IOora3F+PHjcfPNN2PMmDEoLi7GPffcI+dYCaIbvsooNOlN6LC7SGT5qDBLqVCIuy81Bu+KsVkh89j0uJAtVJaDq8cKzed+OFUpWjh6k1HIjIuAWqmA2WpDmRuWkDzPY3uhkFFYedU4jEmLhc5owb93FkoeA6PWDzrvnlyPTBYrDl5ogM3mOuA9USV0H43ThiE70T8LCWYW4G1GocNsFb9f04YkitIpZvUKOOwhU2O0Qe9g4g8eXSBkFd7YVdRncak/eig4wzKuLIk7zEvHI4YYKLhRpyD2UPCiPoHhqFHo/D6zecaXgQLLntW0dYgBeaAyCrHhDumRN45yPM87bLMHcL2RM5IChV//+td44403xEDhhhtuwMsvv4z29nasW7cO1dXVWLZsGRUdEz6HNXypkbmYme0oxISroZGoX3cHtpipNXiXUThi13/3VJ9ACEwbkohB0eFoMZjx+WFBeuBNoKBUKMRr6I786GxtG2raOqBRKTBtSBIesS+o/rrtjFva5t5g6XJfOR4BrhuunalpxbSXN2DKSz/gXz0EPKw+YUJmvN8W0nJJj45UNMFstSEpSoMhCZGidMrZBaladDyihQUAXDYyDaPTYtHWYcEbO4t6PdbX3cS70nWOlEN6BHhmkco6GcuRUXBVo9BiMInfuTk+tMIeFB0OjgOsNh4H7Rm2QNUosM7MFhsPvUn6XNrWYRGtZr0pNA8lJAUKaWlpuPXWWztJgR555BHU19ejqqoKOp0OL7/8MpRK36cRiYFNio+kR/6yR2OLma4ZhXpdB74/UYF2o8XV07ohWqO6cDwiHCgUHK4cnQ4A2H1O0PCmeFl86klB8/ZCQQ5wcXYSwtVK3D45G6kx4ahsMeDTQ+e9GocoPfLhgov5tneYrTCYLHhvbwkmr/oeR+yOW/+xN/TrCrMT9XUhszNyBQpMdjR1cCI4jhOlU50yClTI3AmO47BsviMI7s2+2tfdxLsytssc6W2zNQaT3JS48XmrkaErM8OV9GhnSR1sPI+8pCgxgPEFaqUCSV3kiIHKKESGqUQXRG8sUmvsQVx0uAqRGt/UevU3ZPWwUiqVSElJodQr4Tcc0iO5AwX/+HqLgYJTjcLm01UY8/x3WPz6NqT+fh3u+2A3thfW9CjrAJysUamQuU+Y/IjhTUYB8Mwidbu9kJnt8mnUSvx2znAAwCtbTnuVMnf0UPBdcBsd7vBtv/Xtn3H3+7vRbrJgdl4yFByHvaUNLl2cDpX7t5AZAHKSHPbD3ryvYqAwJBGAkBVh52WLM2fpESHwiynZSIkOR3mzHp8d7rmRqqO2xj/So6SocFEmlBSlEaU73pJvr3UorO17HmD3K2/nHsARvDtnFHZ0mWd8iXPxPsf5TqrbFxzHydJ0TU5ZWKjgVaBQUFCA5cuX45prrunUBO38+fNYu3YtGhu7W8gRhJw4ipnllh75K6PApEc2WKw2/OHrw1j02lbUtHUgXK2EzmjBO3tKMPevm5H37Fd46tsj3Qpfm/UmXLDr47vulhHdWTgiFWEqx9Tnbfo/z82MAs/zLgsMH5yVj4gwJY5UNGHLmWrJ4/CHUwfHcaL86JvjFVBwHP505Tj8+LtLMSdf+JuYmwzDarPhSHkzAP8VMgPA4PgIcBxgMFt77bPSoDPim2PlPQbi+y8IgcI0e6AQFxEmZpEO2+UdrDs8ZRQcaNRK/D97EPyXH8/0GKzV+akrszNMfiRXNgEQrFcBwViiL6rFrswyBAr2uiHnGgV/1CcwnD/zaTFan0p1+yJW7KUgPaMgZ6F5qCA5UFi+fDmmTJmC1atX49tvv8WPP/4oPsbzPO644w68//77sgySIHqCRf1M8ykX9WzR5eObF8solLfbcOlr2/DnjSfA88CDM4eiYdVN2PHwQjwwIw8x4WqUNrRj5Q/Hsei1rdAZHTsmx+yyo8HxEaLzA9EzURo15jnttHmrQ3VXelRSr0NFswFqpQIX5zhkmwmRGtx3cR4A973nXSFmFHz8mWVyv/RYLX783QI8ecVYKBUK3DppCADg04OdJVSFtW1oN1mgVSsx3MsOuJ4QplI6dQru2flo+VcFuOZf2/Gqi/e+1WAWGy+yjAIATLRnFZjzEcsoDHRr1K78elY+tGolDlc041iTa924vzMKADDBnnkdmSKfo12+Peg416CDpRepFeC8GPX+89K1RqHdaMEBe3B7iZ8DhUDJjhiOjII3gQIrZKbvMkNSoPDOO+9g9erVuOqqq3D06FGsWLGi0+PZ2dmYNm0avv76a1kGSRA9wSbatg6L23p+d/CXPRoLFNrMwK5z9YgJV+PT+2bh/26/CBFhKsweOghv3HExqv98Az66ZyYSIzXYf74BN735k6j7ZbIj6p/gPswmFZBTetR7oMBkR9OGJHbrc/DwvBFQcBx+OFmF4/bAz1P8tTP78nWT8OiCkTiyYjEuGeoIuG4YnwWlgsPBssZOu6qsqHJ8RjyUCv92bHWnTmGPvVZlzY+nuxWUHyxrAM8LQbjz+8okVOxvc9Qo0C6kM4lRGtw3XQiCvzrvWg7ikMz57737zZzh+H9zhvfaENNTMmIjEK5WwmLjxZ4CPVEta40Ckx4J3/9dJXWw2HgMjo/wS2Gx8857oKxRGWIvBa+kR/Ygzo8ZrmBH0qz9z3/+EyNHjsS6deswZswYhIV138UcMWIECgu9t/xraGjAhg0bsHXr1k7dnwkCEDTTWnuqs0bGrIK/Gq4kRWnEyW1yVjwKnrgCt9h3Zp3Rhqlw+5RsfPfQXESEKbHhVBXu/3APbDbeUchMjkduc/XYTHAcEKZSeG2BxxajzQZTNy9zZ1ghsys5QF5yNK4fL9ROPPLFIY819dsLa3DS3nAvI863O2GLRqbh5esnIanLdyM5OhzzhwmBw2dOhdliozU/FjIz+goUTBYrztYKGQNXBeVd6xMYrCib/W1y7hCHGkvnjQDHAQfrrS5ryQIhPcqIi8Dfbp4im+MRIBglMBlib/Ijo9kqyoTk7aMgnFOUN/qhPgHonEULiYxCm3xBXKggKVA4efIkFi5cCJWq54rwlJQU1NbW9vi4M9u2bcNFF12ExMREzJ07FwUFBQCATz75BDk5OVi8eDEWLlyIrKwsylIQneA4TvxC18hYp+CvjALHcXj3FxfhvmFh2P7b+chN6l2acVF2Ej67fzaUCg7v7zuHJ74uwNGKZgBUyOwJgxMi8fn9s/HZfbO97mIcqVGJN/ze5EddC5m78txV4xGuVmLj6ao+LSWdOVrRhGv+tR1mqw03jM/CKB81CHQHFuR+eshRp8B23f1Zn8Doq5fC2do2WJxqE7oWlO+/IAQC07oEChPtf8uZ2la0GsyotVt8kvSoO3nJ0RiTJnwmd5c2dHvc330UfAmrUyjsJVBgVt5qpUKUDXkDq1Fo7TDDYrU55hk/yI6AztKjQFmjMsQaBS8aWFKNQnckBQoqlQomU+8XorKyElFRfX9oCgsLccUVV2D//v0AgB07duCKK65AQUEB7r33XuTl5eF3v/sdbrrpJrS1teGWW27BmTNnpAybCFGYdKRaxl4K/ipmBoDFo9NxzRD3u5IuHp2BN++4CADw8uZT2HtekE6QNapn3DBhMK4Zl9n3gW7Ql/PR+UYdzje2Q6ngMCPXdYf5EamxeP6q8QCAR/57yKV7UFdKG3S4/LUf0dphxuy8ZHx4z8yAus5dPy4LKgWHIxVNOFPTCp7nHRmFzEBmFFxLQU5UNQMARqbGuCwo31cqfLe6ZhRSY7RIi9GC54HNZ6pg43koOC4kFru+YHq28Jnf2yVQsNpsoq2mPzMKviLfjYLmcnufhbSYcFm+q2wXHRAkcPucuoj7g+DMKMjgekQZBRFJgcLYsWOxdetWWK2ui5P0ej02b96MyZMn93muVatWwWq1YsuWLWhoaMDOnTvR0tKC2267DZdeeikOHDiAV199FZ9++im+/vprmEwm/PWvf5UybCJESfVBLwV/2aNK5Z6L8/DCNRMACB1Gw9VKcTeL8D99FTQzu8LJWQmI0qh7PM/v5g3HrLxk6IwW3PvBnl4tcet1Hbjsta2oajVgTFosvn5wLsID6DgCCJr0S0ekAgDWHjqPC03taNKboFJwGJMW5/fx9CU9OlElyLVm5ibj/ulDATgKymvbOnChSQ+OAyZnJXZ7LpNSfXeiEoDQEdvfNRj9hYuyhfeP9S5hNLSbwPOCrWaiTDalgcQd5yNmPjEqTZ7Mn0qpQEy4MKesP1EJk8WG9FitKIPyNc4L6oDXKNgzNHL0UaCMggNJs9p9992Hs2fP4te//jWMxs6a3NbWVtxzzz2orq7GkiVL+jzXzp07ccUVV2DevHkAgOnTp+PKK69EUVERnn766U5N2y677DLMnj27k8MSQTh6KciTUeB53pFRiA7em9fjC0fht3MF+8HJWQlQKWmREijYTbmngmbRrjC/dzmAUqHAu3dOR0SYEtsKa/DajrMuj2s3WnDl69twtrYNWfER+OF/5weN4xVzP1p76DwK7N2LR6fFBcQ2kQUKFxrbYbV1d6I5bg8URqfF4eF5wzsVlLP6hBEpMYjRdg/umJTqe3ugQNaoPTPdHigcLGuC0ezYYGSFzAkRmpCYv9yRHok9b9Lly7Ax+dGXR4V+FXOGDvJbZjEjVgutWomIMGXAMwqx4cweVVpGwWbjZW2GFypIDhRuu+02vPXWW0hOTsZbb70FAJg2bRoyMjLw+eef4+6778ZNN93U57kuXLiAkSNHdvrdsGHDAACjRnV3JBg3bhwuXLjQ7ffEwEXujEJbhwUmi7CoCNaMAiDUN6y5YTLWPTAb7901PdDDGdD0lVFw6Ib7lgPkJUfjpWsnAgAe/6oAhfZiW0C4kW07W4Or/m8b9p1vQEJEGDb873yfdl/1lGvHZUKtVOB4VQs+2H8OQGBkR4Bg4RqmUsBi40XJhzNMejQ6NRa5SY6C8le3nurUkdkV7G8SHY+oPqFH8pKiEKvmYLLaxJoVwNnSN3g3ZDyBSY/ONbT3aJF6RKwpi5PtdZnz0ZYz/i1kBgSjjY2/mY9Nv1kArZf1Xt7ibTFzo94o1iyFghROLiSH8B999BH+9a9/IScnBxUVFeB5HgcOHMDgwYPx+uuv4+2333brPGFhYTCbO0d/Go3woY+I6H7zi4yMhM3FzhAxcJE7o8BkRxFhSq8LXX2NQsHhhgmD+yyCJnwLq1EodlGjUNmsR1FdGxQch1l5yW6d76HZwzB/WAoMZivu+WA3ztS04slvjyD3ma8w72+bsa2wBlq1Et/+ei5GBrB42RXxERpcNjINALDO3pE3EIXMgPD9GBJvL2iu71yn0GG2oqhOCOzG2Ot7Hl0gbE59eKAU3x6vANC9PoHRtcs0ZRR6huM4DI8Tlhu7SurE3we7xNNTMuMioFEpYLbacKGpe10Mz/M4WskyCnGyvS4rimaW2f4qZGbMyhuEGbnuzW2+JC7Cu4ZrbA2RGKmBOgQyXHLh1TuxZMkSHDlyBDqdDuXl5WhtbcWJEyfw4IMPun2O1NRUVFZWdvrdjBkzsGzZMpfHl5eXIzk58B9IIniQO6PgL2tUInRg0qPyZn0naQXgyCZMyIxHrNY9eZBCweHtOy9GdLgKu0rqMWLlN3juh+M439iOmHA1HpiRhz2PXobpQXBzdsUtkwZ3+jkQ1qiMnuoUztS0wsbziI8IE+eQi3OSMCM3CSaLDQftRdg9BQrZiZGdCklJ09w7I2IF6dmuc45AgblFhcrurWCRyuoUumcXzze2o63DArVSgREyBvgJTrLDlOhwWW1f+xPs+yi1jwI5HrlGlpBJq9UiPT3dLZejrkyYMAF79+7t9LuFCxfi5Zdfdnn80aNHXUqSiIGLI6MgV6AQOnZ9hH8YFB2OyDAVeB4o7dJsqbf+Cb0xJCEKf7lxCgBAwXG4fFQaPr5nJqr/fAPeuOPioLbDvXZsFjQq4fbCcYFtBtiTRepxJjtKi+2k52ZZBUCwsOxp7BzHdQqAKKPQO8Pj7IFCSb1oQcvm2kEhNNeKdQpOkkEGs7IelRor6461c6BwiR/rE4IN0R7Vy4wC1Sd0JuC5lbvvvhtXXXVVn3arAHDw4EEcP35cLHz2hIqKCvzlL3/BokWLMHjwYISFhSE1NRU33nhjt0CF0draimXLlmHIkCHQaDTIzs7GY489Bp2ub9tCwn+w6L+mrcPjRlWuoIwC4Skcx4l1Cs4WqU16IzacrgLQdyGzK+6bnocDyy9H+XPX4/v/mY/bpmQHXAfsDjFaNS4flQ5A0G1Hh/fs9ORrcpJcZxSY49HoLju714zNEDNE49LjenWSmuQkP6Iahd4ZGqOASsGhqtUgynJYjUIozbWiRaqLeiVRdiRzc8wEJ8eoS/wsOwomvLVHpYyCa9y+4+Tm5np8co7jUFxc3Osxl19+OS6//HK3zjd58mTJ9Ql///vfsWrVKuTl5WHRokVITk5GYWEhvvzyS3z55Zf46KOPcOutt4rHt7e3Y86cOTh8+DAWLVqE22+/HQUFBVi9ejW2b9+OHTt2IDycPkzBQEq0cIM2WmxoMZi9dn+hjAIhhdykKByrbBYLmneX1OG2d37GhSY9YrVqtwqZXTG5h2LaYOfBmUPx1dFyXDk6I6Dj6KmXAgsUxnTRiisVCjx5+Vjc88FuXDWm97FTRsF9NEoOEzLjceBCI3aV1GNIQhRqA9CV2dcMTe65O7PYHFPmnjfOjdv8XZ8QTLBAocNshdFs9dhpTcwoRNN32Rm3A4XS0lIolcpeuzEHM9OmTcO2bdswZ86cTr//6aefsGDBAjz00EO47rrrxELql156CYcPH8bjjz+OF198UTz+iSeewKpVq7BmzRqsWLHCr38D4ZpwtRKxWjVaDGZUtRq8DxTaKKNAeA7bhS6qa8OqTSfwh2+OwGrjMTQ5Gp/eNyto7Ev9xRWjM3D+T9cFfHeupxqFE07So67cfXEuLhk6CFnxvbtJORdpk1yhb6ZnJ9oDhTrcPiXbIT0KqUChZ4vUo/YeCnLLBpn0KCEiDKMD0K8kWIgJV4PjhN5CLR1mDPIwUKAeCq7xWHo0d+5cvPfee2htbYXBYOjzX7Bwww03dAsSAGD27NmYN28empqacOzYMQCCM8Gbb76JqKgoPPnkk52Of/LJJxEVFYU333zTL+Mm3IMVkBXYCxC9QcwohIhlH+Ef2GfwtR1n8cRXh2G18bhjSjYOPX5FwFx/As3ghEi3O477ClajUNliQIe90FxvsqDEHjj0tLDKSYrq09t/2KBoDE2ORlZ8BDL7CCqI7o3XHNKj0Jlr8wexLu26Tr079CYLCmuF4EFu6dGkrAQoFRxumjgYCsXArE8AhGLyaHtDy2a953UKVKPgGrcDhZMnT+J3v/sdDh8+jNtuuw3p6elYunSpuLiWwuWXX479+/dLem57eztefPFFvPbaa5Jfn6FWCx8sli0pLCxEZWUlZs6cicjIzg1EIiMjMXPmTJSUlKCsrMzr1ybkYcFwQdax6XS11+eiGgVCCqxGwWLjoVUr8dYvLsYHd88IqD6fEKwOozTC3H7eXmh+qroFPA8kRWm82s1WKhQ49vsrceqPV5OdohtMz04CAByuaEK70RJyrkeAYJEaxixSGx29O05UtcDG80iO0iBF5r93YlYCal+4Ea/fOk3W8/ZH4rwoaGY1CnJfn/6O2zPbiBEjsHr1apSXl2PdunWYPn06XnvtNUyYMAFTpkzB66+/jpaWFo9evK6uDhdffDHmzZuHd955x63n79mzB7/5zW8wZMgQrFy5Eikp3jUWuXDhAjZv3oy0tDSMHTsWgBAoAEB+fr7L57Dfs+OIwLNwuODbvulMldcFzVSjQEhh2pBEJEVpMDY9DvuXX477pucNWPeRYILjuG7yI7E+QQaZRrhaiUhN/5Tk+pus+AhkxkXAauOxq6ROXMwNCqFNGaVC0UmGyGAdmcdnxPtkXkiI1AzobAKDSTylWKRWt7GMQuh8HuXA49lNqVTiuuuuw3XXXYeamhq8++67ePfdd/G///u/ePTRR3H99dfjz3/+MwYPHtznuQ4ePIj//Oc/ePbZZ3H//fdjyZIlGD58OCZPnoyUlBTExcWho6MDjY2NOHPmDA4cOIC2tjYolUrcdttteO6559x6nZ4wm8345S9/CaPRiFWrVkGpFFLkLGCJjXXtcxwTE9PpOFcYjUYYjUbx59bWVvE1uzaY8zXs9fz9uv7kosFx0KgUqGg24FhFI0Z64SNdZ58s4sNVfnnPBsL16c+4e32iwxQoffoqqJUKcBxH19OP9HWNhiRE4FhlM4pqW2DOT8bRCkGiODIlmq6TH3C+PhcNSUB5sx5fHrkAAFAqOESpQ+v7kpsYhVPVrThT04y5Q+1ZFLssdkxaTND9raF0D4qxB+31OkOnv6ewrg3z/rYVD8zIwzNXjOn2PLPVhnq7miAxwj/3fk+Q+xp5ch6vtkFSUlLw+OOP4/HHH8eWLVtwzz334OOPP8Ytt9zi9gL+7rvvxl133YX169fjnXfewbZt2/DBBx90O06hUGDcuHG4/vrr8cADDyAtLc2bocNms+Gee+7Bjh07sGTJEvzyl7/06nxdeeGFF/Dss892+/3GjRtddpz2B5s2bQrI6/qLETEcjjQCf/vvVlw9WHrhaHWLkC4+fmA3Gk76T04Q6tenv0PXJ/jp8Rq1CguALfuPIavlLLYfFSQGtvoyrF/vvVyRcI9NmzYh1p5FWHtAcESMVgE//PB9IIclO0q7pGrTPuHzBgDbjts/c3XnsX59ZY/PDSShMMcZ7fKhn/cdQkSlQxr/cbERtToz3v7pFKbxF7o9r6FDqCdRcMDe7VugCNJssFzXSK/X932QHa/zpfv378fbb7+NTz75BC0tLcjIyEBmZqZH5+A4DldeeSWuvPJKAMCpU6dQXl6OhoYGaLVaJCcnY/To0T3u8HuKzWbDfffdh48++gh33nkn/u///q/T4+x1esoYsOxAb+NZsWJFp+7Sra2tyMrKwqJFi8SMhL8wm83YtGkTFi5cKNZjhCInNadx5NujqFImYvHi2ZLOoTdZYNz0BQDg5qsuQ4wf9OUD5fr0V+j6BD99XaOiyLP49sJhKOJSsHjxDPzuwLcA9Ljl0hmYnRec3a1DCefrk1zZirfPbkF9hyARzUiMweLFlwV4hPJSFluEry8cgjU6GYsXzwLP87hv51cArPjF5ZdgYmZwNUsMpTnus6a92F9/HllDh2Px/BHi71/86xYADajtAOYvvKxbf5RDZY3AT5uRGqPFVfa1aDAh9zVi61h3kBQo1NfX4/3338c777yDEydOQKVS4eqrr8b999+Pyy67DAqFd7uwI0eOxMiRI706R0/YbDbce++9eO+993D77bfj3Xff7TbevmoQ+qphAACNRiNarTqjVqsD9kUM5Gv7gyvGZOD33x7F9qI68JxCkttKs30nKEylQEKU1q8a81C/Pv0duj7BT0/XaOggYXPmfKMeRhtwvknYTRufmUjX1I+o1WpMyU5GuFopOlClRIeH3DUYnhIHACiu10GtVqO8SY9GvQlKBYdxmYlQe2jb6S9CYY5LiBTqC3Qmq/i3tBhM2H9BkH7ZeB6lTQaM7WJRW6+3ABAcj4L5PZDrGnlyDrdX9DabDd9++y1uuOEGZGRk4JFHHgEAvPLKK6ioqMDnn3+OK664wusgwZc4Bwm33nor3n//fbEuwZn8/Hykp6dj586daG/v3KSnvb0dO3fuRE5ODrKysvw1dMINxqXHIzlKg3aTBXtK6yWdw9FDQUOFqAQRIjgXM5+0FzKnxoQjkQwL/E6YSokpgx12waHkeMQQLVIbBItU1pF5+KCYXjt9E94TK7oeOTT4P56tgdXmMDk5VdN9N72GCpl7xO1VfWZmJq699lps27YN999/P/bt24ejR4/i4YcfRlJSki/HKAtMbvTee+/h5ptvxgcffOAySAAEKdQDDzwAnU6HlStXdnps5cqV0Ol0WLJkiT+GTXiAQsHh0uGpAKTbpDocj2iyIIhQgQUKjXqT6OE/kBtTBZoZOQ65VyjOtVnxgkWqyWJDWZPe0ZFZ5v4JRHdYd2Zne9RNp6s6HXPaRaBA1qg947b0qLq6Gmq1GuPHj0dpaSmeeuqpPp/DcRy+++47rwYoF3/605/wn//8B1FRURg2bBiee+65bsdcd911mDBhAgBg+fLl+Oqrr7Bq1SoUFBRg0qRJOHToEDZu3IipU6fi4Ycf9u8fQLjFwhFp+PjgeWw8VYWVV433+PmOHgq000gQoUJ0uBqJkRo0tBvx7fEKAK47MhP+YXqOY3NxUAg2tlQqFMhNjMLpmlYU1bV1skYlfIvYR8Gp4dpG+8bhzNxk7Cypw6nq7vWn1GytZzyqUTCbzdi+fbvbxweTdKO0tBQAoNPp8Pzzz7s8Jjs7WwwUIiMjsX37djzzzDNYt24d/n979x4dZXXvf/wzuU3uCSSQi80VQagVEQliw03QQNElHnN6uPxCSUvQYm3rgUrQI43Y1ksXSFvr0iKnglKP2kKp9XJASwKSQ4FCQGoVAkLAC4jcckGSSbJ/f4QZGJ5MEjCZGSbv11pZyzzPnpk9fM3MfGbv/ezS0lKlpKRozpw5KikpUUQE/zP5o1v6t1wN6x8Hj+vE6Xr1iLy4NyE2WwMCU1ZClI7V1Wv93s8ldc4eCrg0bkEhQF9rr+wVcy4ofHpSEiMK3uAcUTh1pmXq0YFjtdp7tEbBQTbNGtH3bFDwPKLA1COrDgeF/fv3d2U/upxzv4eLERcXp8WLF2vx4sVd0yl0uq/1iNSA5Fh9cLha63YfUf51F7fPhnPqUSIjCkBAyUqI1j8OHpejqeUyiIwo+E5SbIT6JEZr3xe1SgrQD2Z9e7WsU/jnZ6dcU10GpjKi0NXiLhhRcE5DHpaZqBvO7gy++/NqNTcbtw3qzgUFvgS+UIeDQkZGRlf2A+g0t/RP0QeHq/X2h59dQlBg6hEQiJzrFJy+nkxQ8KVFdw7W6p0fu0aBA82VZ4PCX3d9rKZmox6RYboing+hXe3cGoWWEQXn+oRb+icrKyFKYSFBOuNoUtXxOmUlnntNODf1KDCD61fhv5coAi7RLc4FzbsvfkGzc1dmph4BgSUrIcr131fERyg+8tI3ZcRXN3Fgmp6fdqMiw77ydk5+yTmicPDspXgHpsb71XTsQOX8uz75ZYOampv1t7OfA27pn6LgoCD1O1uXD464r1M4XHN2RCGGMHchggICzqi+SQoJsumjL2q172jNRd2WEQUgMJ0/osD6BHQ154iC00AWMntF3NlNUmvrG7W16riOn25QbHiohmYkSJIGnB1J/PC8dQqnGxpVc+bcPgpwR1BAwIkJD3Utlrvwsmjt4fKoQGA6PyiwPgFdLa1HpEKDz33EYiGzdzjXKEjSnyoOSpJu6pekkLO1cAaF80cUjpyddhQeGqyY8MAc4foqCAoISHkDWua9Xux+CowoAIEpo2eUnDM/2EMBXS0kOEjZ582B59Ko3hEWEqzIsJY9sv60oyUo5J23DqZ/Ussu7edf+ej8Kx4xPcyKoICA5Fwgt27PETU1N3foNvWOJlWfvaQaIwpAYLGHBiv77KjCdV/jQxu63pVng4LNxiiWNzkXNFcdr5PUspDZaUCyMyickjEtuzUfdu7KzPqEVhEUEJCGpPdUfESYTn7ZoH8cPN6h23xR1zKaEBxkUw8WOgIB5w+FuXq+YJiuS+vp666gG+jbu+VDad9eMQG7aNsfnT/9KKNnlNt6kat6x8pma9ml/YuzMwjYQ6FtBAUEpOCgII3plyRJKt1zpEO3ca5PSIiyu11fGUBguCEzUYXD+vi6G+gmBqbGS5JrIS28wzmiILWMJpw/nSgiLESZPVtGepzrFNiVuW1EXASsYVmJWrXzkLZWHetQ+6M1rE8AAHSOqUMyFRYSpLFXJbffGJ3m/KCQ18o+Hf2TYrX/WK0+OFytkVcmMaLQDkYUELBy0lu+xdl6sINBgSseAQA6iT00WP8vJ4tvqr3MOfXIZpPG9LOGNOc6BeeO2YwotI2ggIB1fXpP2WzSoROndeTsNwZt4YpHAABc3pwjCten9VRCK+/nrkukHnZOPWr5fJAUw5eErSEoIGDFhIdqQFLLC0JHph+dG1EgKAAAcDly7or9b9emtXr+3CVSW4LCkRpGFNrCGgUEtJyMBP3r8CltqTqm2675Wpttz40o8K0CAACXox+Nvko3ZiVqiIdF5M4vEA+eOK26+kbWKLSDEQUEtJz0lssgdmSdAlOPAAC4vIUEB+nG7F5uO2OfLyHa7nqf33zgC9U3tuy1xNSj1hEUENCGZiZKkrZWHXdtruLJF86pR7xYAAAQsJzTj0orWy6fHhcRqgj2umgVQQEBbWBqvEKDg3Ssrl4HjtW12ZYRBQAAAp9zQXPZ2aDA+gTPCAoIaPbQYF17RbwkaUvVF222ZY0CAACBzxkUNh9omZaczEwCjwgKCHg5Zxc0ba067rFNU3Ozjp9mRAEAgEDnnHrkaGpZn8CIgmcEBQS8oRntb7x2rK5BziUMCVEEBQAAApVz0zWnJK545BFBAQHPOaKw7eBxNTU3t9rGuYdCz8gwhXi4UgIAALj8pcVHKTIs2PU7U4884xMRAl7/pFhFhYWorqFRHxyubrXN0Zqz0454sQAAIKAFBdnU/+x+ChJTj9pCUEDACw4K0vXO/RQ87NDMrswAAHQfznUKEputtYWggG6hvXUKXPEIAIDu4/x1CowoeEZQQLdw7spHjCgAANDdDWDqUYcQFNAt5KS3BIWdn5xUvaPJcp7N1gAA6D6ceykE2Wy897eB/arRLWQmRCkhyq5jdfXa+ckJDc1MdDt/bkSBqUcAAAS6AcmxKvpmH6XGRXK1wzYQFNAt2Gw2Dc1I0Fv/+lRbq461EhQYUQAAoLuw2Wx6buowX3fD7xGh0G041ylsaWWdwtEaRhQAAADOR1BAt5HjvERqK1c+co0oxDCiAAAAIBEU0I04RxQ+PFKtmjMO1/HmZqMv6rg8KgAAwPkICug2kmIjlN4jUsZI2w4edx0/+WWDmpqNJCkxihEFAAAAicXM6GZyMhJ08MRp/XzNP/XXf34sSTr1ZcvoQkx4iOyhwb7sHgAAgN8gKKBb+WZWL63ccUh/231Yf9t92O1ceo8oH/UKAADA/xAU0K3cNfxKNTQ16+SXDW7HbZLuHJTum04BAAD4IYICupVoe6jm5V3t624AAAD4PRYzAwAAALAgKAAAAACwICgAAAAAsCAoAAAAALAgKAAAAACwICgAAAAAsCAoAAAAALAgKAAAAACw6FZBYcWKFbr77rs1ZMgQ2e122Ww2LVu2zGP76upqzZ49WxkZGbLb7crMzNT999+v2tpa73UaAAAA8IFutTPzQw89pKqqKiUmJiolJUVVVVUe29bV1WnUqFHasWOH8vLyNGXKFFVUVGjhwoVav369NmzYoPDwcC/2HgAAAPCebjWisHTpUh04cEBHjx7V97///Tbb/vKXv9SOHTtUXFysNWvW6PHHH9eaNWtUXFysrVu3avHixV7qNQAAAOB93Soo3HzzzcrIyGi3nTFGS5cuVXR0tObPn+92bv78+YqOjtbSpUu7qpsAAACAz3WroNBRlZWV+vTTT5Wbm6uoqCi3c1FRUcrNzdVHH32kQ4cO+aiHAAAAQNciKLSisrJSktS3b99WzzuPO9sBAAAAgaZbLWbuqFOnTkmS4uLiWj0fGxvr1q419fX1qq+vd/1eXV0tSXI4HHI4HJ3V1Q5xPp63HxcdQ338G/Xxf9TIv1Ef/0Z9/F9n1+hi7oeg0EUee+wxLViwwHJ87dq1ioyM9EGPpLffftsnj4uOoT7+jfr4P2rk36iPf6M+/q+zanT69OkOtyUotMI5kuBpxMA5OuBpxEGSHnjgAc2ePdvtNmlpacrLy3ONSHiLw+HQ22+/rVtuuUWhoaFefWy0j/r4N+rj/6iRf6M+/o36+L/OrpHzc2xHEBRa0d4ahPbWMEiS3W6X3W63HA8NDfXZH6IvHxvtoz7+jfr4P2rk36iPf6M+/q+zanQx98Fi5lb07dtXqampKi8vV11dndu5uro6lZeXKysrS2lpaT7qIQAAANC1CAqtsNlsKioqUm1trX72s5+5nfvZz36m2tpazZw500e9AwAAALpet5p6tHTpUm3cuFGStGvXLtexsrIySdLw4cNVVFQkSZo7d67+8pe/6IknnlBFRYUGDx6s7du3a+3atcrJydF9993ni6cAAAAAeEW3CgobN27U8uXL3Y6Vl5ervLzc9bszKERFRWn9+vV6+OGHtXLlSpWWliolJUVz5sxRSUmJIiIivNp3AAAAwJu6VVBYtmyZli1b1uH2cXFxWrx4sRYvXtx1nQIAAAD8EGsUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFNqxdetWTZgwQfHx8YqKitKwYcP06quv+rpbAAAAQJcK8XUH/FlpaanGjRun8PBwTZ48WTExMVq5cqUmTZqkQ4cOac6cOb7uIgAAANAlGFHwoLGxUTNnzlRQUJA2bNigJUuWaNGiRdq5c6f69eunBx98UFVVVb7uJgAAANAlCAoerFu3Tvv27dPUqVM1aNAg1/G4uDg9+OCDamho0PLly33XQQAAAKALERQ8KCsrkyTl5eVZzo0bN06StH79em92CQAAAPAagoIHlZWVkqS+fftaziUnJys6OtrVBgAAAAg0LGb24NSpU5Japhq1JjY21tWmNfX19aqvr7fc3/Hjx+VwODqxp+1zOBw6ffq0jh07ptDQUK8+NtpHffwb9fF/1Mi/UR//Rn38X2fXqKamRpJkjGm3LUGhizz22GNasGCB5XhWVpYPegMAAACcU1NT4/ELcSeCggfOfzhPowbV1dXq0aOHx9s/8MADmj17tuv35uZmHT9+XAkJCbLZbJ3b2XZUV1crLS1Nhw4dUmxsrFcfG+2jPv6N+vg/auTfqI9/oz7+r7NrZIxRTU2NUlNT221LUPDAuTahsrJS119/vdu5w4cPq7a2VkOHDvV4e7vdLrvd7nYsPj6+0/t5MWJjY3kR8GPUx79RH/9Hjfwb9fFv1Mf/dWaN2htJcGIxswejRo2SJK1du9Zybs2aNW5tAAAAgEBDUPBg7Nixys7O1ksvvaQdO3a4jp86dUqPPvqowsLC9J3vfMd3HQQAAAC6EFOPPAgJCdHSpUs1btw4jRw5UpMnT1ZMTIxWrlypqqoqLVy4UJmZmb7uZofY7XaVlJRYpkLBP1Af/0Z9/B818m/Ux79RH//nyxrZTEeujdSNbdmyRSUlJfq///s/ORwOXXPNNZo9e7YmTZrk664BAAAAXYagAAAAAMCCNQoAAAAALAgKAAAAACwICgFs69atmjBhguLj4xUVFaVhw4bp1Vdf9XW3uo1PPvlEv/rVr5SXl6f09HSFhYUpOTlZ+fn52rx5c6u3qa6u1uzZs5WRkSG73a7MzEzdf//9qq2t9XLvu68nnnhCNptNNptNf//73y3nqZFv/PnPf9Ytt9yihIQEhYeHKysrS1OmTNGhQ4fc2lEf7zLGaNWqVbrpppuUkpKiyMhIXXXVVbr77rv10UcfWdpTn66xYsUK3X333RoyZIjsdrtsNpuWLVvmsf3F1qG5uVlPPfWUrrnmGkVERKhXr16aMmVKqzWGVUfr43A4tHLlSk2fPl0DBgxQdHS0YmJidMMNN+iZZ55RU1OTx8f4wx/+oKFDhyoqKko9evTQbbfdpu3bt3/1zhsEpHXr1pnQ0FATExNjZs6caWbPnm0yMjKMJLNw4UJfd69bKC4uNpJMnz59zIwZM8y8efNMfn6+CQ4ONkFBQebll192a19bW2sGDRpkJJm8vDxTXFxs8vLyjCSTk5NjvvzySx89k+5j165dxm63m6ioKCPJbNq0ye08NfK+5uZmc9ddd7n+lu655x5TXFxspk2bZtLT0827777rakt9vG/27NlGkklJSTHf//73zdy5c824ceOMzWYzMTExZteuXa621KfrON/fExMTXf/9/PPPt9r2UupQVFRkJJmrr77azJ071xQUFJiwsDDTs2dPs2fPni5+dpe/jtbngw8+MJJMdHS0mThxopk7d665++67TWpqqpFkbrvtNtPc3Gy53c9//nMjyWRkZJjZs2ebmTNnmpiYGGO3283GjRu/Ut8JCgHI4XCYPn36GLvdbioqKlzHT548afr162fCwsLMgQMHfNfBbmLlypWmrKzMcnzDhg0mNDTU9OjRw5w5c8Z1/Kc//amRZIqLi93aOwPHo48+2uV97s4aGhrM4MGDzQ033GAKCgpaDQrUyPt+9atfGUnmnnvuMY2NjZbzDofD9d/Ux7s+++wzExQUZDIyMszJkyfdzj355JNGkvnud7/rOkZ9us7bb7/tel9/7LHH2gwKF1uHdevWGUlm5MiRpr6+3nX8zTffdIUNtK2j9fn444/N008/bWpra92O19bWmiFDhhhJ5tVXX3U7t2fPHhMSEmL69evn9ndYUVFh7Ha7GTBggGlqarrkvhMUAtCaNWssL9BOy5YtM5LMggULfNAzODm/vdm6dasxpuVb09TUVBMdHd3qC0R0dLTJzs72RVe7jZKSEmO32837779vpk+fbgkK1Mj7Tp8+bXr06GGys7PdAkFrqI/3bdq0yUgyU6dOtZzbs2eP6xtQY6iPN7X1QfRS6jBlyhQjyaxfv95yf6NHjzaSTFVVVac+h0DWXpDz5KWXXjKSzA9+8AO34w888ICRZJYvX265TWFhocfadRRrFAJQWVmZJCkvL89ybty4cZKk9evXe7NLuEBoaKiklo39JKmyslKffvqpcnNzFRUV5dY2KipKubm5+uijjyzzsdE5tm/frl/84hcqKSnR17/+9VbbUCPvW7t2rU6cOKE77rhDTU1NWrVqlR5//HE9++yz2rt3r1tb6uN9ffv2VVhYmMrLy1VdXe127vXXX5ckjR07VhL18ReXUoeysjLXuQvxmcJ7Lvzc4NTVn/kICgGosrJSUsuL+IWSk5MVHR3tagPvO3jwoN555x2lpKTommuukdR2zc4/Tt06X319vb7zne9o0KBBmjt3rsd21Mj7tm3bJkkKDg7WwIEDlZ+frwceeECzZs3SVVddpZ/85CeuttTH+xISEvT444/r4MGD6t+/v2bNmqXi4mKNHz9excXFuueee3TvvfdKoj7+4mLrUFdXp88++0xZWVkKDg5utz26zu9//3tJ1kBQWVmp6OhoJScnW27TGfUJab8JLjenTp2SJMXFxbV6PjY21tUG3uVwODRt2jTV19friSeecL3wdqRm57dD5/npT3+qyspKbdu2rdU3Qidq5H2ff/65JOnJJ5/U4MGDtWXLFg0YMEAVFRW66667tGjRIvXp00ezZs2iPj7yn//5n7riiitUVFSkZ5991nV8+PDhmjp1quvbT+rjHy62DtTNPyxZskRvvfWWxowZowkTJridO3XqlHr37t3q7TqjPowoAF7S3NyswsJCbdiwQTNnztS0adN83aVub9OmTVq4cKEeeughfeMb3/B1d3CB5uZmSVJYWJhWr16tnJwcRUdHa8SIEfrjH/+ooKAgLVq0yMe97N4eeeQRFRQU6MEHH9ShQ4dUU1Ojd999V2fOnNHo0aP12muv+bqLwGXt9ddf17333quMjAytWLHC649PUAhAzuTvKUFWV1d7/HYAXaO5uVnf+9739NJLL6mgoMDtmzepYzU7vx2+usbGRk2fPl0DBw7UvHnz2m1PjbzP+W85ZMgQpaamup37xje+oezsbO3bt08nT56kPj7wzjvvqKSkRPfee6/mzZunr33ta4qOjtbw4cP117/+VaGhoZozZ44k/n78xcXWgbr51ptvvql///d/V1JSktatW6eUlBRLm7i4uC6tD0EhALU1J+3w4cOqra31OD8Rna+5uVnf/e53tXz5ck2ZMkXLli1TUJD7n1578wjbm1eKi1dbW6vKykrt2LFDYWFhrk3WbDabli9fLkm68cYbZbPZtHr1amrkA1dddZUkKT4+vtXzzuNffvkl9fGBt956S5J00003Wc4lJyerf//+2rt3r9t7DvXxrYutQ1RUlFJSUrR///5WN/uibl3njTfe0J133qnExESVlpYqOzu71XZ9+/ZVbW2tDh8+bDnXGfUhKASgUaNGSWq5YsiF1qxZ49YGXcsZEl544QVNmjRJL774oscFYampqSovL1ddXZ3bubq6OpWXlysrK0tpaWne6nrAs9vtmjFjRqs/zhfV22+/XTNmzFBmZiY18gHnB9APPvjAcs7hcGjv3r2KiopSr169qI8PNDQ0SJKOHj3a6vmjR48qKChIoaGh1MdPXEodRo0a5Tp3IednipEjR3Ztx7uZN954Q/n5+erZs6dKS0t15ZVXemzb5Z/5LvnCqvBbDofDZGdnt7nh2v79+33Wv+6iqanJdT3+b3/72+1eB57NiPxHa/soGEONfMG558hzzz3ndvyRRx4xkkxBQYHrGPXxrv/5n/9x7dZ74YZrzzzzjJFkcnNzXceoj3ew4Zp/a68+b775prHb7SY5Odl8+OGH7d7f7t27u3TDNZsxxlx6zIC/Ki0t1bhx4xQeHq7JkycrJiZGK1euVFVVlRYuXOiaN4qu8/DDD2vBggWKjo7Wj3/8Y8u1jyXpjjvu0KBBgyS1fJuTm5urnTt3Ki8vT4MHD9b27du1du1a5eTkaP369YqIiPDys+ieCgsLtXz5cm3atEnDhg1zHadG3rdv3z5985vf1Oeff65bb71V/fv3V0VFhdatW6eMjAz9/e9/d10WkPp4V1NTk8aMGaMNGzaod+/euv322xUfH6/t27dr3bp1ioiIUFlZmYYOHSqJ+nSlpUuXauPGjZKkXbt2afv27crNzXV9Ez18+HAVFRVJurQ6zJw5U0uXLtXVV1+tW2+9VZ999pleeeUVRUdHa9OmTerXr593n/BlpqP1+fDDDzVo0CDV19dr8uTJrumX58vMzFRhYaHbsV/84hd66KGHlJGRofz8fNXU1Ojll19WQ0OD/va3v7W6B0aHXXLEgN/bvHmzGT9+vImNjTURERFm6NCh5uWXX/Z1t7oN57fSbf1c+I3CyZMnzX333WfS0tJMaGioSU9PN3PmzDHV1dW+eRLdlKcRBWOokS8cPHjQFBYWmuTkZBMaGmrS0tLMD37wA3PkyBFLW+rjXWfOnDGPPfaYue6660xkZKQJCQkxV1xxhSkoKDD/+te/LO2pT9do7/1m+vTpbu0vtg5NTU3m17/+tbn66quN3W43CQkJZtKkSWbv3r1eeHaXv47Wp7S0tN3PDaNGjWr1MVasWGGGDBliIiIiTFxcnJkwYYLZtm3bV+47IwoAAAAALFjMDAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAAAAAMCCoAAAAADAgqAAAAAAwIKgAADwmYcfflg2m01lZWW+7ooyMzOVmZnp624AgN8gKAAAPDpw4IBsNlubP3y4BoDAFOLrDgAA/F+fPn1UUFDQ6rn4+PhLvt97771XkydPVnp6+iXfBwCgaxAUAADtuvLKK/Xwww93+v0mJiYqMTGx0+8XAPDVMfUIANBpbDabRo8erY8//lhTpkxRYmKiIiMjlZubq3feecfS3tMahdLSUn3rW99Samqq7Ha7kpKSNGLECC1ZssRyH+Xl5br11lvVs2dPhYeHq3///iopKdHp06db7eNf/vIX5eTkKCIiQklJSZo5c6ZOnDjh8Tk1NDToySef1ODBgxUVFaWYmBiNGDFCr7322sX94wDAZYagAADoVCdOnFBubq4qKytVVFSkKVOmaOfOnRo/frxWr17d7u3feOMNjR07Vps3b9a4ceM0Z84c3X777aqvr9eLL77o1vaPf/yjRo0apbKyMt1xxx267777FBkZqUceeURjxozRmTNn3Nq/8MILuuOOO7Rnzx5NmzZN06dPV3l5uW6++WY1NDRY+lJfX+/qgzFGM2bMUEFBgaqqqjRx4kT99re//Ur/VgDgz2zGGOPrTgAA/NOBAweUlZXV5hqFYcOGafz48ZJaRhQkaerUqVqxYoXr9/fee085OTmKi4tTVVWVIiIiJLWMKCxYsEClpaUaPXq0JCk/P1+rVq3Sjh07dO2117o91rFjx5SQkCBJqq6uVnp6us6cOaMtW7Zo4MCBkqTm5mZNnTpVr7zyih555BHNnz/f1T4tLU1NTU3avn27+vXrJ0lyOBy6+eabtWHDBmVkZOjAgQOux/uv//ovPfroo5o/f74WLFjgej41NTUaM2aM3nvvPe3fv1+pqalf6d8ZAPySAQDAg/379xtJbf78+Mc/drWXZIKDg82BAwcs9zVjxgwjyfzpT39yHSspKTGSTGlpqevYnXfeaSSZ3bt3t9m3F154wUgys2bNspyrqqoyISEhJjs723Vs+fLlRpL54Q9/aGn/7rvvGkkmIyPDdaypqcn06NHD9OnTxzQ3N1tu89prrxlJ5qmnnmqznwBwuWIxMwCgXePGjdP//u//dqhtenq6MjIyLMdHjBih//7v/1ZFRYXy8/M93n7y5MlatWqVhg0bpqlTp2rs2LEaMWKEZdFzRUWFJLlGIi7sQ3Z2tvbs2aOamhrFxMRo586drn5c6MYbb1RIiPtb4u7du3XixAmlpqZqwYIFltscPXpUkvThhx96fC4AcDkjKAAAOlVSUlKbx0+dOtXm7b/97W9r9erVevLJJ/Xss8/q6aefls1m00033aRFixZp0KBBklqmErX1eCkpKdqzZ4+qq6sVExPjetzevXtb2gYHB7umNDkdP35ckvT+++/r/fff99jfurq6Np8PAFyuWMwMAOhUR44cafN4XFxcu/cxceJErV+/XidOnNBbb72loqIilZWVafz48Tp58qQkKTY2ts3HO3z4sFs75+N+/vnnlrZNTU06duyY2zHn7fLz82WM8fjz/PPPt/t8AOByRFAAAHSqgwcPqqqqynL83XfflSRdd911Hb6vmJgYjR8/XkuWLFFhYaGOHDmizZs3u93PhZdWlaRDhw5p3759ys7OVkxMjCS5FkY7+3G+TZs2qbGx0e3YgAEDFBsbq3/84x9yOBwd7jMABAqCAgCgUzU1NenBBx+UOe+ieu+9955efPFF9erVSxMmTGjz9hs2bFBTU5PluHMkIDw8XFLLqENcXJyef/55t6lBxhgVFxersbFRhYWFruMTJ05UbGysfv/732vPnj2u4w6HQw899JDl8UJCQjRr1ixVVVXpJz/5Sath4Z///GerIxQAEAhYowAAaNfevXvb3Jl53rx5rg/wAwcO1MaNG5WTk6Obb75ZR48e1SuvvKLGxkYtWbLEdWlUT370ox/p008/1fDhw5WZmSmbzaaNGzdqy5YtGjZsmIYPHy6pZWrQc889pylTpuiGG27QpEmT1KtXL73zzjvatm2bhg4dqvvvv991v3FxcfrNb36jwsJC5eTkaPLkyYqLi9Prr7+uiIgIpaSkWPqyYMECbd++Xb/5zW/0xhtvaOTIkerdu7c++eQT7dq1Szt37tSmTZtaXfcAAJc9n11vCQDg9zpyeVRJ5sSJE8aYlsujjho1yhw6dMhMmjTJ9OzZ04SHh5sbb7zRrF271nL/rV0e9eWXXzb/8R//Yfr06WMiIyNNXFycufbaa80TTzxhampqLPexYcMG861vfcvEx8ebsLAw069fPzN//nxTW1vb6nP685//bK6//npjt9tN7969TVFRkTl+/LjJyMhwuzyqU2Njo/nd735ncnNzTWxsrLHb7SY9Pd2MHz/ePPPMMx4fBwAud2y4BgDoNDabzbVTMgDg8sYaBQAAAAAWBAUAAAAAFgQFAAAAABZc9QgA0GlY9gYAgYMRBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABYEBQAAAAAWBAUAAAAAFgQFAAAAABb/H6VxntX0MhedAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x450 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Make plots\n",
    "figs_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/\"\n",
    "model_name = 'result_advantage_critic_greedy'\n",
    "kernel_size = 3\n",
    "fig, ax = plot_episode_rewards(x, kernel_size, save_path=os.path.join(figs_path, f\"{model_name}_rewards.png\"))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savetxt('/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_random_results', rewards, fmt='%d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.loadtxt('/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_random_results', dtype=float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "Critic loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random.pth\n",
      "BackBone loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random.pth\n",
      "Actor loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random.pth\n",
      "Number of trainable parameters in the Critic model: 15927297\n",
      "Number of trainable parameters in the BackBone model: 7474272\n",
      "Number of trainable parameters in the Actor model: 18055257\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[7], line 133\u001b[0m\n\u001b[1;32m    130\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNumber of trainable parameters in the Actor model: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mactor_num_parameters\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    132\u001b[0m \u001b[38;5;66;03m# Training\u001b[39;00m\n\u001b[0;32m--> 133\u001b[0m rewards, losses \u001b[38;5;241m=\u001b[39m \u001b[43mlearning_system\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain\u001b[49m\u001b[43m(\u001b[49m\u001b[43msampling_prob_scale\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msampling_priority_scale\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\n\u001b[1;32m    134\u001b[0m \u001b[43m                                        \u001b[49m\u001b[43mreplace_samples\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreplace_samples\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\n\u001b[1;32m    135\u001b[0m \u001b[43m                                        \u001b[49m\u001b[43mpriority_offset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpriority_offset\u001b[49m\u001b[43m,\u001b[49m\u001b[43m  \u001b[49m\n\u001b[1;32m    136\u001b[0m \u001b[43m                                        \u001b[49m\u001b[43mimportance_sampling_beta\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mimportance_sampling_beta_start\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    138\u001b[0m \u001b[38;5;66;03m#Save Model\u001b[39;00m\n\u001b[1;32m    139\u001b[0m learning_system\u001b[38;5;241m.\u001b[39msave(critic_save_path, backbone_save_path, actor_save_path)\n",
      "File \u001b[0;32m~/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/github/can_chess_with_hexagons_rl/engines/experiment_a2c.py:577\u001b[0m, in \u001b[0;36mAdvancedA2CLearning.train\u001b[0;34m(self, sampling_prob_scale, replace_samples, priority_offset, importance_sampling_beta)\u001b[0m\n\u001b[1;32m    574\u001b[0m average_losses \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m    575\u001b[0m average_steps_per_episode \u001b[38;5;241m=\u001b[39m []\n\u001b[0;32m--> 577\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43minitialize_memory\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m  \u001b[38;5;66;03m# Fill the replay buffer with episodes\u001b[39;00m\n\u001b[1;32m    579\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m epoch \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mepochs):\n\u001b[1;32m    580\u001b[0m     epoch_losses \u001b[38;5;241m=\u001b[39m []  \u001b[38;5;66;03m# Store the losses for each epoch\u001b[39;00m\n",
      "File \u001b[0;32m~/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/github/can_chess_with_hexagons_rl/engines/experiment_a2c.py:876\u001b[0m, in \u001b[0;36mAdvancedA2CLearning.initialize_memory\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    874\u001b[0m average_reward \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m    875\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmemory\u001b[38;5;241m.\u001b[39mlength \u001b[38;5;241m<\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmemory_size:\n\u001b[0;32m--> 876\u001b[0m     episode, step \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgenerate_episode\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    877\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmemory\u001b[38;5;241m.\u001b[39madd(episode)\n\u001b[1;32m    878\u001b[0m     average_steps_per_episode\u001b[38;5;241m.\u001b[39mappend(step)\n",
      "File \u001b[0;32m~/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/github/can_chess_with_hexagons_rl/engines/experiment_a2c.py:701\u001b[0m, in \u001b[0;36mAdvancedA2CLearning.generate_episode\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    698\u001b[0m state_tensor \u001b[38;5;241m=\u001b[39m state_tensor\u001b[38;5;241m.\u001b[39mto(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdevice)\n\u001b[1;32m    699\u001b[0m action_tensor \u001b[38;5;241m=\u001b[39m action_tensor\u001b[38;5;241m.\u001b[39mto(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdevice)\n\u001b[0;32m--> 701\u001b[0m state_features \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackbone\u001b[49m\u001b[43m(\u001b[49m\u001b[43mstate_tensor\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    702\u001b[0m log_action_probs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mactor(state_features, action_tensor)\n\u001b[1;32m    703\u001b[0m log_action_probs \u001b[38;5;241m=\u001b[39m log_action_probs\u001b[38;5;241m.\u001b[39mview(\u001b[38;5;241m91\u001b[39m, \u001b[38;5;241m91\u001b[39m)\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1527\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1525\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1526\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1536\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1531\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1532\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1533\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1534\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1535\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1536\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1538\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1539\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m~/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/github/can_chess_with_hexagons_rl/engines/experiment_a2c.py:328\u001b[0m, in \u001b[0;36mResidualBackBone.forward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m    326\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m layer \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39minitial_convs:\n\u001b[1;32m    327\u001b[0m     x \u001b[38;5;241m=\u001b[39m layer(x)\n\u001b[0;32m--> 328\u001b[0m x \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresidual_blocks\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    329\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m x\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1527\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1525\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1526\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1536\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1531\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1532\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1533\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1534\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1535\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1536\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1538\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1539\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/container.py:217\u001b[0m, in \u001b[0;36mSequential.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    215\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;28minput\u001b[39m):\n\u001b[1;32m    216\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m module \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m:\n\u001b[0;32m--> 217\u001b[0m         \u001b[38;5;28minput\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[43mmodule\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m    218\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28minput\u001b[39m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1527\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1525\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1526\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1536\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1531\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1532\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1533\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1534\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1535\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1536\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1538\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1539\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m~/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/github/can_chess_with_hexagons_rl/engines/experiment_a2c.py:101\u001b[0m, in \u001b[0;36mResidualBlock.forward\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     99\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, x):\n\u001b[1;32m    100\u001b[0m     out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrelu1(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbn1(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconv1(x)))\n\u001b[0;32m--> 101\u001b[0m     out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbn2\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mconv2\u001b[49m\u001b[43m(\u001b[49m\u001b[43mout\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    102\u001b[0m     out \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mshortcut(x)\n\u001b[1;32m    103\u001b[0m     out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrelu2(out)\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1527\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1525\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m   1526\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/module.py:1536\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1531\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[1;32m   1532\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[1;32m   1533\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[1;32m   1534\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[1;32m   1535\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[0;32m-> 1536\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1538\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1539\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/modules/batchnorm.py:175\u001b[0m, in \u001b[0;36m_BatchNorm.forward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    168\u001b[0m     bn_training \u001b[38;5;241m=\u001b[39m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrunning_mean \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m) \u001b[38;5;129;01mand\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mrunning_var \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m    170\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    171\u001b[0m \u001b[38;5;124;03mBuffers are only updated if they are to be tracked and we are in training mode. Thus they only need to be\u001b[39;00m\n\u001b[1;32m    172\u001b[0m \u001b[38;5;124;03mpassed when the update should occur (i.e. in training mode when they are tracked), or when buffer stats are\u001b[39;00m\n\u001b[1;32m    173\u001b[0m \u001b[38;5;124;03mused for normalization (i.e. in eval mode when buffers are not None).\u001b[39;00m\n\u001b[1;32m    174\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m--> 175\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mF\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbatch_norm\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    176\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    177\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;66;43;03m# If buffers are not to be tracked, ensure that they won't be updated\u001b[39;49;00m\n\u001b[1;32m    178\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrunning_mean\u001b[49m\n\u001b[1;32m    179\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mnot\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtraining\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrack_running_stats\u001b[49m\n\u001b[1;32m    180\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43;01melse\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    181\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrunning_var\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mnot\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtraining\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrack_running_stats\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01melse\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    182\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    183\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbias\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    184\u001b[0m \u001b[43m    \u001b[49m\u001b[43mbn_training\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    185\u001b[0m \u001b[43m    \u001b[49m\u001b[43mexponential_average_factor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    186\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43meps\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    187\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/DL/lib/python3.11/site-packages/torch/nn/functional.py:2511\u001b[0m, in \u001b[0;36mbatch_norm\u001b[0;34m(input, running_mean, running_var, weight, bias, training, momentum, eps)\u001b[0m\n\u001b[1;32m   2508\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m training:\n\u001b[1;32m   2509\u001b[0m     _verify_batch_size(\u001b[38;5;28minput\u001b[39m\u001b[38;5;241m.\u001b[39msize())\n\u001b[0;32m-> 2511\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbatch_norm\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   2512\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbias\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mrunning_mean\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mrunning_var\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtraining\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmomentum\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43meps\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackends\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcudnn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menabled\u001b[49m\n\u001b[1;32m   2513\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random.pth\"\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_greedy.pth\"\n",
    "backbone_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random.pth\"\n",
    "backbone_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_greedy.pth\"\n",
    "actor_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random.pth\"\n",
    "actor_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_greedy.pth\"\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = GreedyPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "hidden_dim = 256\n",
    "\n",
    "# Actor parameters\n",
    "input_dim = 256\n",
    "hidden_dim = 256\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 120\n",
    "episodes_per_epoch = 2\n",
    "batch_size = 4\n",
    "learning_rate = 1e-3\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "simple_critic = False\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 12\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "    \n",
    "    if actor_load_path is not None:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim, model_path=actor_load_path)\n",
    "    else:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "    actor.to(device)\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "    \n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in actor.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone,\n",
    "                                    actor=actor, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=simple_critic\n",
    "                                    )\n",
    "\n",
    "\n",
    "    \n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    actor_num_parameters = count_parameters(actor)\n",
    "    print(f\"Number of trainable parameters in the Actor model: {actor_num_parameters}\")\n",
    "\n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path, actor_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "Number of trainable parameters in the Critic model: 15927297\n",
      "Number of trainable parameters in the BackBone model: 7474272\n",
      "Number of trainable parameters in the Actor model: 18055257\n",
      "Memory initialized with 12 episodes. Average steps per episode: 68.00, Average reward: 45.42\n",
      "Critic Loss: 0.9899817705154419\n",
      "Actor Loss: 0.026279829442501068\n",
      "Critic Loss: 0.9899957180023193\n",
      "Actor Loss: 0.010193939320743084\n",
      "Critic Loss: 0.9473665952682495\n",
      "Actor Loss: 0.037505701184272766\n",
      "Critic Loss: 0.968747615814209\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.9743563532829285\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9884853959083557\n",
      "Actor Loss: 0.00845985859632492\n",
      "Critic Loss: 0.9891161918640137\n",
      "Actor Loss: -3.317128118851542e-07\n",
      "Critic Loss: 0.9899824261665344\n",
      "Actor Loss: 2.670288097306184e-07\n",
      "Critic Loss: 0.9761875867843628\n",
      "Actor Loss: -3.633045082551689e-07\n",
      "Critic Loss: 0.9821397662162781\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.9899775981903076\n",
      "Actor Loss: -2.670288097306184e-07\n",
      "Critic Loss: 0.973681628704071\n",
      "Actor Loss: 2.0077354179193208e-07\n",
      "Epoch 1: Average Steps per Episode: 71.00, Average Reward: 57.50, Average Loss: 0.9869\n",
      "Critic Loss: 0.9736813902854919\n",
      "Actor Loss: 2.0077354179193208e-07\n",
      "Critic Loss: 0.9899815917015076\n",
      "Actor Loss: -0.4737030565738678\n",
      "Critic Loss: 0.9899958372116089\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.985912024974823\n",
      "Actor Loss: -2.149125180039846e-07\n",
      "Critic Loss: 0.973681628704071\n",
      "Actor Loss: -4.0154708358386415e-07\n",
      "Critic Loss: 0.9899957180023193\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9899816513061523\n",
      "Actor Loss: -0.47301581501960754\n",
      "Critic Loss: 0.9891161918640137\n",
      "Actor Loss: -2.487846018084383e-07\n",
      "Critic Loss: 0.9899814128875732\n",
      "Actor Loss: -0.47281256318092346\n",
      "Critic Loss: 0.9761877655982971\n",
      "Actor Loss: -9.082612706379223e-08\n",
      "Critic Loss: 0.9821394681930542\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.9736813902854919\n",
      "Actor Loss: -3.011602984770434e-07\n",
      "Epoch 2: Average Steps per Episode: 56.00, Average Reward: 28.00, Average Loss: 0.8654\n",
      "Critic Loss: 0.9821397066116333\n",
      "Actor Loss: 6.811959707420101e-08\n",
      "Critic Loss: 0.9884853959083557\n",
      "Actor Loss: 1.75388379375363e-07\n",
      "Critic Loss: 0.9736815094947815\n",
      "Actor Loss: 2.0077354179193208e-07\n",
      "Critic Loss: 0.9761874675750732\n",
      "Actor Loss: -1.8165225412758446e-07\n",
      "Critic Loss: 0.9761875867843628\n",
      "Actor Loss: 1.8165225412758446e-07\n",
      "Critic Loss: 0.9687479138374329\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9090033769607544\n",
      "Actor Loss: -1.0403720125395921e-06\n",
      "Critic Loss: 0.9870093464851379\n",
      "Actor Loss: -3.9633218307244533e-07\n",
      "Critic Loss: 0.9687477946281433\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.9884853363037109\n",
      "Actor Loss: 1.75388379375363e-07\n",
      "Critic Loss: 0.9899853467941284\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9090033769607544\n",
      "Actor Loss: 3.467906708465307e-06\n",
      "Epoch 3: Average Steps per Episode: 66.83, Average Reward: 47.50, Average Loss: 0.9681\n",
      "Critic Loss: 0.968747615814209\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9891160130500793\n",
      "Actor Loss: -8.292820297128856e-08\n",
      "Critic Loss: 0.9821396470069885\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9884855151176453\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899957776069641\n",
      "Actor Loss: 2.288818308215923e-07\n",
      "Critic Loss: 0.9870094060897827\n",
      "Actor Loss: 2.9724913019890664e-07\n",
      "Critic Loss: 0.9821398258209229\n",
      "Actor Loss: 2.043587983280304e-07\n",
      "Critic Loss: 0.9761877655982971\n",
      "Actor Loss: -1.8165225412758446e-07\n",
      "Critic Loss: 0.9891160130500793\n",
      "Actor Loss: 8.292820297128856e-08\n",
      "Critic Loss: 0.9821396470069885\n",
      "Actor Loss: -7.493155749216385e-07\n",
      "Critic Loss: 0.9899854063987732\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Epoch 4: Average Steps per Episode: 72.25, Average Reward: 44.00, Average Loss: 0.9778\n",
      "Critic Loss: 0.9090031981468201\n",
      "Actor Loss: 6.935813416930614e-07\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9882001280784607\n",
      "Actor Loss: -9.873334647636511e-07\n",
      "Critic Loss: 0.989126443862915\n",
      "Actor Loss: 3.317128118851542e-07\n",
      "Critic Loss: 0.9899956583976746\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9899853467941284\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9761878848075867\n",
      "Actor Loss: -1.8165225412758446e-07\n",
      "Critic Loss: 0.9891263842582703\n",
      "Actor Loss: -2.487846018084383e-07\n",
      "Critic Loss: 0.9615362882614136\n",
      "Actor Loss: -1.027033931677579e-06\n",
      "Critic Loss: 0.9761877655982971\n",
      "Actor Loss: -3.633045082551689e-07\n",
      "Critic Loss: 0.9891262054443359\n",
      "Actor Loss: -8.292820297128856e-08\n",
      "Critic Loss: 0.985912024974823\n",
      "Actor Loss: 4.298250360079692e-07\n",
      "Epoch 5: Average Steps per Episode: 64.60, Average Reward: 34.00, Average Loss: 0.9775\n",
      "Critic Loss: 0.9615365266799927\n",
      "Actor Loss: -4.401574074108794e-07\n",
      "Critic Loss: 0.9090033769607544\n",
      "Actor Loss: 2.0807440250791842e-06\n",
      "Critic Loss: 0.9870091080665588\n",
      "Actor Loss: -2.9724913019890664e-07\n",
      "Critic Loss: 0.9899851679801941\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899852275848389\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9696759581565857\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.985912024974823\n",
      "Actor Loss: -1.074562590019923e-07\n",
      "Critic Loss: 0.9090033769607544\n",
      "Actor Loss: 2.0807440250791842e-06\n",
      "Critic Loss: 0.9899851679801941\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9090033769607544\n",
      "Actor Loss: 1.3871626833861228e-06\n",
      "Critic Loss: 0.9891263842582703\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9840716123580933\n",
      "Actor Loss: -9.688120599093963e-07\n",
      "Epoch 6: Average Steps per Episode: 61.83, Average Reward: 5.50, Average Loss: 0.9645\n",
      "Critic Loss: 0.9852784276008606\n",
      "Actor Loss: 1.682954717807661e-07\n",
      "Critic Loss: 0.9090031981468201\n",
      "Actor Loss: 1.3871626833861228e-06\n",
      "Critic Loss: 0.9859121441841125\n",
      "Actor Loss: 3.223687770059769e-07\n",
      "Critic Loss: 0.9696759581565857\n",
      "Actor Loss: 1.155968902821769e-07\n",
      "Critic Loss: 0.9696757793426514\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.988200306892395\n",
      "Actor Loss: 8.975758447604676e-08\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: 2.0807440250791842e-06\n",
      "Critic Loss: 0.989126443862915\n",
      "Actor Loss: -2.487846018084383e-07\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: 1.3871626833861228e-06\n",
      "Critic Loss: 0.981478214263916\n",
      "Actor Loss: 7.064254248234647e-08\n",
      "Critic Loss: 0.9615362882614136\n",
      "Actor Loss: 1.467191310666749e-07\n",
      "Epoch 7: Average Steps per Episode: 61.71, Average Reward: 38.50, Average Loss: 0.9620\n",
      "Critic Loss: 0.9899846911430359\n",
      "Actor Loss: 4.577636616431846e-07\n",
      "Critic Loss: 0.9899852275848389\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.981478214263916\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9870094060897827\n",
      "Actor Loss: 9.908304576811133e-08\n",
      "Critic Loss: 0.9882002472877502\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Critic Loss: 0.9615362882614136\n",
      "Actor Loss: -2.934382621333498e-07\n",
      "Critic Loss: 0.9899699687957764\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9696759581565857\n",
      "Actor Loss: 6.935813416930614e-07\n",
      "Critic Loss: 0.9814783930778503\n",
      "Actor Loss: 1.4128508496469294e-07\n",
      "Critic Loss: 0.9899699091911316\n",
      "Actor Loss: 6.866454782539222e-07\n",
      "Critic Loss: 0.984071671962738\n",
      "Actor Loss: -2.494691034371499e-05\n",
      "Critic Loss: 0.9615362882614136\n",
      "Actor Loss: -4.401574074108794e-07\n",
      "Epoch 8: Average Steps per Episode: 66.50, Average Reward: 28.00, Average Loss: 0.9812\n",
      "Critic Loss: 0.9899846911430359\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9615363478660583\n",
      "Actor Loss: -1.467191310666749e-07\n",
      "Critic Loss: 0.9882002472877502\n",
      "Actor Loss: 1.2566061968755093e-06\n",
      "Critic Loss: 0.9696759581565857\n",
      "Actor Loss: 4.623875611287076e-07\n",
      "Critic Loss: 0.9696757197380066\n",
      "Actor Loss: 7.513797868341499e-07\n",
      "Critic Loss: 0.9840718507766724\n",
      "Actor Loss: -0.0001203143474413082\n",
      "Critic Loss: 0.9761730432510376\n",
      "Actor Loss: 1.8165225412758446e-07\n",
      "Critic Loss: 0.9899846911430359\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9891262054443359\n",
      "Actor Loss: 8.292820297128856e-08\n",
      "Critic Loss: 0.9761878848075867\n",
      "Actor Loss: -9.082612706379223e-08\n",
      "Critic Loss: 0.9761731624603271\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899699687957764\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Epoch 9: Average Steps per Episode: 66.17, Average Reward: 58.00, Average Loss: 0.9801\n",
      "Critic Loss: 0.9761731624603271\n",
      "Actor Loss: -3.633045082551689e-07\n",
      "Critic Loss: 0.9814783334732056\n",
      "Actor Loss: 1.4128508496469294e-07\n",
      "Critic Loss: 0.9615100622177124\n",
      "Actor Loss: 1.1737530485333991e-06\n",
      "Critic Loss: 0.9899700880050659\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9899846911430359\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9814781546592712\n",
      "Actor Loss: 2.119276274470394e-07\n",
      "Critic Loss: 0.9840717315673828\n",
      "Actor Loss: -0.003054664470255375\n",
      "Critic Loss: 0.9852784276008606\n",
      "Actor Loss: -2.2439395763740322e-07\n",
      "Critic Loss: 0.9761875867843628\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9882311820983887\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Critic Loss: 0.984071671962738\n",
      "Actor Loss: -0.0822523832321167\n",
      "Critic Loss: 0.9615101218223572\n",
      "Actor Loss: -1.467191310666749e-07\n",
      "Epoch 10: Average Steps per Episode: 62.95, Average Reward: 10.00, Average Loss: 0.9729\n",
      "Critic Loss: 0.9761776924133301\n",
      "Actor Loss: 4.5413063531896114e-08\n",
      "Critic Loss: 0.9814783930778503\n",
      "Actor Loss: 1.4128508496469294e-07\n",
      "Critic Loss: 0.9696759581565857\n",
      "Actor Loss: 1.155968902821769e-07\n",
      "Critic Loss: 0.9840713739395142\n",
      "Actor Loss: -0.2946193516254425\n",
      "Critic Loss: 0.985278308391571\n",
      "Actor Loss: -4.4878791527480644e-07\n",
      "Critic Loss: 0.9814784526824951\n",
      "Actor Loss: 0.05685954540967941\n",
      "Critic Loss: 0.9899702072143555\n",
      "Actor Loss: 5.340576194612368e-07\n",
      "Critic Loss: 0.9615102410316467\n",
      "Actor Loss: -5.868765242666996e-07\n",
      "Critic Loss: 0.981478214263916\n",
      "Actor Loss: 1.4128508496469294e-07\n",
      "Critic Loss: 0.9882315397262573\n",
      "Actor Loss: 8.975758447604676e-08\n",
      "Critic Loss: 0.9615102410316467\n",
      "Actor Loss: -5.868765242666996e-07\n",
      "Critic Loss: 0.973681628704071\n",
      "Actor Loss: -2.0077354179193208e-07\n",
      "Epoch 11: Average Steps per Episode: 63.50, Average Reward: 36.00, Average Loss: 0.9581\n",
      "Critic Loss: 0.9761776924133301\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.9852784276008606\n",
      "Actor Loss: -1.9634471470908466e-07\n",
      "Critic Loss: 0.9761875867843628\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.9615102410316467\n",
      "Actor Loss: 2.934382621333498e-07\n",
      "Critic Loss: 0.9499981999397278\n",
      "Actor Loss: 5.722046125811175e-07\n",
      "Critic Loss: 0.9899700880050659\n",
      "Actor Loss: 3.8146970382513246e-07\n",
      "Critic Loss: 0.9615103006362915\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.988231360912323\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Critic Loss: 0.9882311820983887\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Critic Loss: 0.9761775732040405\n",
      "Actor Loss: -1.8165225412758446e-07\n",
      "Critic Loss: 0.9852781891822815\n",
      "Actor Loss: -2.8049245770489506e-07\n",
      "Critic Loss: 0.9814783930778503\n",
      "Actor Loss: 7.064254248234647e-08\n",
      "Epoch 12: Average Steps per Episode: 62.04, Average Reward: 40.00, Average Loss: 0.9767\n",
      "Critic Loss: 0.9835736155509949\n",
      "Actor Loss: 6.253601725347835e-08\n",
      "Critic Loss: 0.9761776924133301\n",
      "Actor Loss: -9.082612706379223e-08\n",
      "Critic Loss: 0.961510419845581\n",
      "Actor Loss: 4.401574074108794e-07\n",
      "Critic Loss: 0.9899845719337463\n",
      "Actor Loss: 2.288818308215923e-07\n",
      "Critic Loss: 0.9499980211257935\n",
      "Actor Loss: 5.722046125811175e-07\n",
      "Critic Loss: 0.9761877655982971\n",
      "Actor Loss: -9.082612706379223e-08\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: 2.288818308215923e-07\n",
      "Critic Loss: 0.9899699091911316\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9714259505271912\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899848699569702\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9899699091911316\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9835734963417053\n",
      "Actor Loss: -7.504322070417402e-07\n",
      "Epoch 13: Average Steps per Episode: 60.96, Average Reward: 28.00, Average Loss: 0.9794\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -3.8146970382513246e-07\n",
      "Critic Loss: 0.9860963821411133\n",
      "Actor Loss: 2.119276274470394e-07\n",
      "Critic Loss: 0.9761777520179749\n",
      "Actor Loss: -9.082612706379223e-08\n",
      "Critic Loss: 0.9835735559463501\n",
      "Actor Loss: 1.250720345069567e-07\n",
      "Critic Loss: 0.9499983191490173\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9899701476097107\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9860963821411133\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9714259505271912\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9761776924133301\n",
      "Actor Loss: -1.8165225412758446e-07\n",
      "Critic Loss: 0.9761874675750732\n",
      "Actor Loss: 4.541306282135338e-07\n",
      "Critic Loss: 0.9899702072143555\n",
      "Actor Loss: -7.24792471373803e-07\n",
      "Critic Loss: 0.988231360912323\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Epoch 14: Average Steps per Episode: 63.18, Average Reward: 49.00, Average Loss: 0.9803\n",
      "Critic Loss: 0.9714259505271912\n",
      "Actor Loss: -4.359654042218608e-07\n",
      "Critic Loss: 0.9899837374687195\n",
      "Actor Loss: 2.288818308215923e-07\n",
      "Critic Loss: 0.961510419845581\n",
      "Actor Loss: 1.467191310666749e-07\n",
      "Critic Loss: 0.9761776924133301\n",
      "Actor Loss: 9.082612706379223e-08\n",
      "Critic Loss: 0.9835734367370605\n",
      "Actor Loss: -8.755042699704063e-07\n",
      "Critic Loss: 0.9860963821411133\n",
      "Actor Loss: 7.417467031700653e-07\n",
      "Critic Loss: 0.9880910515785217\n",
      "Actor Loss: 1.8165225412758446e-07\n",
      "Critic Loss: 0.9615104794502258\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9736816883087158\n",
      "Actor Loss: 1.0038677089596604e-07\n",
      "Critic Loss: 0.9899700880050659\n",
      "Actor Loss: 1.9073485191256623e-07\n",
      "Critic Loss: 0.9880910515785217\n",
      "Actor Loss: 9.082612706379223e-08\n",
      "Critic Loss: 0.9714259505271912\n",
      "Actor Loss: -3.2697406027182296e-07\n",
      "Epoch 15: Average Steps per Episode: 65.63, Average Reward: 24.50, Average Loss: 0.9785\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: -1.1444091541079615e-07\n",
      "Critic Loss: 0.9835734963417053\n",
      "Actor Loss: -1.250720345069567e-07\n",
      "Critic Loss: 0.9899700880050659\n",
      "Actor Loss: -1.5258788153005298e-06\n",
      "Critic Loss: 0.9860966205596924\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9835736751556396\n",
      "Actor Loss: -7.504322070417402e-07\n",
      "Critic Loss: 0.9899823069572449\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9841130375862122\n",
      "Actor Loss: -3.6330453667687834e-07\n",
      "Critic Loss: 0.9736816883087158\n",
      "Actor Loss: -4.0154708358386415e-07\n",
      "Critic Loss: 0.9841129779815674\n",
      "Actor Loss: -2.422030149773491e-07\n",
      "Critic Loss: 0.9714259505271912\n",
      "Actor Loss: -2.179827021109304e-07\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9880913496017456\n",
      "Actor Loss: 3.633045082551689e-07\n",
      "Epoch 16: Average Steps per Episode: 65.50, Average Reward: 45.50, Average Loss: 0.9845\n",
      "Critic Loss: 0.9499981999397278\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.9714260697364807\n",
      "Actor Loss: -1.089913510554652e-07\n",
      "Critic Loss: 0.9830263257026672\n",
      "Actor Loss: -6.465588597848182e-08\n",
      "Critic Loss: 0.9899819493293762\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9835736155509949\n",
      "Actor Loss: -5.002881380278268e-07\n",
      "Critic Loss: 0.9714258313179016\n",
      "Actor Loss: -1.089913510554652e-07\n",
      "Critic Loss: 0.9860965013504028\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9841129183769226\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9830265641212463\n",
      "Actor Loss: -2.586235439139273e-07\n",
      "Critic Loss: 0.9499979019165039\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9880911707878113\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -0.0\n",
      "Epoch 17: Average Steps per Episode: 63.79, Average Reward: 30.50, Average Loss: 0.9776\n",
      "Critic Loss: 0.9714260697364807\n",
      "Actor Loss: 1.089913510554652e-07\n",
      "Critic Loss: 0.9285699725151062\n",
      "Actor Loss: -1.0899135531872162e-06\n",
      "Critic Loss: 0.9830262660980225\n",
      "Actor Loss: -7.758706033200724e-07\n",
      "Critic Loss: 0.9843716025352478\n",
      "Actor Loss: -5.960464477539063e-08\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -7.629394076502649e-07\n",
      "Critic Loss: 0.9880911707878113\n",
      "Actor Loss: -3.633045082551689e-07\n",
      "Critic Loss: 0.984371542930603\n",
      "Actor Loss: -1.1920928955078125e-07\n",
      "Critic Loss: 0.928570032119751\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9835736155509949\n",
      "Actor Loss: -5.628241410704504e-07\n",
      "Critic Loss: 0.988091230392456\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.9830265641212463\n",
      "Actor Loss: -3.879353016600362e-07\n",
      "Critic Loss: 0.9899753332138062\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Epoch 18: Average Steps per Episode: 64.11, Average Reward: 50.00, Average Loss: 0.9753\n",
      "Critic Loss: 0.9899753928184509\n",
      "Actor Loss: -7.629394076502649e-07\n",
      "Critic Loss: 0.9843713641166687\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.9899812936782837\n",
      "Actor Loss: -7.629394076502649e-07\n",
      "Critic Loss: 0.9499855041503906\n",
      "Actor Loss: -1.9073486612342094e-07\n",
      "Critic Loss: 0.9499854445457458\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.986650824546814\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.988091230392456\n",
      "Actor Loss: 1.8165225412758446e-07\n",
      "Critic Loss: 0.9843716621398926\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.9499855041503906\n",
      "Actor Loss: 1.9073486612342094e-07\n",
      "Critic Loss: 0.9899753928184509\n",
      "Actor Loss: -5.340576194612368e-07\n",
      "Critic Loss: 0.9866507053375244\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Epoch 19: Average Steps per Episode: 63.89, Average Reward: 16.00, Average Loss: 0.9783\n",
      "Critic Loss: 0.9899837970733643\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9882189035415649\n",
      "Actor Loss: 1.7951516895209352e-07\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899751543998718\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -8.392333938900265e-07\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: -5.449567765936081e-07\n",
      "Critic Loss: 0.9882189631462097\n",
      "Actor Loss: 8.975758447604676e-08\n",
      "Critic Loss: 0.9830264449119568\n",
      "Actor Loss: 8.40526524825691e-07\n",
      "Critic Loss: 0.9841129779815674\n",
      "Actor Loss: -1.2110150748867454e-07\n",
      "Critic Loss: 0.986650824546814\n",
      "Actor Loss: -5.086263143994074e-08\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9882189631462097\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Epoch 20: Average Steps per Episode: 63.92, Average Reward: 42.00, Average Loss: 0.9831\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: -1.1444091541079615e-07\n",
      "Critic Loss: 0.9843716025352478\n",
      "Actor Loss: -2.980232238769531e-07\n",
      "Critic Loss: 0.983026385307312\n",
      "Actor Loss: -1.2931177195696364e-07\n",
      "Critic Loss: 0.984371542930603\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Critic Loss: 0.9843716025352478\n",
      "Actor Loss: 3.5762786865234375e-07\n",
      "Critic Loss: 0.9841129779815674\n",
      "Actor Loss: 1.2110150748867454e-07\n",
      "Critic Loss: 0.9882189631462097\n",
      "Actor Loss: -3.5903033790418704e-07\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9843716025352478\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9499854445457458\n",
      "Actor Loss: -1.9073486612342094e-07\n",
      "Critic Loss: 0.9830264449119568\n",
      "Actor Loss: 6.465588597848182e-08\n",
      "Epoch 21: Average Steps per Episode: 65.38, Average Reward: 49.50, Average Loss: 0.9779\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9499856233596802\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.9866509437561035\n",
      "Actor Loss: 3.306070937014738e-07\n",
      "Critic Loss: 0.9772700071334839\n",
      "Actor Loss: -2.6009300313489803e-07\n",
      "Critic Loss: 0.9887599349021912\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9866509437561035\n",
      "Actor Loss: -5.086263143994074e-08\n",
      "Critic Loss: 0.9772700071334839\n",
      "Actor Loss: -8.669766771163268e-08\n",
      "Critic Loss: 0.9899814128875732\n",
      "Actor Loss: -6.103515488575795e-07\n",
      "Critic Loss: 0.989980161190033\n",
      "Actor Loss: -3.623962356869015e-07\n",
      "Critic Loss: 0.9772700071334839\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9285699725151062\n",
      "Actor Loss: -1.0899135531872162e-06\n",
      "Critic Loss: 0.9629607200622559\n",
      "Actor Loss: 5.651403398587718e-07\n",
      "Epoch 22: Average Steps per Episode: 65.30, Average Reward: 42.50, Average Loss: 0.9754\n",
      "Critic Loss: 0.984371542930603\n",
      "Actor Loss: 5.960464477539063e-08\n",
      "Critic Loss: 0.9899812936782837\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9899753332138062\n",
      "Actor Loss: 3.814697180359872e-08\n",
      "Critic Loss: 0.9899848699569702\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: 5.340576194612368e-07\n",
      "Critic Loss: 0.989984929561615\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9899800419807434\n",
      "Actor Loss: -4.0054320038507285e-07\n",
      "Critic Loss: 0.984371542930603\n",
      "Actor Loss: -1.1920928955078125e-07\n",
      "Critic Loss: 0.9887599349021912\n",
      "Actor Loss: -8.572353493718765e-08\n",
      "Critic Loss: 0.9899845123291016\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9899814128875732\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Epoch 23: Average Steps per Episode: 66.80, Average Reward: 26.50, Average Loss: 0.9889\n",
      "Critic Loss: 0.9899799823760986\n",
      "Actor Loss: -8.583068478174027e-08\n",
      "Critic Loss: 0.9499855041503906\n",
      "Actor Loss: -1.125335711549269e-05\n",
      "Critic Loss: 0.9899814128875732\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9705706238746643\n",
      "Actor Loss: -1.682954717807661e-07\n",
      "Critic Loss: 0.9887600541114807\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9882187247276306\n",
      "Actor Loss: -3.5903033790418704e-07\n",
      "Critic Loss: 0.9899799823760986\n",
      "Actor Loss: 1.907348590179936e-08\n",
      "Critic Loss: 0.9629608392715454\n",
      "Actor Loss: -1.1302806797175435e-06\n",
      "Critic Loss: 0.9629607796669006\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9899848699569702\n",
      "Actor Loss: -1.9073485191256623e-07\n",
      "Critic Loss: 0.9705705046653748\n",
      "Actor Loss: 2.524432147765765e-07\n",
      "Critic Loss: 0.9882189035415649\n",
      "Actor Loss: -3.5903033790418704e-07\n",
      "Epoch 24: Average Steps per Episode: 65.75, Average Reward: 39.00, Average Loss: 0.9785\n",
      "Critic Loss: 0.989984929561615\n",
      "Actor Loss: 3.814697180359872e-08\n",
      "Critic Loss: 0.9899800419807434\n",
      "Actor Loss: 3.814697180359872e-08\n",
      "Critic Loss: 0.9782580137252808\n",
      "Actor Loss: -4.1464102196187014e-07\n",
      "Critic Loss: 0.989975094795227\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9795888066291809\n",
      "Actor Loss: -1.557019260189918e-07\n",
      "Critic Loss: 0.9882187843322754\n",
      "Actor Loss: -1.7951516895209352e-07\n",
      "Critic Loss: 0.9705706238746643\n",
      "Actor Loss: 5.6098489409350805e-08\n",
      "Critic Loss: 0.9629607796669006\n",
      "Actor Loss: 1.4128508496469294e-07\n",
      "Critic Loss: 0.9772700071334839\n",
      "Actor Loss: -1.7339533542326535e-07\n",
      "Critic Loss: 0.9887600541114807\n",
      "Actor Loss: 1.285852988530678e-07\n",
      "Critic Loss: 0.9795886874198914\n",
      "Actor Loss: -3.114038520379836e-07\n",
      "Critic Loss: 0.9899848103523254\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Epoch 25: Average Steps per Episode: 64.80, Average Reward: 35.50, Average Loss: 0.9821\n",
      "Critic Loss: 0.973673403263092\n",
      "Actor Loss: 3.011602984770434e-07\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9899846911430359\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9899851083755493\n",
      "Actor Loss: 1.1444091541079615e-07\n",
      "Critic Loss: 0.9887601137161255\n",
      "Actor Loss: 8.572353493718765e-08\n",
      "Critic Loss: 0.9782577157020569\n",
      "Actor Loss: -1.658564059425771e-07\n",
      "Critic Loss: 0.9899753332138062\n",
      "Actor Loss: -6.866454782539222e-07\n",
      "Critic Loss: 0.9899753928184509\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9899751543998718\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9705707430839539\n",
      "Actor Loss: -3.365909435615322e-07\n",
      "Critic Loss: 0.9899800419807434\n",
      "Actor Loss: -1.335144048653092e-07\n",
      "Epoch 26: Average Steps per Episode: 64.71, Average Reward: 10.00, Average Loss: 0.9859\n",
      "Critic Loss: 0.9782577157020569\n",
      "Actor Loss: 1.658564059425771e-07\n",
      "Critic Loss: 0.9654923677444458\n",
      "Actor Loss: 2.630825690630445e-07\n",
      "Critic Loss: 0.9599113464355469\n",
      "Actor Loss: -1.220703097715159e-06\n",
      "Critic Loss: 0.989975094795227\n",
      "Actor Loss: 3.433227391269611e-07\n",
      "Critic Loss: 0.973673403263092\n",
      "Actor Loss: 1.0038677089596604e-07\n",
      "Critic Loss: 0.9599113464355469\n",
      "Actor Loss: -6.103515488575795e-07\n",
      "Critic Loss: 0.9818148016929626\n",
      "Actor Loss: -3.4679064242482127e-07\n",
      "Critic Loss: 0.989980161190033\n",
      "Actor Loss: -2.5749204723979346e-07\n",
      "Critic Loss: 0.973673403263092\n",
      "Actor Loss: -5.019338686906849e-07\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: -5.722045557376987e-07\n",
      "Critic Loss: 0.989980161190033\n",
      "Actor Loss: -1.7166136956348055e-07\n",
      "Critic Loss: 0.9629608988761902\n",
      "Actor Loss: 1.4128508496469294e-07\n",
      "Epoch 27: Average Steps per Episode: 63.87, Average Reward: 34.50, Average Loss: 0.9763\n",
      "Critic Loss: 0.9599114060401917\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9777591228485107\n",
      "Actor Loss: -1.6954210479980247e-07\n",
      "Critic Loss: 0.9818150401115417\n",
      "Actor Loss: -1.3871625981209945e-07\n",
      "Critic Loss: 0.9899753332138062\n",
      "Actor Loss: -3.814697180359872e-08\n",
      "Critic Loss: 0.9654923677444458\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9818150401115417\n",
      "Actor Loss: 2.0807439682357654e-07\n",
      "Critic Loss: 0.9599111676216125\n",
      "Actor Loss: -2.899169885495212e-06\n",
      "Critic Loss: 0.9795886874198914\n",
      "Actor Loss: 2.3355289613391506e-07\n",
      "Critic Loss: 0.9899752736091614\n",
      "Actor Loss: -6.103515488575795e-07\n",
      "Critic Loss: 0.9795886874198914\n",
      "Actor Loss: 4.671057922678301e-07\n",
      "Critic Loss: 0.9777592420578003\n",
      "Actor Loss: 4.238552619995062e-08\n",
      "Critic Loss: 0.9899753928184509\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Epoch 28: Average Steps per Episode: 63.98, Average Reward: 42.00, Average Loss: 0.9778\n",
      "Critic Loss: 0.9736734628677368\n",
      "Actor Loss: -1.0038677089596604e-07\n",
      "Critic Loss: 0.9795886874198914\n",
      "Actor Loss: 4.671057922678301e-07\n",
      "Critic Loss: 0.9654923677444458\n",
      "Actor Loss: -2.630825690630445e-07\n",
      "Critic Loss: 0.9856998920440674\n",
      "Actor Loss: -1.373290979245212e-05\n",
      "Critic Loss: 0.9599111676216125\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9777592420578003\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9705706238746643\n",
      "Actor Loss: -1.682954717807661e-07\n",
      "Critic Loss: 0.9782578349113464\n",
      "Actor Loss: 2.487846018084383e-07\n",
      "Critic Loss: 0.9818148612976074\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9705705046653748\n",
      "Actor Loss: -2.2439395763740322e-07\n",
      "Critic Loss: 0.9777592420578003\n",
      "Actor Loss: 4.238552619995062e-08\n",
      "Critic Loss: 0.9654922485351562\n",
      "Actor Loss: 2.630825690630445e-07\n",
      "Epoch 29: Average Steps per Episode: 63.97, Average Reward: 41.00, Average Loss: 0.9739\n",
      "Critic Loss: 0.9887600541114807\n",
      "Actor Loss: 4.2861767468593825e-08\n",
      "Critic Loss: 0.9857001304626465\n",
      "Actor Loss: -5.63485300517641e-05\n",
      "Critic Loss: 0.9899753928184509\n",
      "Actor Loss: -6.103515488575795e-07\n",
      "Critic Loss: 0.9777593016624451\n",
      "Actor Loss: -4.238552619995062e-08\n",
      "Critic Loss: 0.9782577157020569\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9777593016624451\n",
      "Actor Loss: 8.477105239990124e-08\n",
      "Critic Loss: 0.9599111676216125\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9899753928184509\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.973673403263092\n",
      "Actor Loss: 1.0038677089596604e-07\n",
      "Critic Loss: 0.9887598752975464\n",
      "Actor Loss: 1.714470698743753e-07\n",
      "Critic Loss: 0.9899753332138062\n",
      "Actor Loss: -1.9073485191256623e-07\n",
      "Critic Loss: 0.9777592420578003\n",
      "Actor Loss: -1.6954210479980247e-07\n",
      "Epoch 30: Average Steps per Episode: 64.18, Average Reward: 19.50, Average Loss: 0.9815\n",
      "Critic Loss: 0.9755806922912598\n",
      "Actor Loss: -3.721655730259954e-07\n",
      "Critic Loss: 0.9887598752975464\n",
      "Actor Loss: -3.000323829382978e-07\n",
      "Critic Loss: 0.9818150401115417\n",
      "Actor Loss: 2.774325196241989e-07\n",
      "Critic Loss: 0.989984929561615\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.982452929019928\n",
      "Actor Loss: -6.023205969540868e-07\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: 3.8146970382513246e-07\n",
      "Critic Loss: 0.9654924273490906\n",
      "Actor Loss: 5.26165138126089e-07\n",
      "Critic Loss: 0.9599112272262573\n",
      "Actor Loss: -9.155273232863692e-07\n",
      "Critic Loss: 0.9777592420578003\n",
      "Actor Loss: 4.238552619995062e-08\n",
      "Critic Loss: 0.9818148016929626\n",
      "Actor Loss: 5.548650392483978e-07\n",
      "Critic Loss: 0.9857000112533569\n",
      "Actor Loss: -0.0006252288585528731\n",
      "Critic Loss: 0.9824527502059937\n",
      "Actor Loss: -1.3384902786128805e-07\n",
      "Epoch 31: Average Steps per Episode: 64.24, Average Reward: 39.50, Average Loss: 0.9801\n",
      "Critic Loss: 0.9755808115005493\n",
      "Actor Loss: 6.512897243737825e-07\n",
      "Critic Loss: 0.982452929019928\n",
      "Actor Loss: -6.692451393064403e-08\n",
      "Critic Loss: 0.977759063243866\n",
      "Actor Loss: -4.238552619995062e-08\n",
      "Critic Loss: 0.9887601137161255\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9755809903144836\n",
      "Actor Loss: 5.582483595389931e-07\n",
      "Critic Loss: 0.9654923677444458\n",
      "Actor Loss: 2.630825690630445e-07\n",
      "Critic Loss: 0.9857000112533569\n",
      "Actor Loss: -0.0023524693679064512\n",
      "Critic Loss: 0.981814980506897\n",
      "Actor Loss: 2.774325196241989e-07\n",
      "Critic Loss: 0.9755808711051941\n",
      "Actor Loss: -5.582483595389931e-07\n",
      "Critic Loss: 0.9791520833969116\n",
      "Actor Loss: 7.947286206899662e-08\n",
      "Critic Loss: 0.9899846911430359\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9687479138374329\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Epoch 32: Average Steps per Episode: 64.36, Average Reward: 18.50, Average Loss: 0.9787\n",
      "Critic Loss: 0.9887600541114807\n",
      "Actor Loss: -3.428941397487506e-07\n",
      "Critic Loss: 0.9755808711051941\n",
      "Actor Loss: -1.860827865129977e-07\n",
      "Critic Loss: 0.9886195659637451\n",
      "Actor Loss: 4.334883385581634e-07\n",
      "Critic Loss: 0.9791521430015564\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Critic Loss: 0.968747615814209\n",
      "Actor Loss: -1.1920928955078125e-07\n",
      "Critic Loss: 0.989984393119812\n",
      "Actor Loss: 5.340576194612368e-07\n",
      "Critic Loss: 0.9857000112533569\n",
      "Actor Loss: -0.005005373153835535\n",
      "Critic Loss: 0.9824529886245728\n",
      "Actor Loss: 2.0077354179193208e-07\n",
      "Critic Loss: 0.9887600541114807\n",
      "Actor Loss: 3.428941397487506e-07\n",
      "Critic Loss: 0.9755805730819702\n",
      "Actor Loss: -1.860827865129977e-07\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9687476754188538\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Epoch 33: Average Steps per Episode: 64.30, Average Reward: 35.00, Average Loss: 0.9814\n",
      "Critic Loss: 0.9677397608757019\n",
      "Actor Loss: 1.230547468367149e-07\n",
      "Critic Loss: 0.9824528098106384\n",
      "Actor Loss: -1.3384902786128805e-07\n",
      "Critic Loss: 0.9755808115005493\n",
      "Actor Loss: -1.1164967190779862e-06\n",
      "Critic Loss: 0.968747615814209\n",
      "Actor Loss: 3.5762786865234375e-07\n",
      "Critic Loss: 0.9791518449783325\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.9824528694152832\n",
      "Actor Loss: -2.676980557225761e-07\n",
      "Critic Loss: 0.9857001304626465\n",
      "Actor Loss: -0.007123238872736692\n",
      "Critic Loss: 0.9857105612754822\n",
      "Actor Loss: -1.089913510554652e-07\n",
      "Critic Loss: 0.9877889156341553\n",
      "Actor Loss: -2.7912417976949655e-07\n",
      "Critic Loss: 0.9886195659637451\n",
      "Actor Loss: 3.0344182277985965e-07\n",
      "Critic Loss: 0.9687477946281433\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.9856998920440674\n",
      "Actor Loss: -0.008639144711196423\n",
      "Epoch 34: Average Steps per Episode: 64.65, Average Reward: 47.50, Average Loss: 0.9786\n",
      "Critic Loss: 0.9893397688865662\n",
      "Actor Loss: 2.4349131422241044e-07\n",
      "Critic Loss: 0.9899835586547852\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Critic Loss: 0.9677396416664124\n",
      "Actor Loss: -2.461094936734298e-07\n",
      "Critic Loss: 0.9860886335372925\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9886196851730347\n",
      "Actor Loss: 1.3004650156744901e-07\n",
      "Critic Loss: 0.9899836778640747\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9677396416664124\n",
      "Actor Loss: -4.922189873468596e-07\n",
      "Critic Loss: 0.9877890348434448\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9857105016708374\n",
      "Actor Loss: 2.179827021109304e-07\n",
      "Critic Loss: 0.9899836778640747\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9860888719558716\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9893395900726318\n",
      "Actor Loss: 8.116376903899436e-08\n",
      "Epoch 35: Average Steps per Episode: 65.26, Average Reward: 21.50, Average Loss: 0.9849\n",
      "Critic Loss: 0.9886195659637451\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9857106804847717\n",
      "Actor Loss: 6.539481205436459e-07\n",
      "Critic Loss: 0.9846003651618958\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9893397688865662\n",
      "Actor Loss: -8.116376903899436e-08\n",
      "Critic Loss: 0.9893397688865662\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899836778640747\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9846004247665405\n",
      "Actor Loss: 3.5212590887567785e-07\n",
      "Critic Loss: 0.9877890348434448\n",
      "Actor Loss: -1.860827865129977e-07\n",
      "Critic Loss: 0.9886196851730347\n",
      "Actor Loss: -2.6009300313489803e-07\n",
      "Critic Loss: 0.9899851679801941\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9857106804847717\n",
      "Actor Loss: 2.179827021109304e-07\n",
      "Critic Loss: 0.9893398284912109\n",
      "Actor Loss: 8.116376903899436e-08\n",
      "Epoch 36: Average Steps per Episode: 64.51, Average Reward: 9.50, Average Loss: 0.9878\n",
      "Critic Loss: 0.9860888719558716\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.983047604560852\n",
      "Actor Loss: -1.939676508300181e-07\n",
      "Critic Loss: 0.9677397608757019\n",
      "Actor Loss: -2.461094936734298e-07\n",
      "Critic Loss: 0.9846002459526062\n",
      "Actor Loss: -2.34750601180167e-07\n",
      "Critic Loss: 0.9899817705154419\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9877889156341553\n",
      "Actor Loss: -9.304139325649885e-08\n",
      "Critic Loss: 0.9857106804847717\n",
      "Actor Loss: -1.089913510554652e-07\n",
      "Critic Loss: 0.983047604560852\n",
      "Actor Loss: -1.939676508300181e-07\n",
      "Critic Loss: 0.9899818897247314\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Critic Loss: 0.9166061878204346\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.986088752746582\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9846001863479614\n",
      "Actor Loss: -0.0\n",
      "Epoch 37: Average Steps per Episode: 64.92, Average Reward: 36.00, Average Loss: 0.9788\n",
      "Critic Loss: 0.9830474257469177\n",
      "Actor Loss: 1.939676508300181e-07\n",
      "Critic Loss: 0.9893396496772766\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899834990501404\n",
      "Actor Loss: 3.8146970382513246e-07\n",
      "Critic Loss: 0.9846003651618958\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.916665256023407\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899835586547852\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.983047604560852\n",
      "Actor Loss: -1.939676508300181e-07\n",
      "Critic Loss: 0.9166061878204346\n",
      "Actor Loss: 1.5894572698016418e-06\n",
      "Critic Loss: 0.9857105612754822\n",
      "Actor Loss: -2.179827021109304e-07\n",
      "Critic Loss: 0.986088752746582\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899817705154419\n",
      "Actor Loss: -3.814697180359872e-08\n",
      "Critic Loss: 0.9677398204803467\n",
      "Actor Loss: 2.461094936734298e-07\n",
      "Epoch 38: Average Steps per Episode: 64.68, Average Reward: 36.50, Average Loss: 0.9736\n",
      "Critic Loss: 0.9899834394454956\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9886137843132019\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.9166061282157898\n",
      "Actor Loss: -1.5894572413799324e-07\n",
      "Critic Loss: 0.9899770617485046\n",
      "Actor Loss: -1.9073485191256623e-07\n",
      "Critic Loss: 0.9830475449562073\n",
      "Actor Loss: 3.2327943699783646e-07\n",
      "Critic Loss: 0.9899799823760986\n",
      "Actor Loss: 4.577636616431846e-07\n",
      "Critic Loss: 0.9846004247665405\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899818301200867\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Critic Loss: 0.9166061282157898\n",
      "Actor Loss: 1.748402951307071e-06\n",
      "Critic Loss: 0.9860888719558716\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.984600305557251\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899799823760986\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Epoch 39: Average Steps per Episode: 65.44, Average Reward: 21.50, Average Loss: 0.9758\n",
      "Critic Loss: 0.9886138439178467\n",
      "Actor Loss: -4.334883385581634e-07\n",
      "Critic Loss: 0.983047366142273\n",
      "Actor Loss: -1.2931177195696364e-07\n",
      "Critic Loss: 0.9899836778640747\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.916665256023407\n",
      "Actor Loss: -7.947286349008209e-07\n",
      "Critic Loss: 0.9166651964187622\n",
      "Actor Loss: -1.5894572413799324e-07\n",
      "Critic Loss: 0.9899836778640747\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9899799823760986\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899819493293762\n",
      "Actor Loss: -3.8146970382513246e-07\n",
      "Critic Loss: 0.986088752746582\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9899798631668091\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9830474853515625\n",
      "Actor Loss: -3.879353016600362e-07\n",
      "Critic Loss: 0.9846003651618958\n",
      "Actor Loss: -2.34750601180167e-07\n",
      "Epoch 40: Average Steps per Episode: 65.54, Average Reward: 45.50, Average Loss: 0.9757\n",
      "Critic Loss: 0.9860954284667969\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9886136651039124\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.9899824857711792\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.983047366142273\n",
      "Actor Loss: -1.939676508300181e-07\n",
      "Critic Loss: 0.9166061282157898\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.9899770617485046\n",
      "Actor Loss: 1.9073485191256623e-07\n",
      "Critic Loss: 0.9860953092575073\n",
      "Actor Loss: -4.238552548940788e-07\n",
      "Critic Loss: 0.9654920101165771\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9166061282157898\n",
      "Actor Loss: 2.384185791015625e-06\n",
      "Critic Loss: 0.9899798631668091\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9830474853515625\n",
      "Actor Loss: -2.586235439139273e-07\n",
      "Critic Loss: 0.9846002459526062\n",
      "Actor Loss: -1.173753005900835e-07\n",
      "Epoch 41: Average Steps per Episode: 65.51, Average Reward: 16.50, Average Loss: 0.9733\n",
      "Critic Loss: 0.989984393119812\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9899819493293762\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9830474853515625\n",
      "Actor Loss: -6.465588597848182e-08\n",
      "Critic Loss: 0.9899770617485046\n",
      "Actor Loss: -2.670288097306184e-07\n",
      "Critic Loss: 0.9850711822509766\n",
      "Actor Loss: -5.693577875831579e-08\n",
      "Critic Loss: 0.9166654348373413\n",
      "Actor Loss: -1.5894572413799324e-07\n",
      "Critic Loss: 0.9899799227714539\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9899845123291016\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Critic Loss: 0.9850711822509766\n",
      "Actor Loss: 5.693577875831579e-08\n",
      "Critic Loss: 0.9899770617485046\n",
      "Actor Loss: 6.103515488575795e-07\n",
      "Critic Loss: 0.9899818897247314\n",
      "Actor Loss: 6.866454782539222e-07\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Epoch 42: Average Steps per Episode: 65.86, Average Reward: 28.50, Average Loss: 0.9825\n",
      "Critic Loss: 0.9886139631271362\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9833136796951294\n",
      "Actor Loss: -1.9073488033427566e-07\n",
      "Critic Loss: 0.9899770617485046\n",
      "Actor Loss: -3.814697180359872e-08\n",
      "Critic Loss: 0.9654921293258667\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9850711822509766\n",
      "Actor Loss: -2.2774311503326317e-07\n",
      "Critic Loss: 0.9833136796951294\n",
      "Actor Loss: 6.357829107628277e-08\n",
      "Critic Loss: 0.988614022731781\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9860955476760864\n",
      "Actor Loss: 2.119276274470394e-07\n",
      "Critic Loss: 0.9654920101165771\n",
      "Actor Loss: -5.26165138126089e-07\n",
      "Critic Loss: 0.9899770617485046\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Critic Loss: 0.9886139631271362\n",
      "Actor Loss: 8.669766771163268e-08\n",
      "Epoch 43: Average Steps per Episode: 66.65, Average Reward: 30.50, Average Loss: 0.9837\n",
      "Critic Loss: 0.9886139631271362\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9654920101165771\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899826049804688\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9833298921585083\n",
      "Actor Loss: -1.2715658215256553e-07\n",
      "Critic Loss: 0.9899824261665344\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9899851083755493\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9833298921585083\n",
      "Actor Loss: -2.5431316430513107e-07\n",
      "Critic Loss: 0.9886138439178467\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9899824857711792\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Epoch 44: Average Steps per Episode: 65.98, Average Reward: 55.50, Average Loss: 0.9866\n",
      "Critic Loss: 0.9285699129104614\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9665914177894592\n",
      "Actor Loss: 2.5431316430513107e-07\n",
      "Critic Loss: 0.9850711822509766\n",
      "Actor Loss: -2.846788902388653e-07\n",
      "Critic Loss: 0.9899824261665344\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899826049804688\n",
      "Actor Loss: 2.670288097306184e-07\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9665912985801697\n",
      "Actor Loss: 1.0808309980347985e-06\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: -3.814697180359872e-08\n",
      "Critic Loss: 0.9654920101165771\n",
      "Actor Loss: -5.26165138126089e-07\n",
      "Critic Loss: 0.9899817705154419\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9756069183349609\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9665914177894592\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Epoch 45: Average Steps per Episode: 65.30, Average Reward: 28.50, Average Loss: 0.9703\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9899845719337463\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899815917015076\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9772498607635498\n",
      "Actor Loss: 4.334883385581634e-07\n",
      "Critic Loss: 0.9285699129104614\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.989983320236206\n",
      "Actor Loss: 2.288818308215923e-07\n",
      "Critic Loss: 0.9772498607635498\n",
      "Actor Loss: 6.068836455597193e-07\n",
      "Critic Loss: 0.9899826049804688\n",
      "Actor Loss: -3.814697180359872e-08\n",
      "Critic Loss: 0.9654921889305115\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899826049804688\n",
      "Actor Loss: 3.814697180359872e-08\n",
      "Critic Loss: 0.9756069183349609\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9833298921585083\n",
      "Actor Loss: -1.2715658215256553e-07\n",
      "Epoch 46: Average Steps per Episode: 65.45, Average Reward: 19.00, Average Loss: 0.9738\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9899815917015076\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9756069183349609\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.989983320236206\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899749755859375\n",
      "Actor Loss: -3.623962356869015e-07\n",
      "Critic Loss: 0.9899846315383911\n",
      "Actor Loss: 2.288818308215923e-07\n",
      "Critic Loss: 0.9833136796951294\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.989983320236206\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9756069183349609\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9833298921585083\n",
      "Actor Loss: -1.2715658215256553e-07\n",
      "Epoch 47: Average Steps per Episode: 66.18, Average Reward: 22.00, Average Loss: 0.9814\n",
      "Critic Loss: 0.9665914177894592\n",
      "Actor Loss: 2.161661996069597e-06\n",
      "Critic Loss: 0.989983320236206\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9285699129104614\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9756069183349609\n",
      "Actor Loss: 1.860827865129977e-07\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9756069183349609\n",
      "Actor Loss: -9.304139325649885e-08\n",
      "Critic Loss: 0.9787116050720215\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899815917015076\n",
      "Actor Loss: -4.577636616431846e-07\n",
      "Critic Loss: 0.9899850487709045\n",
      "Actor Loss: 3.814697180359872e-08\n",
      "Critic Loss: 0.9833298921585083\n",
      "Actor Loss: -6.357829107628277e-08\n",
      "Critic Loss: 0.9787115454673767\n",
      "Actor Loss: 8.116376903899436e-08\n",
      "Critic Loss: 0.9285699129104614\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Epoch 48: Average Steps per Episode: 66.17, Average Reward: 27.00, Average Loss: 0.9679\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -1.9073485191256623e-07\n",
      "Critic Loss: 0.9880809783935547\n",
      "Actor Loss: 4.5413063531896114e-08\n",
      "Critic Loss: 0.9833298921585083\n",
      "Actor Loss: -3.814697606685513e-07\n",
      "Critic Loss: 0.9899787902832031\n",
      "Actor Loss: -6.484985419774603e-07\n",
      "Critic Loss: 0.9772498607635498\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.989974856376648\n",
      "Actor Loss: 5.149840944795869e-07\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -1.220703097715159e-06\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9285698533058167\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.989983320236206\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9772498607635498\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9899749755859375\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Epoch 49: Average Steps per Episode: 66.86, Average Reward: 21.50, Average Loss: 0.9769\n",
      "Critic Loss: 0.989983320236206\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9787116050720215\n",
      "Actor Loss: 8.116376903899436e-08\n",
      "Critic Loss: 0.9899749755859375\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -9.91821252682712e-07\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899749755859375\n",
      "Actor Loss: 4.577636616431846e-07\n",
      "Critic Loss: 0.9880809783935547\n",
      "Actor Loss: 4.5413063531896114e-08\n",
      "Critic Loss: 0.9772498607635498\n",
      "Actor Loss: 3.9013951891320175e-07\n",
      "Critic Loss: 0.8570863604545593\n",
      "Actor Loss: 1.3623919130623108e-06\n",
      "Critic Loss: 0.9665912985801697\n",
      "Actor Loss: 1.5258790426742053e-06\n",
      "Critic Loss: 0.9787115454673767\n",
      "Actor Loss: 6.493101523119549e-07\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -1.220703097715159e-06\n",
      "Epoch 50: Average Steps per Episode: 66.30, Average Reward: 36.50, Average Loss: 0.9735\n",
      "Critic Loss: 0.8570860624313354\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: 1.074562590019923e-07\n",
      "Critic Loss: 0.989974856376648\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9772214293479919\n",
      "Actor Loss: 1.3004649872527807e-06\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -9.155273232863692e-07\n",
      "Critic Loss: 0.9799821376800537\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9899787902832031\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9772214293479919\n",
      "Actor Loss: 2.7743253667722456e-06\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -8.010864007701457e-07\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: 1.9073485191256623e-07\n",
      "Critic Loss: 0.9899750351905823\n",
      "Actor Loss: -9.536742595628311e-08\n",
      "Epoch 51: Average Steps per Episode: 65.92, Average Reward: 9.00, Average Loss: 0.9645\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -6.866454782539222e-07\n",
      "Critic Loss: 0.9799821376800537\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9899787902832031\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Critic Loss: 0.9787115454673767\n",
      "Actor Loss: 1.6232753807798872e-07\n",
      "Critic Loss: 0.9899750351905823\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.9772214293479919\n",
      "Actor Loss: 1.2137672911194386e-06\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9787115454673767\n",
      "Actor Loss: 1.6232753807798872e-07\n",
      "Critic Loss: 0.9899750351905823\n",
      "Actor Loss: -3.8146970382513246e-07\n",
      "Critic Loss: 0.9880810976028442\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -0.0\n",
      "Epoch 52: Average Steps per Episode: 65.67, Average Reward: 35.50, Average Loss: 0.9750\n",
      "Critic Loss: 0.9714174866676331\n",
      "Actor Loss: -4.904611046185892e-07\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.9722015857696533\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: 5.372812950099615e-08\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: 3.223687770059769e-07\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -9.155273232863692e-07\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9787116050720215\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.988081157207489\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -7.629394076502649e-07\n",
      "Critic Loss: 0.9787116050720215\n",
      "Actor Loss: 8.116376903899436e-08\n",
      "Critic Loss: 0.9899834990501404\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Epoch 53: Average Steps per Episode: 65.71, Average Reward: 22.00, Average Loss: 0.9621\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -1.1444091541079615e-07\n",
      "Critic Loss: 0.9285699129104614\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Critic Loss: 0.9722016453742981\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9799821376800537\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899821281433105\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9705671668052673\n",
      "Actor Loss: 4.4878791527480644e-07\n",
      "Critic Loss: 0.9899693131446838\n",
      "Actor Loss: -6.866454782539222e-07\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9285699129104614\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Critic Loss: 0.9799821376800537\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9899834990501404\n",
      "Actor Loss: -2.288818308215923e-07\n",
      "Epoch 54: Average Steps per Episode: 64.94, Average Reward: 31.00, Average Loss: 0.9636\n",
      "Critic Loss: 0.9799821376800537\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9807500243186951\n",
      "Actor Loss: 1.467191310666749e-07\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: 2.68640633294126e-07\n",
      "Critic Loss: 0.8570862412452698\n",
      "Actor Loss: -5.449567765936081e-07\n",
      "Critic Loss: 0.9772214889526367\n",
      "Actor Loss: 2.514232392059057e-06\n",
      "Critic Loss: 0.9714174866676331\n",
      "Actor Loss: 3.2697406027182296e-07\n",
      "Critic Loss: 0.9705671668052673\n",
      "Actor Loss: -4.4878791527480644e-07\n",
      "Critic Loss: 0.9722016453742981\n",
      "Actor Loss: -5.298190686175985e-08\n",
      "Critic Loss: 0.9829710721969604\n",
      "Actor Loss: 1.487085341977945e-06\n",
      "Critic Loss: 0.9807499647140503\n",
      "Actor Loss: 2.934382621333498e-07\n",
      "Critic Loss: 0.9859118461608887\n",
      "Actor Loss: -1.074562590019923e-07\n",
      "Epoch 55: Average Steps per Episode: 64.76, Average Reward: 8.00, Average Loss: 0.9585\n",
      "Critic Loss: 0.9705672860145569\n",
      "Actor Loss: 2.2439395763740322e-07\n",
      "Critic Loss: 0.9899834990501404\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9807500243186951\n",
      "Actor Loss: 2.934382621333498e-07\n",
      "Critic Loss: 0.9857105612754822\n",
      "Actor Loss: 1.6348703013591148e-07\n",
      "Critic Loss: 0.9857105612754822\n",
      "Actor Loss: 5.44956755277326e-08\n",
      "Critic Loss: 0.9772214889526367\n",
      "Actor Loss: 2.86102294921875e-06\n",
      "Critic Loss: 0.9829710721969604\n",
      "Actor Loss: -2.1983000806358177e-06\n",
      "Critic Loss: 0.9722016453742981\n",
      "Actor Loss: -3.7087335158503265e-07\n",
      "Critic Loss: 0.9799821376800537\n",
      "Actor Loss: 4.577636616431846e-07\n",
      "Critic Loss: 0.9807500243186951\n",
      "Actor Loss: -1.467191310666749e-07\n",
      "Critic Loss: 0.9899834990501404\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9829710721969604\n",
      "Actor Loss: -2.0043323729623808e-06\n",
      "Epoch 56: Average Steps per Episode: 65.27, Average Reward: 25.50, Average Loss: 0.9816\n",
      "Critic Loss: 0.9714174866676331\n",
      "Actor Loss: 2.179827021109304e-07\n",
      "Critic Loss: 0.9829710721969604\n",
      "Actor Loss: -6.465588597848182e-08\n",
      "Critic Loss: 0.9899834990501404\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9857105612754822\n",
      "Actor Loss: 1.6348703013591148e-07\n",
      "Critic Loss: 0.9899816513061523\n",
      "Actor Loss: -0.004595031496137381\n",
      "Critic Loss: 0.9899815320968628\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9687263369560242\n",
      "Actor Loss: 8.940696716308594e-08\n",
      "Critic Loss: 0.9857106804847717\n",
      "Actor Loss: 1.6348703013591148e-07\n",
      "Critic Loss: 0.9899819493293762\n",
      "Actor Loss: -0.02783103846013546\n",
      "Critic Loss: 0.9857109189033508\n",
      "Actor Loss: -1.089913510554652e-07\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: 3.0517577442878974e-07\n",
      "Critic Loss: 0.9705672860145569\n",
      "Actor Loss: 2.2439395763740322e-07\n",
      "Epoch 57: Average Steps per Episode: 65.28, Average Reward: 15.00, Average Loss: 0.9807\n",
      "Critic Loss: 0.9899842739105225\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9899818301200867\n",
      "Actor Loss: -0.04525810107588768\n",
      "Critic Loss: 0.9899853467941284\n",
      "Actor Loss: 7.629394360719743e-08\n",
      "Critic Loss: 0.9829691648483276\n",
      "Actor Loss: -3.879353016600362e-07\n",
      "Critic Loss: 0.9899815917015076\n",
      "Actor Loss: -0.04573989659547806\n",
      "Critic Loss: 0.9883559346199036\n",
      "Actor Loss: 3.548555582710833e-07\n",
      "Critic Loss: 0.9285699725151062\n",
      "Actor Loss: 0.024039270356297493\n",
      "Critic Loss: 0.9829697012901306\n",
      "Actor Loss: 9.051823894878908e-07\n",
      "Critic Loss: 0.9899815917015076\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Critic Loss: 0.9899851679801941\n",
      "Actor Loss: -1.9073485191256623e-07\n",
      "Critic Loss: 0.9285699725151062\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.9714175462722778\n",
      "Actor Loss: -5.44956755277326e-08\n",
      "Epoch 58: Average Steps per Episode: 65.88, Average Reward: 31.50, Average Loss: 0.9713\n",
      "Critic Loss: 0.9899817705154419\n",
      "Actor Loss: -0.04992298036813736\n",
      "Critic Loss: 0.9787108898162842\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9705671668052673\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9687262773513794\n",
      "Actor Loss: -5.960464477539063e-08\n",
      "Critic Loss: 0.9899818897247314\n",
      "Actor Loss: -0.04927566275000572\n",
      "Critic Loss: 0.9883561134338379\n",
      "Actor Loss: 8.871388956777082e-08\n",
      "Critic Loss: 0.9285699725151062\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9705672860145569\n",
      "Actor Loss: 4.4878791527480644e-07\n",
      "Critic Loss: 0.9899815320968628\n",
      "Actor Loss: -0.05060951039195061\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -9.155273232863692e-07\n",
      "Critic Loss: 0.9807496666908264\n",
      "Actor Loss: 1.467191310666749e-07\n",
      "Critic Loss: 0.9899855852127075\n",
      "Actor Loss: -3.0517577442878974e-07\n",
      "Epoch 59: Average Steps per Episode: 66.01, Average Reward: 17.50, Average Loss: 0.9655\n",
      "Critic Loss: 0.9899852275848389\n",
      "Actor Loss: 0.00501670828089118\n",
      "Critic Loss: 0.9899813532829285\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9687262773513794\n",
      "Actor Loss: -5.364418029785156e-07\n",
      "Critic Loss: 0.9899842739105225\n",
      "Actor Loss: -7.629394360719743e-08\n",
      "Critic Loss: 0.9899842739105225\n",
      "Actor Loss: -5.7220457705398076e-08\n",
      "Critic Loss: 0.9787108898162842\n",
      "Actor Loss: -1.6232753807798872e-07\n",
      "Critic Loss: 0.9829666614532471\n",
      "Actor Loss: 6.465588739956729e-07\n",
      "Critic Loss: 0.9807499051094055\n",
      "Actor Loss: 1.467191310666749e-07\n",
      "Critic Loss: 0.9899755120277405\n",
      "Actor Loss: -1.5258788721439487e-07\n",
      "Critic Loss: 0.9807497262954712\n",
      "Actor Loss: 2.934382621333498e-07\n",
      "Critic Loss: 0.9787110090255737\n",
      "Actor Loss: -1.6232753807798872e-07\n",
      "Critic Loss: 0.9899843335151672\n",
      "Actor Loss: 5.7220457705398076e-08\n",
      "Epoch 60: Average Steps per Episode: 66.08, Average Reward: 49.00, Average Loss: 0.9846\n",
      "Critic saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random2.pth\n",
      "Backbone saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random2.pth\n",
      "Actor saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random2.pth\n",
      "Finished training iteration 1/1. Model saved.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = None\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random2.pth\"\n",
    "backbone_load_path = None\n",
    "backbone_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random2.pth\"\n",
    "actor_load_path = None\n",
    "actor_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random2.pth\"\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = RandomPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "hidden_dim = 256\n",
    "\n",
    "# Actor parameters\n",
    "input_dim = 256\n",
    "hidden_dim = 256\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 60\n",
    "episodes_per_epoch = 2\n",
    "batch_size = 4\n",
    "learning_rate = 1e-4\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "simple_critic = False\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 12\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "    \n",
    "    if actor_load_path is not None:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim, model_path=actor_load_path)\n",
    "    else:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "    actor.to(device)\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "    \n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in actor.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone,\n",
    "                                    actor=actor, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=simple_critic\n",
    "                                    )\n",
    "\n",
    "\n",
    "    \n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    actor_num_parameters = count_parameters(actor)\n",
    "    print(f\"Number of trainable parameters in the Actor model: {actor_num_parameters}\")\n",
    "\n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path, actor_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwkAAAGsCAYAAACW1ev4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAACpVElEQVR4nOzdd3hUdd428HtaJr2H9E5ogdCRXqWDothwrSiW99lnVVRYd9eKDUVxd3XdfURBsbKiiILSayjSSyhppPc66ZnMzPvHzJlkIGVmMjW5P9fltcvM5MwvOSnne37fItJoNBoQERERERHpiO29ACIiIiIiciwMEoiIiIiIyACDBCIiIiIiMsAggYiIiIiIDDBIICIiIiIiAwwSiIiIiIjIAIMEIiIiIiIywCCBiIiIiIgMMEggIiIiIiIDDBKIiIiIiMhArwoSYmJiIBKJ2v1v6tSpN7y+qakJr732GhISEuDq6oqwsDA89thjKCkpsf3iiYiIiIhsRGrvBdiaj48Pnn766Rsej4mJMfi3Wq3Grbfeih07dmDs2LFYvHgx0tLSsG7dOuzZswfHjh1DUFCQbRZNRERERGRDIo1Go7H3ImxFCASysrK6fO369euxdOlSLFmyBF999RVEIhEA4N///jeefPJJPPbYY/jPf/5jxdUSEREREdkHg4QOjB8/HkePHkVWVhaio6P1j2s0GvTt2xfFxcUoLS2Fm5ublVZLRERERGQfvaomAdDWGWzYsAFvvvkmPvzwQxw/fvyG1zQ2NuL48ePo37+/QYAAACKRCDNnzkRdXR1Onjxpq2UTEREREdlMr6tJKCoqwsMPP2zw2OjRo/HNN98gPj4eAJCRkQG1Wo2EhIR2jyE8npaWhkmTJrX7mqamJjQ1Nen/rVarUVFRgYCAAH3qEhERERGRNWk0GtTU1CAsLAxisfH7A70qSHj44YcxadIkDB48GJ6enkhNTcX777+PjRs3YsaMGbhw4QK8vLxQXV0NQFvk3B5vb28A0L+uPW+99RZeffVVy38SREREREQmys3NRUREhNGv71VBwssvv2zw72HDhuGLL74AAGzcuBGffPIJli9fbpH3euGFFwyOVV1djaioKFy7dg1eXl4WeQ9jKZVK7Nu3D9OmTYNMJrPpe5NxeI6cA8+T4+M5cg48T46P58g5GHOeampqEBsba/L1Z68KEjry+OOPY+PGjUhOTsby5cv1Owgd7RQoFAoAHe80AIBcLodcLr/hcX9/f/1OhK0olUq4u7sjICCAP+gOiufIOfA8OT6eI+fA8+T4eI6cgzHnSXjc1HT3Xle43J7AwEAAQF1dHQAgLi4OYrEYaWlp7b5eeLyjmgUiIiIiImfGIAHQdzgSWqS6ublhzJgxuHr1KrKzsw1eq9FosGvXLnh4eGDUqFG2XioRERERkdX1miDhypUrqK+vb/fxlStXAgDuvfde/eOPPfYYAG1tQdtREv/5z3+QmZmJP/zhD5yRQEREREQ9Uq+pSfj222/x/vvvY/LkyYiOjoaHhwdSU1Oxfft2KJVKvPDCC5g8ebL+9Q8++CC+++47fPPNN7h27RqmTJmC9PR0/PDDD4iNjcXrr79ux8+GiIiIiMh6ek2QMG3aNFy+fBlnzpzBoUOHUF9fj8DAQMybNw//7//9P8yaNcvg9WKxGD/99BPefvttbNy4EWvXroW/vz8eeeQRvP766wgKCrLTZ0JEREREZF29JkiYMmUKpkyZYtLHyOVyvPzyyze0TiUiIiIi6sl6TU0CEREREREZh0ECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGpPZeABGRM9BoNPjqZDYa69X2XgoREZHVcSeBiMgIOy4X4uGvjuO9C432XgoREZHVMUggvSalCpX1TfZeBpFDOpFdDgBIV6iRV1Vv59UQERFZF4ME0rvz00OIenEL0koU9l4KkcO5WFil//+/XSq030KIiIhsgEECAQBqGpXYllKA2qYWbPz9mr2XQ+RwLhZU6///9ssMEoiIqGdjkEAAgONZZVBrNACA705nQ6P7/0SkTcVLbbPDtje1GI1KlR1XREREZF0MEggAkJxZqv//qSU1OJ9fZb/FEDmY1BIFWtQa+LjKECAXob5Zhf1pxfZeFhERkdUwSCAAwOEMbZDgKpMA0O4mEJHWxUJtqlFiqA9GBmp/RrZdzLfnkoiIiKyKQQKhRaXG0WtlAIBnpw8AAGxiyhGR3sWCKgBAYog3RgVqx8v8kpLPnxEiIuqxGCQQzuVXoq65BT5uMqycmQg3mQQZZbU4nVth76UROYS2OwlJARLIpWJkldfhchE7gRERUc/EIIH0qUYT4oLg5SrDgsHhAIBNp3PsuSwihyG0P00M9YGrRISpffsAALalMOWIiIh6JgYJhMO6ouWJcUEAgLtGRAMANp1hyhFRXVMLMstqAQCDQrwBAHMHhQJgXQIREfVcDBJ6OY1Gg8MZJQCAifHau6PzEsPg4SJFVnmdfsosUW91qUibahTs5YogT1cArUHC4cxSTiknIqIeiUFCL3etvBZFika4SMUYHR0AAHB3kWLhEG3KEbscUW8nFC0PDvPVPxYb4IlBIT5QqTXYycFqRETUAzFI6OWEeoRRUf769qcAcLcu5ei/Z3KgVjPliHovoR5hcKiPwePzB4cBALalFNh6SURERFbHIKGXa1u03NacQWHwcpUit7Iex7LK7LE0IodwsUCbbtR2JwGAvsB/e0oBVGq1rZdFRERkVQwSernDmbp6hLg+Bo+7yiS4dUgEAO3MBKLe6oKQbhTqa/D4+Ngg+Lq5oLyuCb9nsXaHiIh6FgYJvVhZbaO+z/v4uMAbnr+LKUfUy5XXNqFQ0QAAGBRimG4klYgxe6CuyxFboRIRUQ/DIKEXO5KpTSMaGOKNQF3XlrZmDQiFj5sMBdUNSNa1SSXqTVKKqgAA0f4e8HaT3fC8UJfwy0XWJRARUc/CIKEX6yjVSCCXSbAoKRIAU47IenZcKkD4X3/Azxfy7L2UG+jrEa4rWhbMHRQGkUg7tTyvst6WSyMiIrIqBgm9mLA7MDE+qMPX3D0iCgDw/dkcs4ozPz+WiTkf7UV5LXvJU/u+PZWNguoGh2y3q+9sdF3RsiDQ0xVjY7SpetuZckRERD0Ig4ReqqG5BSeyKwB0HiTM6B8CP3cXFCkacSjdtJSjA2nFWPrVMey4XIitDniXmBzDeV1h8NVihX0X0g5hRsKQDoIEAJifqO1y9AunLxMRUQ/CIKGXOplTAaVKjVBvN8QGeHb4OhepBLcP1aYcmXKnt1jRgCXrk6HWaAue86sdOxWjsr4JHx64itKaRnsvpVdpUamRortbf6VYAY3GcQrkNRoNLhYK6Ua+Hb5OaIW6J7UIjUqVLZZGRERkdQwSeqnDGdp6hAnxQRCJRJ2+VuhytPlsDlpUXaccqdRq3LshWd8VBoDD52t/dDAV//vfk1i965K9l9KrpJXWoKlF+z1V29SCwuqGLj7CdgqrG1BZ3wyJWIT+wd4dvi4p3BcRvu6ob1Zhf1qxDVdIRERkPQwSeqnDQj1CXMepRoLp/YIR4CFHaW2TURdBr26/gL2pxfBwkWL59AEAgLwqxw4SssrrAABn8yvtvJLeRZhBILjiQClHwi5CQpCXwTTy64lEIsxLFLocMeWIiIh6BgYJvZBarTGqaFkglYixeJjQ5Sin09fuuFSA13dcBAD835IxuLl/CAAgr8px7hC3p0SXZnSluNrOK+ldzl8XlDlUkCAMUeukHkEgpBxtS8l3qJQpIiIiczFI6IVSCqtQ3aCEh4sUQ8P9jPqYu9ukHCk7SDnKq6zHfV8cgUYDPDExAfeOjkWEr7v2OQffSSjWBQn5VQ1QNCjtvJre43x+FQDA21U7g+BqiQMFCUJnow7an7Y1vV8I5FIxssrr9AMKiYiInFmvDxJWr14NkUgEkUiEY8eO3fC8QqHA8uXLER0dDblcjpiYGDz//POora21w2otQ9hFGBcbCKnEuG+ByX37oI+XKyrqm7H3atENzytVaty9/hDKapswItIfaxePBABE+GmDhPK6Jocu6iypbS1Y5m6C7QidjW5NigAAXHGgC2xjipYFHnIppvULBgD8cpGdvIiIyPn16iDh4sWLePnll+Hh4dHu83V1dZgyZQrWrl2LAQMG4JlnnkH//v2xZs0aTJ8+HY2NztkJ57AJqUYCqUSMO4SUozM3phy98NNZHMksg4+bDP99ZKI+h9vXzQXuLtr/n+/Auwklbboa8U6wbVQ3NCO7QlsLcudw7TwOR9lJUKs1+q5LxqQbAcCCRCHliNOXiYjI+fXaIEGpVOLBBx/EsGHDcNttt7X7mnfeeQdnz57FypUrsWPHDrz99tvYsWMHVq5ciRMnTmDt2rU2XrVlHM4QgoT2Jy13ROhy9MPZXDS3tO4K/HQ+F+/tvQwAWH/fOMQFeumfE4lECPdx7JSjuqYW1De3fj6XuZNgE0LOf6Sfu34gWXZFHeqbW+y4Kq2silrUN6sgl4oRH9hxi+C25uvqEpIzS1FZz+GBRETk3HptkPDGG28gJSUFn332GSSSGzuXaDQarFu3Dp6ennjxxRcNnnvxxRfh6emJdevW2Wq5FpNbWYfsijpIxCLcFBNg0sdOjA9CqLcbqhqasVuXcpRZVoMHNx4FACyfPgC36WYqtCWkHDlqkFBcY1hUzZ0E2zjfZlBZoKcc/u4uAIC0kho7rkrrYoE2UBwY4mN0Sl5MgCcGhfhApdZgx6VCay6PiIjI6nplkHD69Gm88cYbePnllzFo0KB2X5OWloaCggJMmDDhhnQkDw8PTJgwAZmZmcjNzbXFki0mWbeLMCzcD55ymUkfKxGLcYcuLeS7U9loVKpw56eHUd2gxLjYQLx96/B2P87Ri5dLagzv+l4u4k6CLQhFy0lhvhCJWmcROEJNiClFy23NH6xthbotha1QiYjIufW6IKGpqQkPPPAAhg0bhhUrVnT4urS0NABAQkJCu88Ljwuvcxbm1CO0dfcIbZCw5Xwe/rjpBE7nViDAQ47vlk6ErIM7rkKQkO+gbVCFegRhnRlltWhy4CLrnuJ8gbb9aZKuw9YAXZBw1SF2EqoAGF+PIBBaof56qRAqddeDB4mIiByV1N4LsLWXXnoJaWlpOHXqVLtpRoLqau3dTB+f9u8kent7G7zuek1NTWhqar1DrVBoU1iUSiWUStu22BTeT6lU4lC6dtLy2Gh/s9YxKsIX4T5uyK9uwKdHMyASARv+MAYhni4dHi/USw4AyKmotfnnbozCam3xbFKYDxSNSigalbhUWGnyXeTuaHuOegONRoMLup2EgX08oVQqkaDL/b9UWGX3r4Mw5G1AkKfBWro6T6MjfOHrJkN5XROS04sxLjbQ6mslQ73tZ8lZ8Tw5Pp4j52DMeTL3HPaqIOHo0aNYs2YNXnnlFQwePNiq7/XWW2/h1VdfveHxnTt3wt3d3arv3ZEt23fiQoH2grjh2jlsL7hg1nFG+rQgXxcb3REjg+raGWy/dqbD1xeWaAtRL2UXYvv27Wa9pzUdymwGADRVlSJEroaiEfjmtwOYEGz7H49du3bZ/D3tobhBjZqmFkhFQMapw8gWi1Cj+z45kZZn1++TFrUGV4q0PyclV05je/bZG17T2Xka7K3B4Qbgnz8fQmVfubWWSV3oLT9Lzo7nyfHxHDmHzs5Tfb156d69JkhoaWnBgw8+iKSkJPz5z3/u8vXCDkJHOwXCzkBHOw0vvPACli9fbvD6yMhIzJo1S78LYStKpRK7du2CR/xQaPYfQXygJ+67bZ7Zx4saXo3f3tuF6f364ItHJ0Ii7jxrLSS3Am+d241auGDePPPf11p2/3AGyEjDiIF9EV7TiNTfs+AWFo95sxNttgbhHM2cORMymWm1IrZ0PKsc7++7ivdvG4ZwX/OD3Z8v5gOHkzEo1Be3LJgFAIgvUeCtc7+hqEmEOXPmQiwWWWrZJkkprEbLnh3wlEvx4O3zIRK1rsOY81TZJxuHvzqO1CYPzJs3y1bLJh1n+Vnq7XieHB/PkXMw5jwJ16ym6jVBQm1trb5+wMXFpd3XjBs3DgDw448/6guaO6o56KpmQS6XQy6/8S6iTCaz2w/b8RxtDvjE+D7dWsPwqECUrb4DHi5Soy7kYgK1QVFRTSMglnRYu2AvZfXanYQwXw8EeroCAFJLa+1ynuz5/WGMV35LwZ6rRejbxwvvLBph9nEuFWvrDoZG+Ok/3/4hfpCKRahvVqGkXolIv/bnl1jb1VLtoMTBob4d/q7o7DwtGBIBkeg4zhdUobhWqe/uRbbl6D9LpMXz5Ph4jpxDZ+fJ3PPXa4IEuVyORx55pN3nDh48iLS0NNxyyy0ICgpCTEwMEhISEBYWhuTkZNTV1Rl0OKqrq0NycjJiY2MRGXljy09HdeRaGQDzi5bb8nI1/hsuyNMVMokYSpUaRYoGu138dUQoXO7jKdd/XmyDeqNGpUo/rftAWkm3jtW2s5FAJhEjPsgLV4sVuFqssNv3iTBpeYiJRcuCQE9XjIsNxJHMMnx18hpWzrTdjhQREZGl9Jogwc3NrcO5Bg899BDS0tLwwgsvYOzYsfrHH330Ubz22mtYtWoV3n77bf3jq1atQm1tLf7yl79Yfd2WolRr8Ht2BQBgQlz3gwRTiMUihPm4IbuiDnmV9Y4bJHi5IiZAWzx7tUQBlVrdZSpVb3IksxSNuq5Pp3IrUNOoNClYbEsoDBY6GwkGBHvjarECV4oVuHlAaLfWa67WzkbmF64/Oq4vjmSW4aODqXh2+kCjZy30BBqNBmfyKpEY4gO5rOPmEERE5Nh6z18uM6xYsQJDhw7F6tWrMXv2bLzwwguYPXs2Vq9ejdGjR+Ppp5+29xKNdq1GjQalCgEecn2rSVty5FkJxW2ChNgAD8ilYjQqVciuqLPzyrpHpVajptFyXSn26AboaY+twRHdroKpGppbkKprc3r93fr+fXRtUIvtt5Mj7CQMDvU1+xhLRsUgyFOO3Mp6/HjOuWapdNdP5/MwcvWvWP7DKXsvhYiIuoFBQic8PDxw4MABPP3007h8+TLee+89XLlyBc8++yz27NkDNzc3ey/RaJcqtXeAJ8QFGRRi2oqjBgkqtRplddpWtcFerpCIxeinu1B19pSjP246iYCV3+NUTrlFjrc3tRgA4OOm3T04kG5eytGlomqoNRoEesoR4u1q8NwA/UA1+3zt65tbkFGmDWBMnZHQlqtMgicmauuV/r7/qiWW5jQO6wY2fnMqGy0qzoogInJWDBIAbNiwARqNxiDVSODj44O1a9ciJycHzc3NyM7Oxpo1a+Dl5WWHlZrvcpU2SLBEPYI5HDVIKK9rhkYDiERAgIe20Fy4UHXmycvVDc1YfywDSpUaG3+/1u3jKRqUOKELNp6ZNgAAcCC92KxjXT9pua3+dg4SLhdVQ6MBgjzl6OPl2vUHdOLJSf0gk4iRnFlqsUDNGaSWaM9dZX2zvoaFiIicD4OEXkCj0eBKlfaO3kQb1yMIhA4vjjZ1WahHCPCQ6/PGB4Zoc9Ev2zHlpbt+PJeLphbtOf/lYj40Gk23jncgvRgqtQZ9g7xw/5hYAMCJ7ArUNbWYfKyO6hGA1nSjvKp61DbZfoDPxQJdqlE3dhEEoT5uuEs3obw37SaklrZOzN56Ic+OKyEiou5gkNALpJXWolqpgatMghGR/nZZQ7iPNjXL0XYS9PUInq13jQf2gJ2Er09m6f9/RlmtvgbAXEKq0fR+wYgN8ESErzuUKjWOXjP9TvF5XZDQXvegAE85Aj21OzrdXbM5LhZWAehePUJbT03V7rp8eyobRQrHCpCtoUWlRmZZrf7fWy90P0AlIiL7YJDQCxzRXciNjvK3W7cRR003atvZSKDfSShSOOUFTpGiAXuuai/qhdSpbRfzu3VMoWh5Rv8QiEQiTEnoA8D0ugSNRoNz+dp5HUkd3K0X1myP4uXWIMH8zkZtjY4OwLjYQChVanx8qP2ZKz1JdkUdlCo15FIxXKRipJfW2LUInYiIzNdrWqD2ZsmZ2vkI42MD7baG1nSjeqjVGrtN072eECQEtwkS+vXxgkgEVDU0o6SmEcHezlOgDgCbTmdDrdHgppgALBkZg6c3n8K2lHwsnzHQrOOV1DTqU4SmJQQDAKYmBOOrE1k4kGZaXUJxTSPKapsgFokwqIML8QHB3jicUWqXugRLphsJnp42AEevHca/D6fhL7MSe3RbUKEeISHICxF+7vjtUiG2XsjDgBDLBF1EnTmbV6GveeqKm0yChUMi4NqDfx6JuotBQi+QfM3+QUKItxvEIhFa1BqU1DYixEEuvNvbSXBzkSI2wBOZZbW4XKRwuiDhm5PZAIAlI2Mwf3A4nt58CgfTS1Dd0Awft/YnCHdmr24XISncF0G6r9OUvtqdhOPZ5WhoboGbi3G/Si7o/oAn9PGCewcfY682qFX1zfqdrkQL7SQAwG1DIxHh6468qnp8eyobD46Ns9ixHU2arh4hoY83Zg4I0QUJ+VjBgXJkZVX1zRj33k79LBdjrLh5EFYvGm7FVRE5N6Yb9XDFigakl9ZCBGBsTIDd1iGTiPXtLvMdKOWodUaC3OBxfV1CsXPVJWSW1eBYVhnEIhHuGhGNvkFe6NfHCy1qDXZdKer6AO0Q6hFm9AvRP9Y3yAuh3m5oblHjeJbxnXuEeoSOUo0A+7VBTdGlGkX6uZsVTHVEJhHjfyb3AwD8ff8Vp0xhM5ZQR9KvjxcWDA4HoE13LNX9nBFZS3ppDRqVKrjJJJg9MLTT/4Quf58ezUCTCUEFUW/DIKGHE1oQRnmK4eduuQsfc4Q7YF1CSTuFy4BhXYIzEXYRpvcLRqiuWFy4WDO3LmFPams9gsCwLsH4lKPzunqE9oqWBUIb1NQSBdRq211QW2KIWkeWje8LN5kEZ/Iq9XMEeiIh3ahfH29E+nlgeIQfNBpgW0r3amKIupJTqR1+OTTcD7/9z/RO/9v3p5sR4euO8rqmXjfskMgUDBJ6uGvldZCIRRjoa/9TrS9ernSgIKFWV5Pg3VGQ4Dw7CRqNBl+d1M5EuHdUjP7x+YnaIGH7pQKTL7qzymuRWVYLiViEyboUI4GQcmRK8bJ+JyHct8PXxAZ4QiYRo0GpQm6V7aZeX9StbXCY5fPnAzzluG+0tnXs3/dfsfjxHYU+3ShIO0fmliERAICfLzBIIOvK0f1difJ37/K1UokYS8fFAwA+OZJu1XUROTP7XzmSVT07YyBK31yEe+Lsu4sAOGaHo5Ia7bTlG3YSnDDd6Hx+FS4XKSCXinH7sEj94xPjg+DlKkVJTSNOmjjUS0g1GhMdAC9XmcFzU3RFzEevlRm1Za9UqXFJF3Qlhd04I0EglYjRN8gTAHDFhjs51txJAIA/Te0PAPjxXB6yymu7eLXzaVSqkF2hDer69dEFCUnaIGHH5UKTcsWJTJWj+96L8vMw6vWPjIuHSKT9HZdeavt2y0TOgEFCL+Apl8FXbv9THeHreLMS2itcBlp3EvKrGqBosP1QL3N8cyoLgHbnoG1OvYtUglkDQgEA21IKTDpm29an1xsQ7I0+Xq5oVKr005g7k1ZSg+YWNbxcpYj27/wP+YBg7df/aoltggSNRqPv4GTJzkZtDQ7zxc39Q6DWaPDRwVSrvIc9ZZTVQKMBvF1l+p+n4RF+CPd1Q11zC/almlcTQ2QMId3I2CAhyt8DcwaGAQDWcTeBqF32v3KkXkPYScivdoyhUnVNLahr1k4Mvj5I8HV30RdaX3GC3QS1WoNvdAPUlrRJNRKYU5eg0WiwV6hH6HdjkCAStaYgHUjrOuVIqEcYHOrbZQtcWxcvF9c0oryuCSJR6y6SNTyl201YdyTDrGnVjiytpDXVSCTSnl+RSISFg7W7CVuZckRWZEq6keCxCX0BAOuPZaK5hTtdRNdjkEA2E+5gNQnCLoKbTAJP+Y3tOJ2pePnItVLkVNbDy1WK+YlhNzw/d5D2sVO5FSg0Mki7VFSNIkUjXGUSjO2gfa5Ql7DfiHkJxnQ2Eti6DapQj9A30Mvodq7mmJcYjr5BXqhqaMYXv2da7X3sobVo2cvg8VuGaAPUrRfyenRnJ7KvXN1OQqSROwkAMH9wOEK8XVFS08i6GaJ2MEggm2lbk+AIFwtC0XIfL1f9nc+2nKku4WvdLsLtQ6PavcgN9nbD6GhtC9ztRnaa2aub2jwxLqjDgUNCkHDkWimUKnWnxxN2EpLCO65HEAwIse1OglCP0FnXJUsQi0X43ynadqj/2H/Vpt2brK21/anhTsy0fiHwcJGioLoBp3Mr7LE06uGalCoUKbS/z6P8jN9JkEnEWDpWW8D8f8lMOSK6HoMEshlhJ6FBqUJlfbOdV9NxPYLAWXYSlCo1Np3OAWDY1eh6+pQjI+sS2mt9er3EUF/4u7ugvlmFk9md1yVcMKKzkUDYSSiobkBNo/VrQi5auR6hrYduioeXqxRXihXYdaXQ6u9nK62D1Ax3Elx1fesBdjki6xDq3NxkEgR4yLt4taFHxmuDhF1XC3GtrOc1FCDqDgYJZDOuMgkCPbW/wB2heFk/SM2z/T8qztIGdfeVQpTXNaGPlyum9wvu8HVCGtKuK4VddiNqUan1KUSdBQlisUjf5aizVqhV9c36nGFjugf5ursgWBe82SLl6KJukNpgC05a7oi3m0x/9/KDHtQOte2MhOsJrVC3Xsiz6Zqod9AXLft7tLsr3Jm4QC/c3D8EGg3w6VHuJhC1xSCBbEpfvOwAQYKwkxDs7dbu80K6UUZZrUNP5RRSje4aHgWppOMf6eER/gjxdkVtUwsOdjHb4ExeJaoblPBxk2FEZOfpQa3zEjquSxB2EaL83OFr5FC//jYqXlarNUgR2p/aYCcBAP53Sn+IRMBvlwpxxcGDUGMoGpT6dA9hRkJb8xLDIBaJcCavUp87TmQpORW6omUTUo3aEgqYPzuaiZYu0iaJehMGCWRT4T5CG1T7dzgq6WInIdTHDd6uMqg1Gn0qhaOpb27Bj+e0d2c7SzUCtHf95yUKKUedp30IrU+nJgRDIu7814QweflwRmmHf2BNqUcQCB2OrN0GNaeyDrVNLXCRitG3nQtca4gP8tJ3/fnngas2eU9rSi/T/nwEecrbDQKDvFwxTlf8zpQjsjRT259e79akCAR5ylGoaOB0cKI2GCSQTTnSQLWuahJEIhEG6gpoHTXl6OcLeahrbkFMgEeHHYjaMrYuQT8foZ3Wp9cbEuYLXzcX1Da14ExeZbuvMaWzkcBWbVCFeoQBwd6QdbITY2lCO9TPj19DlQPU6HRHanHHqUaCtl2OiCypbbqROVykEjw0Ng4A8AkLmIn0GCSQTUX4OU6QUNxFkAAAA3VDvS7bqMuOqb45lQ0AuHdkjFG5uDf3D4FMIkZ6aY3+wu56jUoVDmeWAgCm9++4xkEgEYsxKT4IAHCgg1aophQtC4R0I3NqEjQaDfanFiM5owTZFbWddl6y9qTljkzrF4zBoT6oa27Bp0czbPrelpZaKnQ26ngnRqhL2JdWbJNidOo9uptuBACPjtOmHP16qZApcUQ61msITtSOCAealdCabtRJkODAOwmV9U3YrtsRuHd0jFEf4+Uqw5S+fbD7ahF+uZiP5e0MDjt2rQyNShVCvF0xKMS4Qt4pCcH4+WI+DqSX4LmbBxk8p1a3TjMeEmZ6ulFqiQIqtbrLtKe2PjuagUe/Pq7/t0gEBHu5IsLXvfU/P+3/CgPjbFG03JZIJMJT0wZg2dfH8f7ey3h0fLzBpGxnkqZLCWuvHkHQP9gbCUFeSCutwc7LhVg8PMpWy6MerrvpRgDQL9gbUxOCsT+tGJ8dzcDL85IstTwip8WdBLKp1qnLDhAk1DYBAIK9OwsSHLcN6uazuVCq1EgK90WiCXfBW1OO2s+9FVqfTu8XYnSnEKF4+WB6CVRqw7v2WRW1+pz/zu40Xy/a3wNyqRhNLWr9nUJjaDQafHgwFYA2R95FKoZGAxQpGnEypwJbzufhw4Op+PNPZ3Hf50ew64ouSLBR0XJb942ORd8gLxRUN2DlljM2f39L6WhGQlsikYhdjsjiNBpNt9ONBMt07VA/PZpxw+8xot6IQQLZVLiD1CSo1GqU6YKEztKN2hbPOtofDaGr0ZKRMSZ93HxdkHAwvQSKhhvTPoR6hM7aqV5vWIQfvFylUDQqcS6/yuC587p/J4b4dNp96XoSsVh/Z/qKCQPtTuVU4GxeJeRSMS7/bSEa196DkrcW4/TKudj6+BT86+7ReGFWIu4fE4tp/YKREOSF8XGBmGbC52sprjIJ1t17EwDgP8np2J/a9eRqR6PRaNoECZ0HgQuHtNbEsIsMWUJFXTPqm7Xd54SbUOa6fVgU/N1dkFtZjx2Xe84MEyJzMUggmxJ+iVc3KO2al1xe1wy1bupzYCfDd2IDPOEiFaNRqUJ2hePkqeZX1evnGNwzMtqkj+0b5IV+fbzQotZg53XDvGoalfhdNxSts/kI15NKxJgYp2uFel1dQms9gvGpRgJz2qB+ckRbeLh4WBQCPOUQiUQI8nLF8Eh/LBwSgScn9cObtwzDFw+Mx94/3YzUl29B8vLZ8JTLTF6fJUxJCMYTExMAAI9+fQz1zS12WYe5yuuaUNWgLbyO76I71IS4IPi5u6C8rglHr5XZYnnUwwm7CMFerh1OhjeWq0yCB27SFjBzAjMRgwSyMS9XGbxdtRdj9pyVINQjBHjIO727LZWI0U934eNIKUebTmdDo9FedMUEeJr88fqUo4uGKUfadCEN4gI9TT6u0Ar1+qFq5/X1CL4mr3OAicXLtU1K/Q6L0PvcGay+dTgifN2RUVaLF385Z+/lmETYRYj0c4e7S+dlblKJWD/UjylHZAmWSjUSLBuv/b3xy8V8FDhAgw0ie2KQQDbnCG1Q9YPUOkk1Egh1CdZuxWkK4UK4q9kIHZmvm5ew/VIB1GqN/nFTWp9eb0pfbbrOofQSg2PqZySYFSSY9rX/9lQ2apta0K+PFybr6iScgbebDP9ZMgYA8MG+qzie5Tx32dOMqEdoi3UJZElCvVJkN1ONBINCfTAhLggqtQbrj2Va5JhEzopBAtlc69Rl+w1U62pGQlutxcuO0eEotViBkzkVkIhFuNPMDjET44Pg5SpFSU0jTuVW6B9vLVo2PT9/ZJQ/PFykqKhvxsXCKgDaYW/CIDpT2p8K+ps4UE1IEVg2vq/RRdeOYl5iOO4bHQO1RoNHvjrm0FO+20o1orNRW7MHhkEmESO1pMas9rZEbVl6JwFo3YVcdzTd4IYHUW/DIIFsLtxXmLpsx52EWiFI6LgeQTBQd6F62YTiWWv65lQWAGDWgFAEGRHktMdFKsGsAaEAtNvqAFBa06gvMp5uQj2CQCYRY0KcMC9Bm3J0qbAaGo02GAv2djP5mP11d6eLFI2obuh84NjZvAqcyC6HTCLGg7q8YmfzweJR6OPlipTCary5M8XeyzGKMTMS2vJ2k2GqLjXtZ+4mUDflVnZ/RsL17hgeBR83GbLK67Bbt7tK1BsxSCCbc4R0o2JF1zMSBG3boGo0pt1VMvX1xhxP39VolGkFy9e7vi5hn67geEiYr1E7LO1prUvQHqs79QiA9oIyVBdcdHXXWZiUetvQCLODJ3sL8JTjwztHAQDe3HFRn6rlyIR0I2N3EoCenXKkVmt499mGLDEj4XruLlLcPzoWQGsjBKLeiEEC2ZwjBAnCTkJnMxIE/fp4QSQCKuub9WlKxth7tQg+z2/C+3sum73O653Nq0RqSQ1cZRIsSors1rHmDtIWkJ7KrUBhdYNZrU+v13Zegkaj6VY9gmBASNcdjuqbW/DliSwArYWHzuqO4VFYlBSBFrUGS7885tCtQtVqDdJKtefF2JoEoLUVanJmGcpqjf+ZcnQ/nstF1Es/YvIHuyx+g4DaZ410IwBYpks52nIuF8UK+6XGEtkTgwSyuQg/BwgSanQzEozYSXBzkSJW1+nH2A5HzS0qPPnd76hpbMG6o5a7EyX07p49MBRert1r2Rns7YbR0QEAgO0p+dir69FvSuvT642ODoCbTILS2iZcLlLodxLMqUcQDDCiDeqm09lQNCoRF+iJ6WYUXTsSkUiEf909Br5uLjiVW4G1+67Ye0kdKqhuQH2zChKxCLGBxnfDivb3xNBwP6g1Gv3UcGdWWtOIuz87hNs/OYj8qgYkZ5aisLpnXFjmVtZh4vs78e9DqfZeyg2UKjUKdF9nS6YbAdqWzTfFBKBFrcHnx1nATL0TgwSyudbCZft3NzI2rcbUuoSPD6XpW0NeLlKgyEJ3ovbpLuS7c7e/LSHl6N+H05BeWgOJWKTvUmQOF6kE42IDAQD704r1NQ7mzEgQCHUJnaUbCSkBj46Lh1jsXAXL7Qn1ccP7t48AALy07TxSHbTAV9hFiA3whMyEQXkAcItuN8GZU440Gg2+PZmFQW/8gk2ncyARi+ChawMrBMjO7tXtF5CcWYq3d12y91JukF9VD40GkEvFCDLiho+phF3JT45kcGeIeiUGCWRz4T7aIKG0tgmNdurgUmxqkNCmLqEr5bVNeGX7BQDQXzhZYpJuc4sKhzO1BcHTEiwTJAg960/maDscjY4KgLdb93YopujW9u2pLJTXNUEsEmGQ7utnjq52ElIKq3AkswxSsQgPj4s3+30czUNj4zBzQAgalSo8+vUxh8xzN3bScnuEuoQdlwudMuWosLoBt39yEEs2JKOstglJ4b44/txszB+s/Zm60AOChNzKOnzx+zUAQHZFnV1v7LRHSDWK9POwys2Bu0dGQy4VI720BtfKay1+fCJHxyCBbM7fw0U/GbOg2j5/dEyZkwCY1gb1le3nUdXQjKRwXzw5STtJd19a94OEE9nlqG9WIdBTjsRQ324fDwCGR/gjpE1dhiV2KIS6hEMZpQC0F5DdmYQqtEFNL6tpNz9fKFheOCQCIWZ0UHJUIpEI/7fkJni4SHEooxT/Ppxm7yXdQGh/ako9gmBEpD8i/dxR29SCvq9uxTu7UtDgBNOmNRoNvjieicQ3fsGW83mQikV4Zd4QnHh+DkZGBSApTLtr5gxF5115d/dlKNv8zB3JLLXjam4kzEiwdKqRwFMuQ5wujS6jlEEC9T5GBQlisRgSicTk/6TSzqdvUu8kEonsWrxc19SCOt3FiKXTjS4VVuNj3cXcB4tH4mZdfv8+C+wk7Ne1FZ2aEGyxu2ZisQjzdIPVgO7VIwhuigmEXNr6q6U79QiAtmuJq0yC5hY1sirqDJ5rVKr0dzqdacKysWICPPHWLcMAACt/OoOc6z5/ezOns5FALBbhh0cnY2i4H6oblFj501n0X/UzNv6e6ZC7JgCQV1mPBf/ejwc3HkVlfTNGRvrj1Mq5eHleElyk2kBY+H539nSjkppGfRqf0J3syDXHGvJnjc5G14sP1H5vZ5TVWO09iByVUUHC5MmTb/hvyJAh0Gg0EIvFiI6OxpgxYxAdHQ2xWAyNRoMhQ4Zg0qRJ1l4/OSl9kFBp+yChVJfa4CqTwFNuXCAr7CTkVzVA0aDs8HXP/ngKKrUGi5IiMK1fCCb37QOxSIS00ppub9ULuxFTLTxJWKhLkEvFGK+bc9AdrjIJbooJ1P9buLNqLrFYpE9nub4uYfPZHFTWNyPKzx0zBzh3wXJH/mdyP0yIC0JtUwse//a4Q+VGmzoj4XqjogNwauUcfH7/OET6uSO3sh4PfHEUI9/5Fbt0RfqOQKPR4JPkdCS+8Qu2pxTARSrGW7cMw7HnZt9QbyNcUF8uUhjchXc2a/deRqNShTHRAVg5cxAAINnhdhKs09moLWEnIZPpRtQLGRUk7N+/H/v27dP/t3HjRlRWVuLee+9FRkYGMjMzcfToUWRmZiIjIwNLlixBVVUVNm7caO31k5MSBqrl26EDiL4ewVNu9FReX3cXfVrOlQ52E35Nycdvlwohk4jx7m3aolMfNxeMiNReROzvRspRk1Kl/wM9zcLde+YnhmHpuHisuW1Et9KC2prSJpDp7k4C0HFdgjBh+dHxfSER98zsSbFYhE//MBZyqRi/XSrElvOOUejbolIjQx8kmJ5uJJCIxXjgpjhcfXEh3r51GLxdZTibV4lZH+3FnI/2OkTazgtbz+Kxb45D0ajE2JhAnP3zPPx5ViKk7RRrR/t7wMtVCqVK7bQTpSvrm/CRrpvRX2cP1g9JPJNbgXoHSgnLscIgtevFM92IejGz/qo+99xzCA0NxZdffomoqCiD56KiovDVV18hJCQEzz//vEUWST2PPdONTO1sJOiseFmpUmP5D6cBAE9N7Y++bdIvpuny/LuTcnQ8uwyNShWCvVwxMMT8C7L2uEgl+PQPY/HHKf0tdsypbQqrzR2k1taAYO3Xvu1F19ViBQ6ml0AsEuHhsT2nYLk9/YO98cfJ2vOz+WyOnVejlV1Rhxa1Bq4yif7nuTvcXKRYOTMRGa/cgqem9odMIsaOy4UY9vZ2PLzxqF12HQHt99l7ulknb986DIeXz9T/LmiPSCRy+rqEDw+koqaxBUPCfLFgcDii/T0Q6u2GFrUGJ7LL7b08PdumGzFIoN7HrCBh9+7dmDFjRqevmT59Onbv3m3Woqjns2e6kalFy4LO6hL+fSgNV4oVCPKU429zBhs8J1wwd6d4WQgwpiYEG737YU/jYgPRr48XRkb6I9oCqQD9deksbXdxhILl+Ylh+tkbPZkwgGzn5UKHyNkXipb7BnpatLNMoKcrPrhjFC7/bQHuGhEFjQbYcDwTCa9txabT2RZ7H2M99+NptKg1WDg4HCtnJhq1Y+XMdQm1TUp8oJvN8cKsRIjFIohEIkyI1+4mOErxskajQbYN0o30OwllNQ6V6kdkC2ZVFjc2NqKwsPN80YKCAjQ0mJ5Kkp6ejuTkZOTl5aGsrAzu7u4ICgrCkCFDMH78eLi59ZzuJb1ZT9pJqKhrwsvbzwMAVi0YCh83F4PnJ8X3gUQsQmZZLXIq6sz6gyYECdMsNB/B2txcpEj56wKIRSKLBDUDdF97Id2oSanC579rBxwt64EFy+0ZFxsIT7kUpbVNOJtfiRGR/nZdT2v7U8vubAnig7zw3dJJWD69DM/9eBqHM0rx0rbzuGtEtFXerz07Lxfil4v5kIpFWKNLITSGMGFcmBPiTP5zOB0V9c3oG+SFu0a0ZgqMjw3E92dyHKZ4ubpBidombepTpBVvEsQGeEIkAmqbWlBa22Ty3w0iZ2ZWkDBy5Eh8++23WLZsGcaNG3fD80eOHMF3332HsWPHGnW83NxcrFu3Dhs2bEBenjbf9vqIXSQSQSqVYtasWXj88ccxf/58p7ijSu2za5BQq5u2bHaQYLiT8OqvF1BZ34whYb54pJ0+/V6uMoyK8sfxrHLsSy3Gg2PjTHrfRqUKx7K0f5gtNR/BFtrL1zaXUBhbWtuEirom7LpSiLLaJoT7umHuoDCLvY8jc5FKMC0hGD9fzMfOy4V2DxLSdPUICWYWLRvrpphA/PDoZPR5YTOuFiugaFB2e5aHMVpUaiz/4RQA4H+n9Ee/YOODISHFztlmJTQqVVizRzs07c8zBxnsmghNDY5klkKt1th9aKGQahTgIYe7i/U6Kcp16XS5lfXIKK1hkEC9ilk/WW+88QZmzJiBSZMmYeHChZg4cSL69OmDkpISHDp0CL/88gukUilef/31To9TWlqKl19+GevWrUNLSwv69u2L++67D6NGjUJwcDD8/f3R0NCAiooKXL16FceOHcPevXuxfft2JCQk4J133sEtt9xi1idO9hWuCxKKFI1oUaktekHZlWLd9OM+Jk7oFNKNMspq0aRUQS6T4EpRNf51UFvgt3bxyA4/j2kJwdogIc30IOHotVI0tagR5uNm9QsyR+UplyHC1x15VfW4WqzAJ0cyAABLx8bb9HvH3mYPDMXPF/Ox43Ih/jwr0a5r6c6MBFMFebkiys8dOZX1OJNXoR/YZ02fHElHSmE1AjzkeHHu4K4/oI3BuiAhr6oeFXVN8PeQW2GFlrfhWAaKFI2I9HPH/WNiDZ4bHuEHV5kEFfXNSC1R6Hf37EXf2cgGqYbxgV7aIKGsFuMs0AGOyFmYFSRMnDgR27dvx2OPPYaffvoJP/30E0Qikf7uf2xsLP7v//4PEyZM6PQ48fHxEIvF+NOf/oT77rsPw4YN6/K96+rq8P3332PdunW47bbbsGbNGjzzzDPmfBpkR3285JCKRWhRa1CkaLRpTrmwkxDsbVqQEOrjBm9XGRSNSqSV1mBwmK8+X/mWIRGdzhiY1i8Eb++6ZFaHI2erR7CW/sHeyKuqx/ZLBdhztQgiEfDI+J5dsHy92bpdk+TMUtQ2KeEpt/4d9Y50Z9qyOUZFBSCnsh4nc6wfJFTVN+PFX7QphK/NT4Kfu2kX+T5uLogJ8EBWeR0uFFTZJKjpLqVKjdW7tbsIz88YpJ/7IHCRSjA6yh+HMkpx5FqZ3YOEXN0utDXrEQRxgZ7Yn1bMNqjULU1KFU7lViAu0NNpBn+afQtuxowZSE9Px4EDB/CPf/wDr732Gv7xj3/gwIEDSE9P77KwGQCeeuopZGVlYc2aNUYFCADg4eGBBx98EIcOHcKuXbsQExNj7qdAdiQRixHmo/0hsXXKkb4mwcSdBJFIpO8sdLmoGjsuFWBbSgFkEjHW3Da804+dEBcEqViE7Io6XDOxS4ZQ8Ows9QjWIrRBfX+vttPM7IGhiPb3tOeSbK5vkBfiAj2hVKktMqDPXI1KlT7dw5xBauYYFaVNrzqZY/3uOqt+u4DyuiYMCvExe0hfa4ejKguuzHq+OZmFrPI69PFyxaMdBN9CK1RHmJfQupNg/SChtQ0qB6qR+a6WKDDh/Z1IfP0Xey/FaGbtJCxduhRDhgzBM888g0mTJpk9NG3VqlVmfZxg+vTp3fp4sq8IXfqA3YIEM3JLBwT74HhWOS4UVGHz2VwAwB8n90NCFykXHnIpxsQE4EhmGfalFSM20LiL2/rmFhzP0l4UOVM9gjUIQUJ9swoA8NiEBHsux25mDQjFvw+nYeflQiwcEmGXNWg7vQDerjKb5WiPigoAAJzMqbDq+6QWK/CP/VcBdJ5C2JUhYT7YeiEP5wscvw2qWq3BWztTAADLpw+AWwc5/m3rEuzNFjMSBK0djriTQObL1H3/xNvoxoolmPXb7+uvv0ZJSYml10K9jD2Kl1VqNUrNLFwGoN9J+Pv+q7hUZFq+snCRvy+1yOj3S84ohVKlRqSfu37yZ2/Vv00gFuzlqp8U3dvMHhgKANhhx4nEbVONbJUCN1K3k5BeWoPK+iarvc/zW86gRa3BvMQwzNJ9rc0hTGJ2huLlH87l4kqxAr5uLnhyYr8OXzcuVjtJ/UqxAuW11jsHxtDPSLBBulHrrATuJJD5hCAzLsB5/pabFSTEx8d32QLVHDU1NcjMzIRSqTR4/LvvvsMf/vAHPPLIIzh9+rTF35fsI9xHGyTk2zBIqKhrhlpXOxPkaXox4UDdUC9Fo/Z71JR8ZWFS8r60YqP7bQs1DNN6eT0CAAxoM0Ru6bh4yHpRwXJb0/uFQCoWIa20xuTUNUtJ0xUt2yrVCAD8PeT6QPl0rnXuzu++UoitF/IgEYvwngktT9uT1KbDkSPMteiIRqPBmzsuAgD+d0q/TjtHBXq6or9uR+/oNfvuJtg03ShI+31XpGhEXZPjTJwm55KpCzKd6YafWX9lly5dim3btiE/P9+ii1mxYgWGDh1qECR8/PHHuPfee/HNN99g/fr1mDRpEq5cuWLR9yX7sMdOQkmtNtUowENuVhpB22nHpuYrj48NhItUjPyqBqQbmdvKeoRW4T7uCPd1g4tU3GHOdG/g7SbT39HdcbnALmuw9oyEjlizLkGlbp2a/j+T+3W7MLdvkBdcZRLUN6scuuD110sFOJNXCQ8XKZ6aOqDL14/Xfe/Zc15Ci0qN/GptlzpbpBv5ucvh566df5PJ3QQyk/B7IL6nBwmLFy/GTTfdhPHjx+Ojjz7C77//juzsbOTk5NzwnykOHDiAm2++Ge7urT/0b7/9NsLDw3Hw4EFs2rQJGo0G7777rjnLJgcjdDSyaZCgr0cwryVhbIAnvFy1+bqm5iu7uUgxNkb7B9aYotPaJiVOZGsvhqb28noEABCLRdj/1Ez8/twcxAU6T06nNQhpMDuvGJ+6Zkmt7U9tex6sWZfw6ZEMXCiogp+7C16eO6Tbx5NKxEjUBRrn8x2zLkGj0eAN3S7CExMTEGDE7qoj1CUUKhqgUmsgk4ht1iVGSBFx5ICPHFtGqS7dyImCBLMKl+Pi4vQtT//0pz91+DqRSISWFuO35goLCzFnzhz9vy9fvozc3Fy88847mDhxIgDg+++/x8GDB81ZNjkYe+wkFCvM62wkkErE+OmxKSirbTIrX3laQjAOppdgf1oxHpvYeeHt4YxStKg1iAnwQIwT5TBaU18nKviyptkDw/DiL+ex52oRlCq1zVOvWgep2XYnYWSkdXYSqhua8bdfzgEAXp2XZLG5BkPCfHEqtwIXCqpw+7Corj/Axg6ml+BIZhnkUjGenTHQqI8ROhz9nl1ul+89oDXVKMLX3WZD3eKDPHEqt0J/oUdkCpVajSzd922PDxIeeOABq+RHNzU1wcXFRf/vAwcOQCQSYdasWfrH4uLisHXrVou/N9meECTkVzXYbIKnkG5k6oyEtoTaAnNMTQjGq79e0NcldPZzJOw29PauRnSjEZF+CPCQo7yuCcezyjAxvo/N3lvRoESRLti2ZU0CAP2U6azyOpTXNhl159sYb+y4iNLaJgwI9sYTkyzXNSsp3BeA47ZBFXYRlo6LR6iPcXfk+/fxhp+7Cyrrm3E2rxKjowOsucR22bKzkYDFy9Qd+VUN+qBauPZxBmYFCRs2bLDwMrQiIiJw/vx5/b9/+eUX+Pv7IykpSf9YeXk5PD2dJwqjjoX6uEEk0g7xKatrskkrRXNnJFjK2NhAyKViFCkacbW486mlrEegjkjEYtzcPwTfnc7GjsuFNg0S0kq1qUZ9vFzh6+7Sxasty9fdBQlBXkgrrcGp3IpudR8SpJfW4IN92pan798+wqJ3xoUOR+cdsMPRiexy7LpSBIlYhBU3DzL648RiEcbHBmJbSgGSM0vtFCTYrrORgG1QqTuE4DLG3wMSsfM03XColc6dOxc7d+7Ec889h7/97W/47bffsHDhQoPXpKamIirK8bZtyXQyiRjBusDAVilHJTXmtz+1BFeZRJ/Tu6+T6cuKBiVO6fKupyWYv3NBPZfQCnWnjVuhpumKlm29iyCwdPHyii1noFSpMWdQKOYmWratrtDhKKOsxuG64ggdjf4wKsbkdEZ71yUI6UaRNrwj27qTwCCBTCfMSHCmVCPAwYKEF154AVFRUXj//ffx5ptvIjg4GK+99pr++ZKSEiQnJ2Py5Ml2XCVZkr4uodJGQUKt+YPULKV1XkLHQcKhjBKoNRr0DfLSF3gTtSXcRT+RU27TnvWppa0zEuzBksXL+1KL8OO5XF3L05HdPt71grxcEezlCo0GSCmssvjxzZVfVY+tF7TdCVfOTDT548fHtk5eNradsyXp041suZOga4OaVV6LFpXaZu9LPYN+kJqTBQlmpRsB2pkGH374IXbv3o2CggI0Nd34R0okEiEjI8PoY4aEhCAlJQV79uwBAEyePBne3q2FcWVlZXj33Xcxe/Zsc5dNDibC1x0ncypstpOgL1y2Z5DQLxjYpp2B0FFdAusRqCvhvu4YHOqDi4XV2H21EHePjLHJ+6bpOxvZtmhZMCraMjsJKrUaz2zWtjx9cmICBoV2r+VpR5LCfbHrShHOF1RhjK67mb1t/P0a1BoNJsYHmfV5j4kJgEQsQkF1A3Iq6xDtb9sLH326kQ1voIT7uMNFKkZzixq5lfWIdbKLPbIv/SA1J+vMZ1aQUFpaivHjxyMjIwPe3t5QKBTw8fFBc3MzGhq0vYvDwsIgk3U8lKUjbm5uWLBgQbvPDRo0CIMGGZ87SY7P1h2O9IXLdgwSxkQHwE0mQWltE1IKqzFYl5LQlpCKNDXBdrnm5HxmDQzFxcJq7LxSZLMgIdXO6UbDI/whEgG5lfUoVjQg2MwWmPvTSnAuvxI+bjK8Mi+p6w8wU1KYnzZIcJDiZY1Gg/XHtDfvHh5r3rwRdxcphkf44WROBY5kltk+SKiw/U6CWCxCbIAnrhYrkFleyyCBTCK0zu0V6UavvPIKMjIy8MUXX6CyUtv/+ZlnnkFdXR2OHz+OMWPGICYmBikpKWYtqrm5Gdu3b8f777+PVatW6R9vbGxESUkJ1GrTt/oaGxuxfPlyTJ48GWFhYXB1dUVISAgmTJiA9evX3zDlGQAUCgWWL1+O6OhoyOVyxMTE4Pnnn0dtLXMSLSXc17ZTl1vnJNgvSHCRSvRtBNtLOaqsb8KZPF09Qjc6KVHPN3tgGADtUDVbpH1oNJo2g9TsEyR4ucowQDf191Su+SlHv1zUptssHhZlsS5J7WntcOQYsxKOZJYitaQG7i4S3Dnc/Po+e9Ul1DQqUdXQDACI9LVdkAC0KV42chgmkcBZ043MChK2b9+OGTNm4L777rshVWL06NH49ddfkZWVhVdffdXkY2/duhVRUVFYuHAhnnvuObzyyiv6586fP4/Q0FB8++23Jh+3trYWH3/8MUQiEebPn4/ly5fjtttuQ35+PpYuXYoFCxYYBB91dXWYMmUK1q5diwEDBuCZZ55B//79sWbNGkyfPh2NjY0mr4FuZMudhPrmFtTqigft1d1IIHQs2t9O8fLB9BJoNED/YG+j2xJS7zQpPgiuMgnyqxpwqaja6u9XXtekv0Cz58wKS9QlCEHC/MQwi6ypI/ogoaDKLvn711t/LBMAcOfwaHi5mr7bLxBudCTbOEjI1dUj+Lq5wNvN/PWbg8XLZI7qhmaU12lT8mOdbOaRWUFCYWEhhg8frv+3RCLRpxkBgJ+fH+bOnYtNmzaZdNzk5GTccccdkMvl+Pvf/457773X4PkxY8agb9++2Lx5s8lr9vf3R3V1NQ4cOIBPPvkEb775Jj7++GOkp6dj6tSp2LlzJ3799Vf969955x2cPXsWK1euxI4dO/D2229jx44dWLlyJU6cOIG1a9eavAa6kS2DBGEXQS4V66cm24tQa7A/rRhqteGFw/60EoPXEHXEzUWKKX21KWk7Llm/y5GwixDp5w43F/v9DOmHqmWbV5eQWqxAemkNZBIxZg7ofhvVzgwM9oFELEJlfTMKqhu6/gArqmtqwXenswEAS8fFdetYQvHyufwq1DbduBNvLa3tT23f0KG1DSp3Esh4wi5CkKe8W4G5PZgVJPj4+Bik5/j5+SEvL8/gNd7e3igu7rh7S3tWrVoFX19fnDp1Cn/84x+RkHDjUJtRo0bh3LlzJq9ZLBYbDGoTSKVS3HbbbQCA9PR0ANot9XXr1sHT0xMvvviiwetffPFFeHp6Yt26dSavgW4kdO7Jq6q3+l02IUgI9nK1yjBAU4yKDoCHixQV9c24cF0PdX3RMucjkBFm6S5yd16xRZBg36JlQWsbVPN2EralaHcRpvTtY/U/2nKZBP11Xy97pxxtPpuD2qYWxAd6YlI3Z2tE+Lkjys8dao0Gv2dZdgJ2Z4T2p1F+tk01AjgrgczTmmrkXEXLgJlBQlxcHLKysvT/Hj58OHbt2oXycu0vioaGBvz8888mzzM4fvw4br31VgQGdtwBIjIyEkVFReYsu11qtRq//fYbAGDw4MEAgLS0NBQUFGDChAnw8DD8ReTh4YEJEyYgMzMTubm5FltHbxWuS6epb1bp0xisxRHqEQQyiRiT+t44L6G8rgnndBcSU7mTQEaYPUgbJBxIL0FDs3V78dt7RoJgWIQ/xCIRChUNKDBjF1KfajTYsnMROtI25ciePjuqLVh+aGycRW6UjDcz5UjRoMT9G4/hlxzTf+fbo7ORID6odeqyI6SOkXMQdp6crWgZMLO70axZs7B27VrU19fD3d0djz/+OO644w4MHToU48aNw+nTp5GVlYU33njDpOM2NTUZtDxtT1VVFcTdmFbX3NyMN998ExqNBuXl5dizZw+uXLmChx9+GDNmzACgDRIAtLuTITy+Y8cOpKWlITIyssPPpW1bWIVCewdOqVS2WyRtTcL72fp9jSEVAf7uLqiob0ZWWQ08Q603uqOwWvvHJchD7hBfi8lxQfjtUiH2XC3EYzdpA+p9V7V3gweFeMPPVeIQ66RWjvizlBDgjnAfN+RXN2Df1ULMHGC9Yvcrxdq6h/gAd7t+DVzEwMAQb6QUVuP4tVIsGNxaV9DVOVI0KnEwXZvSN6t/H5t8Hokh2r9rZ3Mr7PZ1yyyrxYH0EohEwL0joiyyjpui/PHtqWwkZ5SYdLwVW07hu9M5cBEDb9Y1wJT64yxdl5hwH1ebfy3DvbTZCDWNLSisqkWQnWvbrM0Rf985o3TdDmy0n5tVvpbGnCdz39esIOGJJ57AoEGD9EHC7bffjnfffRevv/46Nm/eDDc3NyxfvhzPP/+8SceNi4vDiRMnOn3N0aNHMWDAAHOWDUAbJLQtqBaJRHjuuefw1ltv6R+rrtb+IfTxab9/tBDICK9rz1tvvdVu4fbOnTvh7m6f4Vi7du2yy/t2xVvcggoAW3YdQE6g9fKcD17T3rVqqi7F9u3brfY+xpJWqwAAe68U4LedOyERifDl/jMAgBhZg0OskdrnaD9LAzyUyK8G/vPrUSgzrdep53SG9q59dfZVbN9u/AwcawgWNSIFwHf7jkOcc+Pn3NE5OlLcgha1BmHuIqSdOIQ0K68TAJpKtTs8R1Pz7PZz/XW69qbVUH8JLhzdjwsWOKZaof0ddii9CL9s2waxEbsTFytU+M8pbW1GsxpY+99duKmP8b/3z+q+B8uzUrF9+zUzVt09AXIRyps0+OrnXejnI7H5+9uDo/2+cza/X9V+v9cWZGL7dutloHR2nurrzav7NOuKLDQ0FHfffbfBY88++yyefvpplJWVoU+fPmZtZS5evBivv/461q9fj4cffviG59esWYOLFy/inXfeMWfZAABPT09oNBqo1WoUFBTg559/xl/+8hccPXoU27dv73Inw1gvvPACli9frv+3QqFAZGQkZs2aZbH3MJZSqcSuXbswc+ZMs2ZXWNt/8g4h63IhwvoNxryx3Sum68zeH88A6WkYMaAv5lmxL7qxZqnUWHVuC2qaWhA8aDTKLp/EtWZXAEo8MGM05g2NsPcS6TqO+rNUG5qDPV8cQ4bSHfPmWWfYpFqtQcmBHwAA98ydZrcWqIJs7zTs3XwGCnkg5s2bpH+8q3P04ze/A8jCnaMTMG/eMJusdXBlHV4/uw359RrcPGs2XKS2vbhUqdX431XbASjx7LzRmDfC/NanbbWo1HjpzBbUNbcgZsREDO5iMFt9cwuee3cnAMDTRYra5hbkuwRj3rybjH7PZ05tA1CHW6ZNwPg42w+nG5S5D4cyShHcLwnzRkbb/P1tyVF/3zmb5ae137OLpo3HpPggix/fmPMkZLOYyqK3bSUSCYKDzc+jfv7557F582Y8+uij+Prrr/XpOitWrMDRo0dx5MgRDBs2DH/84x+7vVaxWIyIiAg8+eSTCAwMxF133YU33ngDq1ev1u8gdLRTIHyxO9ppAAC5XA65/Ma7WzKZzG4/bPZ8785E6gbiFCqarLq+snrtdluoj7tDfB1kMmBy3z7YllKA5KxKBDercblYG+3PGBDqEGuk9jnaz9KcQREQiYCUwmqU1Cn180csKa+yHvXNKkjEIvQL8YVMYr3UQGPcFKstvD2VVwmpVHrDjan2zpFarcGvl7U1bQuTIm12DuOCfODjJkN1gxIZFQ1ICvezyfsK9l8uRG5VPXzdXHDHiBjIZJYJUmQy4KaYAOxNLcaJnEoMj+r8ov31Xy4gvawW4b5ueP+24bh7/RH8drkIEokUYnHXNxZVajXydB2i4vp42+VnsG+QFw5llCK7ssGhfgdYkyV/32k0GlTUNcPfw8XuDURsoUWlRraubW+/YF+rfs90dp7MfV+zfsvPmDEDb7zxBpKTk9HSYrlCOU9PTxw6dAj33HMP9u/fj8OHD0Oj0WDNmjU4cuQI7rrrLuzevbvdi+/umDVrFgBg//79AFprEYTahOt1VbNAprFVG1RHKlwWCMPSDqSXIKVSO6cjKdwXgT0815UsK8BTjtG62QE7L1uny5HQ2SguwNPuAQIAJIX5QioWoaSm0ejfHadyK1BS0wgvV6lV7uh1RCQSIUk3Wd0ek5eFCcv3joqGq4UCBIGxxcsnssvx/t4rAIB/3z0GCxLD4C4FSmqb8LuRrWyLFY1QqtSQiEUINXPSdndxVkL3fH48E4F//h5fnciy91JsIreyHiq1BnKpGGFOOPfIrN/0ycnJePHFFzF58mT4+vpi1qxZeOutt3Ds2DGoVKpuLcjPzw9fffUVioqKsH37dnz55ZfYunUrCgoK8M0338DPz/J3YAoKCgC0RloJCQkICwtDcnIy6urqDF5bV1eH5ORkxMbGdli0TKbRT12utnKQUOuIQYJ25+1QRinOlWsDbs5HIHPMHqjtcrTDSkFCmm7KbIKd04wEbi5SDNZdeBvbCnWbrqvRrAGhNk/5EXYPrm95bG2V9U344Zw2D/rhsfEWP/74WO3uwZFrZR2+prlFhUe+Oga1RoN7R8VgwZAIyCRijAzQJjNsvZDX4ce2laO7Ixvm4wapnQJVzkronv+eyQHQ2mGspxO+T2IDPI3aLXM0Zv2UVVdXY+/evfjb3/6G4cOH4+DBg/jrX/+KCRMm6AepvfPOO/j999/NXlhAQADmzJmDe++9FwsWLOhWGhMAXLp0qd3Cjfr6en3twLx58wBo7/o8+uijqK2txapVqwxev2rVKtTW1mLZsmXdWg+10u8kVFo3SChWtM5JcBRDw33h6+aCmqYW7C/UBQmcj0BmmKULEnZdKYSqzfR4S3GUGQltmTpUrXXKsm1an7Y1RNhJKLDtrIRvT2WjqUWNwaE+GKmbL2FJ43RD1dJLa/S7tdd7a2cKLhRUIdBTjg8Wj9Q/PipIG6gZHyTYb0aCQN8GtZQ7CabSaDQ4oftZtcWEeEcgzEhwxvangJk1CXK5HFOnTsXUqVPx6quvoqGhAcnJydi/fz/27duHPXv2YOfOnRCJRBZNR+qOTZs24f3338fEiRMRExMDb29v5Ofn49dff0V5eTkmTZqEZ555Rv/6FStW4KeffsLq1atx5swZjBgxAqdPn8bOnTsxevRoPP300/b7ZHqY1nQj600jVas1KK3V1rg40k6CRCzG5L59sPVCHprVgEikrVMgMtVNMYHwdpWhor4Zp3MrMTo6wKLHF6Yt27tgua1RUf749GiGUTsJhdUNOJWrfd3cxLAuXm159ko3Wn8sE4B2F8EaOeC+7i5IDPVBSmE1jmSWYtFQwx32iwVVeGNHCgDgn3eMQlCb378jA6WQiJuRUliNzLIaxHUxbKp1kJp9OgQC2nQ7AChUNKC+uQXudpw87mxyKuv0f4evlijQolLbbUfIVjLLnXeQGmChwmU3NzdEREQgPDwcoaGh8PT0RFVVVZfDRqZPn27W+4lEIuzZs8ekj1mwYAEKCgpw5MgRHD16FLW1tfDx8UFSUhLuueceLF26FFJp65fDw8MDBw4cwCuvvILNmzdj3759CA0NxbPPPouXX34Zbm7Ol1vmqIQgoaqhGbVNSnjKLV/YU1HfBLXu+zHQ03otIs0xrV+w/k7a0DBf+Lk71vrIOcgkYszoH4Ifz+Vix+UCiwcJ+nQjOw9Sa2uUrg7jVG4FNBpNpxfBv17SppWOjg5AiB3y2YXUqILqBpTVNtqk7iilsAonssshFYtw35hYq73P+NggbZBwzTBIUKnVWPrVMShVatwyJAJ3X9cNyFMmwqS4IOxPL8HPF/Lx1LTO25sL6UZR/vbbSfD3cNEXoV8rr0ViqK/d1uJsTmS3BvPNLWpklNWif7Dj7ExaQ0Zv3EkAgIyMDOzbtw979+7F/v37UVxcDI1Gg/j4eNxxxx2YNm0apk2b1ukxhELh64lEonYDDOFxc+6GjBo1CqNGjTLpY3x8fLB27VqsXbvW5Pcj43m7yeDlKkVNYwvyqxrQP9jyQUJJjfbuhb+7i0MUXbbVtgZhagJ3Ech8sweG6oKEQvxtzhCLHTc5o0SfbjQwpPM2l7Y0JEzbZam8rgnZFXWICej4D/E2faqR7XcRAMDLVYa4QE9kltXiQkGVvmmBNa0/qt1FWDA43Ko7qBPigvDJkXQcyTSsS/hg31WcyC6Hj5sMH989ut2/3QsGh2F/egm2XsgzIkiwf7qRSCRCfKAXTudWIKOUQYIpTlyXFphSWNXjgwR9ulEnv5scmVlXS1FRUejXrx8ef/xxHDlyBLNnz8b69euRnZ2NtLQ0/N///R+WLFmCkJDOfwmq1WqD/xoaGrBgwQL069cPGzduRFZWFhoaGpCVlYUvvvgC/fr1w8KFC80eCkGOK9xHV7xspQ5HxTXaVKZgb8dJNRIMCfNFgId2kucUphpRN8waoK1LOHqtDIoGy0z2VDQocf8XR6HRAA+MibVKe1VzyWUSfRpPZylHTUoVdl7RFnTbox5BIKzVFsXLSpUaG09oh41Zo2C5LWFewcmccjQptc1L0ktr8LdfzgEA3rttBMI6+L4RgrYD6SWorG/q9H0cId0IYPGyuU7kaIMEN12Hrd5QlyAECfFBvShIyMvLg0ajwYwZM/DOO+/g3XffxQMPPNDtbj8vv/wyLly4gBMnTuAPf/gDoqKiIJfLERUVhfvuuw/Hjx/HuXPn8PLLL3frfcjxWLsNqr79qQO2FhWLRVi3ZAzuipVhrq74lMgcsYGeSAjygkqtwd7UIosc86nvT+JaeS1iAjzwzztHW+SYljRKV4x7Mqfj4uVDGSWobWpBiLcrRkRavnjXWENsWJewPSUfJTWN6OPlavUajL5BXgjylKOpRY3TuRVQqzV49KtjaFSqMKN/CJaO6zhIiQ/0xKAQH6jUGvx2qfPOXLlV9k83AtoGCSxeNpZardH/jN6mS0lLKezZQUJFXROqGpoBaLsbOSOzgoT3338ft9xyC06ePIklS5YgODgYSUlJeOqpp/DTTz91OISsK19//TUWL14MT8/2v5je3t5YvHgxvvnmG7OOT47L+kGC4xUttzU/MQz39pU7ZYs0ciyWbIW6+UwONhzPhEgEfHH/eHi7Od7wqFG62ovOdhK2pWjrEeYlhtv1Z0xog3reBjsJQsHyA2NirZ5iKRKJDOYlfHIkHQfSS+DuIsEnS27qMkX4liHa3Z3OuhzVN7egTFf0as90I6BNhyMH2kl47ofT8H1+E9JKzJusa21XSxSoaWyBm0yCO4drJ3739J0EYRchxNvVaQvczfrN8fTTT2PLli0oLy/H77//jtWrVyMyMhKff/45brvtNgQGBmLUqFFYuXKlScctLS2FUtn5FnlLSwtKSkrMWTY5sAg/KwcJDjgjgcgahCBhW0o+ahrNTzkqqKrHY98cBwD8eWYiJjloKlzbnYSOmmX8Yud6BIGQbnSxoMoqbWoFxYoGfQ2GtVONBMK8hB/O5eL5LacBAG8uHIZYIwo2bxkSAUBbXK5Utf91ydUVLXu5SuFj52BVv5PgIG1QG5pb8PHhVFQ3KPHd6Wx7L6ddQj3CiEh/JIX7AgCuFGs7HPVUzt7ZCDAzSBCIRCKMHDkSzz33HLZt24aioiKsWbMG/v7+OH36NNasWWPS8eLj4/Hf//4X5eXtbxuXlpZi06ZN6Nu3b3eWTQ7I2rMShBkJDBKop5vaLxiBnnLkVtZj9kd7zapNUKs1ePjLY6iob8aISH+8Ms9yRdCWlhjqC7lUjOoGZbvpH6nFCqSX1kAmEWPmAPum88UHecJNJkGDUqW/y2gNX57IQotagzHRARgUaptC8wm6nYSj18pQ09iCcbGB+OOUfkZ97JiYAPTxckV1gxKH0tu/Cdhaj+BhlVauphA61WRV1Fk12DPWritFqG/W1oLsSy2282raJwQJo6MDEOOv/TloblFb9efA3jJ0HeGctbMR0M0gQa1W63cS5syZg6CgIDz//PMoLS2FTCbDuHHjTDre008/jaKiIowYMQJ///vfcerUKeTm5uLUqVP44IMPMHLkSJSUlBjMM6CeIVw3rjy/2jqzEoSdBEcapEZkDZ5yGX59chp83Vxw9FoZZn20B9W6vFhjfXQwFTuvFMJVJsGXD463+XRiU8gkYgzVpfG0N1RtW4r2jvqUvn3g5WrfO9ASsRiJuot2a9UlaDQarD+WAQB4eGycVd6jPSOjAuAi1V5SuEjF+PQPYyERG3eJIRGLsWBw5ylHjtDZSBDh6w6ZRAylSm213W9TbDmfq///R66V6YvHHYlQtDw6yh9isUjfJa0npxwJOwnO2tkI6EZNwsKFC+Hv749x48bhhRdewJ49e5CYmIgVK1bgt99+Q2VlJQ4fPmzScR999FG8+uqrKCwsxPLlyzFmzBjExMRgzJgxWL58OQoLC/HKK69g6dKl5iybHFhfXY7n2bxK7ND1M7ckfeEygwTqBUZFB2DPn2bAz90Fx7PKMfPDvaiqNy5QuFRYjRU/nQEAvLtouEO1PO1Ia8rRjXUJ+lSjwfbratSWtesSTuZUIKWwGq4yCe4ZGWOV92iPq0yCcTHalKOX5gwx+fumtS4hv920Mf2MBDt3NgK0QU1sgDZYsXfKUYtKja0XtN/jYpEIjUoVjmWVdfFRttXcosLZPO2kcWF+y6AQbevTnly87OydjQAz5yQ899xzEIvFGDZsmH4ewqRJk+Dl1f28qxdffBH33nsvvvrqK5w/fx7V1dXw8fHB0KFDce+99yI+3jb5lWRbA0J88PDYOKw/lom71x/GsWdnY4AFL05auxtxUBn1DiMi/bH3TzNw8z/34kR2OW7+5x7s/ON0+Ht0/DPQ3KLCHz5PRqNShTmDQvE/k41LF7E37VC1NP1EZYGiQYmDuvQVe9cjCFonL1da5fjCLsLtQyPh6+5ilffo8L3vG4fjWWW4c0SUyR97c/9QyKViXCuvRUphtX74nECfbmTnzkaC+EAvpJbUIKOsFtP7228dyZmlKK9rgr+7C6b3D8H3Z3KwL7UYU9rM37G3i4XVaGpRw9fNRX9DUJgv0ZN3EjKcfEYCYGaQsGXLFkyePBm+vr4WXo5WfHw8XnrpJascmxzXx3ePQWpJDZIzS7HwPwdw/LnZnV7QmIKFy9QbDYvQBgoz/rkHp3IrMOOfe7D7jzMQ0EGw/PK28zibV4kADzk++8M4u+d+G0vYSTiVWw61uvUu9K4rhWhRa9CvjxcS+jjG0CahaNMasxIamlvw9cksALZNNRLEBnoaVajcHg+5FDf3D8G2lAJsvZB3Y5CgSzeKdICdBMBxZiX8eE6barRwSATGxwbi+zM52J/mWHUJQj3CqCh//e+UQbqbgD11J6G5RaUvto93oCn1pjIr3eiWW26xWoBAvZdcJsEPyyYj2t8D6aU1uOuzwx12ujBFQ3MLahpbAADBXm7dPh6RM0kK98O+P92MIE85zuZVYsY/96BMFzS3dTC9GKt3XwIAfHLvTQj1cZ6flYEhPnCTSVDT2IK0NhdtQj2CPQeoXU+YlZBRVovaJssMvBNsvZCH6gYlovzcMd0GE50tTehy9LMufaat1nQjB9lJ0LdBtV+6kUajwZbz2hqORUkRmNZPu3twNKsMDc0tdlvX9doWLQuE2pwrxdUOUfxtaTmV9VBrNHCTSZy6FrJbhctnzpzBihUrcMstt+Dmm2/WP56dnY1NmzahoqLjvtWd+eqrrzBz5kwEBQVBLpcjKCgIM2fOxNdff92d5ZIT6OPliq2PT4GHixR7rhbhmc2nun1MYRdBLhXDy9U5exUTdcfgMF/sf2omgr1ccS6/EtP/sUefggcA1Q3NuP/zI9BogKXj4vXDjpyFVCLG8Ehtrv+pXG0aj1qt0c9HcJR6BAAI9HRFqLc2ALtYYNm7qD+e014w3jsqxilnrgjFy8ezy1CkaG1ioVZrkOtAhctAawqJ0MHGHs7mVSK7og5uMglmDQxF3yAvhPm4oblFjaPXHKcuQV+03CZIiAnwgKtMgqYe2uFI+JziAj2dZke2PWYHCStWrMCoUaOwZs0a/PLLL9i3b5/+OY1Gg3vvvRcbN2406ZgqlQqLFy/GAw88gD179qCurg5hYWGoq6vDnj17cP/992Px4sVQ98Cok1olhfvhq4fGQyTSdln5+FBqt47XdpCaM/+wEnXHoFAf7H/qZoR6u+FCQRWm/X03inUXYv/735PIqaxHXKAnPlg80s4rNY+2LgE4rStePp1XiZKaRni5SjEpPsieS7uBkHJkybqE5hYVftU1fbg1ybmCPEGYrztGRwdAo4F+zgMAlNY2oqlFDZEICPd1jB0uoRg1o6y2w/kc1ibsIsweGAp3FylEIpF+N2GfBVKO9lwtxmunG/SpXuaob27RpxSNjmoNEiRiMQYGa1MAe2JdgpCG5swzEgAzg4T169djzZo1WLBgAc6fP48XXnjB4HmhI9HWrVtNOu4//vEP/Pjjj5gwYQKSk5NRX1+Pa9euob6+HkeOHMHEiROxZcsW/POf/zRn2eREbk2KxBsLhgLQXsDsvVpk9rFai5add8uPyBIGhGgDhTAfN1wqqsbUv+/GP/dfxcbfr0EsEmHjA+Pt3ibUXK11CdoL7+26C+ZZA0IdroWrPkiwYF3CwfQSKBqV6OPlijFt7tg6m7ZdjgRCqlGot5vDnEthJ0HRqERFnWkthi1FqEdou/M3TVewbIl5CS/8fA6ny1X4+37zb9Sdya2ASq1BiLfrDQGeMMOjJ9YltN1JcGZmBQn/+te/MHDgQGzevBmDBw+Gi8uNHRQGDBiAtLQ0k477+eefo1+/ftizZ88NMxbGjh2L3bt3o1+/fli/fr05yyYn8+dZifjD6Bio1Brc8ekhpJu5rVusCxKCvRkkEPUL9saBp2ciwtcdV4oV+NP3JwEAf5mdiPFxjnXH3RTCTsKZ/EqoNBpsTykE4Fj1CIKkMG1qlCWLl3/W3XlfODjcKVONBAsHa+sSdl0p1OfVtx2k5ijcXKQI09Xt2KN4OaO0BhcKqiARi/RpWgD0Owm/Z5ejrsn8uoTMshqc1c3y+KmDtrTGOKHb2RsdFXDDTv6gHjwrIbMHdDYCzAwSLl26hJkzZ0Iq7Ti/Ozg4GCUl7U9O7EhqaipuueUWyGTt38mSyWRYuHAhUlO7l35CzkEkEmHdvWMxJjoAlfXNWPjv/SYPhQK4k0B0vb5BXjjw9M36nvOjovzx0lzHnapsjH59vODhIkV9swoXK1Q4revLPs9BWp+2NUTfBrXKIqkqGo1GP4Rs4RDHC4pMkRTuiyg/dzQoVdij20HWD1Lzd4zORoLWDke2z6kXUo2m9O1j0AUwNsATUX7uUKrUOJJZavbxN59tHdCWU1mvn3NgqvaKlgVC8fKlHriTkNEDZiQAZgYJUqkUzc2dX6wVFBTA09O0L46Liwvq6jrPfaurq2t354J6JleZBFsem4JwXzdcKVbgnvWHTe6EwEFqRDeKC/TC4eWz8Patw/DzE1Mhk3Srj4XdScRijNAVL2/K1P59Gh0dgGBvx8hhb2tAsDekYhGqGpotMrH3YkEVssrr4CqT4Ob+oRZYof2IRCJ9lyMh5cjROhsJhHxze+wkCFOWF11Xf6KtS9B2tupOXcLmszkAABfdrwUhtclUnQUJwk7C5WJFj+pwpNFokFmu/Z7olTsJQ4YMwd69e6FStT/6u76+Hrt378bIkaYVwA0fPhybNm1CQUH7E3cLCwuxadMmjBgxwuQ1k/MK9XHD1senwk0mwW+XCrFiyxmTPp4zEojaF+nngZUzExHigBfS5hBSjlKqtBccjjJA7XpymQQDdEWb53UpHd0hXEzf3D8EHnLn7+Cmb4V6MQ9qtcYh042ANsXLNp66XKxoQLJul2DR0Igbnp+a0AeA+XUJuZV1OJ5VDpEIuCdee1NW2LkwRVV9M9J0acJCzVBbcYGekEvFaFSqkFVufnG0oymva0JNYwtEIiCmNwYJS5cuRWpqKp544gk0NTUZPKdQKPDQQw+hqKgIy5YtM+m4y5cvR3l5OUaNGoX33nsPJ0+eRG5uLk6ePIk1a9Zg5MiRqKiowPLly81ZNjmxEZH++Px+bZ3K+3uv4LOjGUZ/bLFCV5PAIIGoR7v+QsQR6xEESeHaXQ9z0zjaElKNhItrZzcloQ+8XKUoUjTiZE45cnW7LY6WbiTcJc4st22Q8PPFfGg0wMhIf0S2EzgJdQkncspR02j6LI4fdKlG42MCMTNcBolYhAsFVSa3ez2pa30aG+CJwHbSfSViMQYE97ziZSHVKNzHHa4yxyi0N5fZQcI999yDTz/9FEFBQfj0008BAGPGjEF4eDi+//57PPjgg7jjjjtMOu7ChQuxZs0alJWVYcWKFbjpppsQExODm266CStWrEBZWZm+qxL1PneOiMYr87R50098+zuSM4yreeFOAlHvMKpNi8UQL1eMiLzx7qWjmKArEv/udHa36hIKqxvwuy6lY4EDzYPoDhepBHMGaneBtl7Ic+CdBPukG7XX1aitaH9PxAZ4QqXW4LCRfyfbElKNbh8aAS+ZCJN1LYRN3U1oTTXq+OdQX5fQg4qXe0pnI6AbcxK+/vpr/Oc//0FsbCzy87WV7ydPnkRUVBQ+/vhjfPbZZ2Ydd/ny5bhy5QpeeeUVLFq0CNOnT8eiRYvw2muv4cqVK3jmmWfMXTL1AC/OGYI7h0dBqVLj7vWHUVpz4+TY67XWJMi7eCURObO+QV7w1rVwnTMo1KG7/CwZFQ1XmQQXCqr0F/nmEKZKj44OcKop2V0RdkW+P5ur71DncEGC7iIwv6rBZhOOaxqV2K0r6G4v1Uhg7ryEIkUDDgupTEna49+qK4Y3tS6hbWejjgh1CSmFVSYd25H1lM5GANCt5MVly5Zh2bJlaGhoQGVlJby9vU0uVm5PXFwcXnzxxW4fh3oesViEz+4biwsFVbhSrMB9nx/B9v83FRJx+/GuWq1Baa1umBq7GxH1aGKxCFMT+mDrhXzcMcyxB4r5uctx5/AobPz9Gv4vOR03xQSadZyt54VUo56xiyCYlxgGiViEq8UKAIC7iwT+Ho7VtCTAQw5vVxkUjUpcK6/T9/23pl8vFaC5RY2EIC/9BXZ7piUE47OjGdifZtpOwo/ncqHRAGOiAxDp544L0LbVffqHMzhyrRTFigajmwH8nq2d+jwmpuMgoXUnQWHSOh2ZfpCak3c2Arqxk9CWm5sbwsLCDAKEQ4cOYerUqZY4PDQaDdLS0pCba151PfUsnnIZvn90EtxkEuy8Uog3dqR0+NrK+mao1Nqt/CCmGxH1eP+5exTeHu2GWQNC7L2ULj02oS8A4NtTWVA0mJ47Xt/cgl26u8o9pR5B4O8hx8Q2czui/Dxu6LNvbyKRqE0bVNukHG3R3c1fNDSi06+HsJNwKqfCpNbhQuvTO4ZH6R+L9HPHqCh/aDSGQ+46U1jdgPyqBohFok7T/vQdjoqqoVbbZ3K1pfWknQSzgoSKigooFO1HfceOHcPMmTMxdepUHDp0yKTj/vDDD3jggQdQWdlayJWVlYWkpCQMGDAAMTExuOeeezrsqkS9R2KoL/59zxgAwCvbz+v7aV9P2Kb2d3dx+haPRNS1AA85Bvg6R7HghLggDAzxRn2zCl+fvGbyx+++UoRGpQrR/h762Qs9SdvAJ9LPsYqWBa11CdYvXm5uUWFbirb7421Jne+Uhfu6IyHIC2qNBofSjdtNKKttxH5detLi63bihFarQuvVrgj1CANDvOEp73iKu9DhqEGpQlaF7edNWINQyB6na5HrzEy6avruu+8QFxeHoKAg+Pn5YejQoThy5AgAbXvS22+/HRMmTMCePXswYsQIbN261aTFfPzxxzh79iz8/Pz0jz3zzDNISUnBtGnTkJSUhP/+979m1ztQz/LATXF4dHw8NBrg3g3JKGin3zhnJBCRoxKJRFg2Xrub8MkR4zu2Cdp2NXK0u+yW0HYwnKPVIwiEu8Wmdv4xx77UYigalQjxdjUqPW1qgml1CT+dz4NKrcGwCL8bLnCFIundV4uM2vU6oets1Fk9AgBIJWL017UD7glD1ZqUKv3sk/jeVLi8Z88eLFmyBFlZWXB3d4e7uzsuXLiAefPm4ffff8eoUaOwZcsWDB8+HD/99BNOnDiB+fPnm7SYS5cuYcyYMfp/19TUYNu2bbj77ruxe/du/P777xg4cCCDBNL7xx2jMDTcDyU1jbhn/WG0qAwHsrCzERE5svvHxMJFKsbp3AqcyjG+gFmt1uCXi9rUj4U9pKvR9RL6eOvnSUT5O2aQIFwI2qINqlA4fOuQCKOK8vXFy0bOS/he19XojmFRNzw3MMQbCUFeaG5R47fL7c+yaquzIWrXSxSKl3tAh6OsijpoNICnXIpAT+dvlmJ0kPDBBx9ALBbj22+/RU1NDWpqavDVV1+hpqYGM2fORGVlJdatW4eTJ09i4cKFZi2moqICISGteaSHDx9GS0sLlixZAgCQyWSYOXMmMjJMv+NCPZObixT/fWQivFylOJRRir/9cs7gee4kEJEjC/R0xWLdXdpPktON/rgTOeUormmEl6sUU3TDs3qiv8xORJSfO25LcsyaC1ulG6nVGvyk2zla1EHr0+sJOwln8ytRUdfU6Wur6pux52r7qUaAdtdL2E3Y0kWXI41GY1KQIBR894SdhExdbUpcoGeP2N0zOkgQdgbuuusu/WNLlizB/PnzUVtbi3//+99YunRptxbj7e2N8vLWOyn79u2DWCzGpEmT9I/JZDLU1fWcyXzUfQl9vPHZH7SD1lbvuoRfLrT2chZqEjhIjYgc1TJdAfNXJ7NQ22RcAbPQ1WjuoDC4SJ2jBsMc94+JQ/aq2zAk3K/rF9uBsJNwrbwWKrW6i1eb73hWGYoUjfB2lWG6boegK6E+bhgQ7A2NBjjUxbyEny/mQalSIzHUBwM66JokBAnbUgrQ3NJxbei18lpU1DdDJhEjyYhamdY2qM4fJAjBYnwPqEcATAgSysvLkZiYeMPjwmOLFi3q9mIGDBiAn3/+GeXl5aiqqsLXX3+NkSNHGtQoZGdnIzjYuB8Q6j3uGB6FP03tDwB4YONRZOm2frmTQESObmpCMPoGeaG2qQXfnco26mN62pRlZxXp5w6ZRIzmFjXyqxqs9j7CILN5iaYFhcamHH1/Rrs7sLidVCPBmOgAhHq7QdGo7PR4wi7C0HBfyI2YOJwY6gsAuFzs/B2OelJnI8CEIEGlUsHV9cYLLeExb2/vbi/mT3/6EwoKChAREYGoqCgUFhbiySefNHjNsWPHMHTo0G6/F/U87y4ajjHRAaisb8Zdnx1Gk1LVGiT0gNxAIuqZtAXM8QCA/zMi5SizrAYXC6shEYswd1CYtZdHnZCIxYjR1UtYqw2qRqPpcspyR6YZUbxc06jEDl2dQXupRgKxWIRbdWlfnQ1WO5GtG6JmRKoRoN2NcZGKUd+sQnaFc2eKtHY26mVBgi0sXrwYH330ERITE9GvXz+sXr0aDz30kP75AwcOQKFQYM6cOfZbJDksF6kEmx6ZCD93F5zILsdzP55m4TIROYWHxsZDJhHj9+xynMur7PS1P+t61U+K7wN/D94AsTdhaJa16hIuFymQVloDF6kYcwaaFhRO0QUJ5/OrUKb7e3i9bRfz0aQb0NZVK11hCvNPF/I6vOtvbGcjgVQiRv8+ug5HTl68LHS56gmdjQATJy5/+OGH+Pbbbw0eKyvTTtQbNGjQDa8XiURISel40FV7nnzyyRt2DwRTpkwxmKFAdL1of09sfGA8Fvx7Pz48mAoXqTYOZk0CETmyPl6uWJQUgf+eycEnR9Lx4V2jO3ztzz28q5GzsXYbVOGu/c39Q+Dt1vHMgfb08XJFYqgPUgqrcSCtBIuH35hOtPmckGoU2WWx7bR+wfB2laFI0YjjWWUY12bgHQCo1GqcyjFtJwHQ1iVcKKhCSmEV5jvp97VGo+ndOwllZWW4cuWKwX9lZWXQaDQ3PH7lyhVcvnzZWusm6tD8weH480xt0Nrcoi0k404CETk6YWbClyeuob65pd3XVNU344AudaTtHAGyH6FI1VptUIUBZou6GKDWkc5SjuqbW7A9RRt03tFOAHE9F6kE8xPDdOvKu+H5K8UK1DW3wMNFioEhxqehJwodjoraH9TrDEpqGlHfrIJYJEK0g7bsNZXRQYJarTbrv87k5OQgJydHP0FZ+Lcx/xF1ZtWCoZjct7UtIIMEInJ0M/qHIDbAE9UNSvz3dPt/5367VIAWtUbbt75P92sBqfv06Uallg8ScivrcDKnAiIRcIuZQaFQvLy/nSDht0sFqG9WISbAAyMi/Y06nlAX8eO5XGg0hilHQtHyiEg/SMTG34cWOhw5SrpRTkUd4l7+Ca9uP2/0xwjpZpF+7j2m45hJ6UaWFhMTA5FIhMuXL6Nfv376f3dFJBKhpaX9uyxEgDbH8ZuHJmDSB7sQ6CGHt6tpW7RERLYmFovw6Ph4/PXnc/jkSDoeHBt3w2vY1cjxCDsJ1qhJ+El3t358bBCCvd3MOsaUvsEQibQtRktqGg1umm0+q92luH1o16lGgjmDwiCXipFWWoPLRQr9nAPAtCFqbSW2mZWgVmuMGhZnTT+cy8W18lq89utFLEqKxNCIrlvw6jsb9ZBUI8DOQcIDDzwAkUgEHx8fg38TWUKYrzuuvLgQUrGI31dE5BQeHhuPl7adR3JmKVIKq/TtIQFAqVJj+yVtFxoGCY5DuCisamhGRV2TRYvJze1q1FaApxxJYX44l1+J/WnFuGtENACgSanCzxe1QUh7U5Y74uUqw839Q7AtpQA/nss1DBKEegQji5YF8UFekEnEqGtuQW5VHaL97XuhfSZX+3moNRo888Mp7PnfGV1eR+jrEXpI+1PAzkHChg0bOv03UXfJJA7VwIuIqFOhPm5YODgcW87nYd2RDKxdPFL/3OGMElQ3KBHoKcdNMaZdhJH1uLtIEerthkJFAzLKai0WJBQpGnAgXTsEbVE3J05PTeiDc/mV2JfaGiTsulKImsYWhPm44aaYQJOOt2hoJLalFGDL+Vz8dc5gAEBziwrn8rXNZUzdSZBJxOjfxwsXC6uRUlht9yDhtC5IALQzJrZeyMOtXdSE9LTORoCDtUAlIiLq7R7TTWD+4ngmGpWtk2236lqfLkgMNynfm6yvtS6h+x2OmpQqvL/nMga9/gtUag2Swn0RH9S9Cb76oWpt6hLaphqZmt5zy5AIiETAyZwK5FZqZxucz69Cc4sa/u4uZqXcDGqTcmRPDc0tuFysLaC+f0wsAOC5H890OmUa6HkzEgAHDhLOnDmDzz77DGvXrsVnn32GM2fO2HtJREREVjdrYCii/NxRUd+MH85qC5g1Go0+P/2Wbt5VJssTUky60+FIrdbgm5NZGPD6z3j2x9OorG/G4FAfbLhvXLfXN7lvH4hEwNViBQqq6qFUqfGTrr7FmK5G1+vj5YoJuvanwvelMB9hVFSAWSm+Qmpdip2DhAsFVVCpNQjylOOju0Yj2MsV6aU1+PBAaqcf11qT0L2AzpE4XJBw4sQJDBs2DKNGjcKyZcvw7LPPYtmyZRg1ahSGDRuGkydP2nuJREREViMRi/GIrh2qMIH5UlE1rpXXQi4VY+aAEHsuj9oh3OlfvesSHvjiCLaezzPYBerKvtQijHn3N9y7IRlZ5XUI83HDp38Yi7MvzMNwI7sOdcbPXY7hEdrjHEgvwb7UYlTWN6OPlysmxgd18dHtuy2ptcsRAPyeJRQtm7deR+lwJKQajYj0h5erDG/eMgwA8NpvF1Ba0/5AuobmFhRUNwDoWelGdq1JuN7p06cxbdo01NfXY8aMGZg0aRKCg4NRXFyMgwcPYu/evZg2bRoOHTqEYcOG2Xu5REREVrF0bDxe3X4BB9JLkFqswFbd3doZ/UPgKWe3NkezeFgk1h1JR25lPTb+fg0bf78GT7kUCweH484R0ZgzMBRuLjdecqUUVmHlljPYlqItSPdyleLPMxPx9LQBcG/n9d0xLSEYp3MrsC+1GMKN/tuSIsxOXVs0NALP/ngaB9JLUFHX1Dpp2cR6BEHrrIRqaDQauzUcOZ2rrasYruto9OBNsfjwwFWcyavEy9vP4193j7nhY66Va1OufNxk8HN3sd1ircyhgoQXXngBSqUSO3bswMyZM294fseOHVi4cCH+/Oc/47fffrPDComIiKwvws8d8xLD8MvFfHxyJB2HM0oBcMqyo0oM9UXWq4tw9Fopvj+bi+/P5CCvqh7fnMrGN6ey4eEixfzBYbhjWBTmJYajuqEZL207j/XHMqHWaCAVi/DExAS8NHcIgqw012dav2C8t/cy9qQWoaZRCQBYbEJXo+vFBXohKdwX5/Or8O2pbFzWDUIztbORoG+QF6RiEWqbWpBbWY8oOw0kO5PXupMAaHf21i4eial/343/HE7H/5vUD4PDfA0+JrNMW4sSF+DZo7opditIaGlpwdWrV1FVVaUfiHa9yZMnG328I0eO4M4772w3QACA2bNn44477sAvv/xi1nqJiIicxWMT+uKXi/n49GgGqhqaAQALBrMewVGJxSJMiO+DCfF98N5tI/B7djm+P5OD78/mILuiDptO52DT6Ry4ybSDthp06Uh3DI/CmwuHWn043sT4IIhFIn3uvJ+7C6bqCprNtSgpEufzq/DmzotQazQI83FDmK+7WceSScTo18cbl4qqcamo2i5BglKlxvmCKgAwGC43JSEYi4dFYvPZXDyz+RR2/nG6QTAgzMjoboG5ozErSNBoNHjppZfwz3/+EzU1nVfydxQ8tLsYqRTR0dGdviY2NhYSSc+YZEdERNSRuYPCEObjps91Hhnpjwg/8y7AyLbEYhHGxgZibGwg3r1tOE7mVOgDBuEifUJcEN5dNBzj4syrCTCVj5sLRkb56weeLUqK6Hab8NuGRuC1Xy8gv0r7PWpuqpEgMdQHl4q0bVDnDArr1rHMcamwGs0tani7yhB73byDdxYNx88X87H7ahG2XczHgjazSnrijATAzCBh1apVeOONN+Dr64sHHngAERERkEq7n7k0fvx4HD9+vNPXHDt2DBMnTuz2exERETkyqUSMR8bFY9VvFwGwq5GzEolEGB0dgNHRAXj71mE4l1+JRqUaN8WY1wWoO6YlBOuDhO6kGgmGhvsh2t8D2RXanHxzU40E9i5eFlKNhkf43dAWNi7QC09PHYB3dl/Csz+exqyBoXCRam9a98Rpy4CZQcJnn32G6OhonDx5EgEBlhvo8u6772LChAn429/+hr/85S9wd2+9Y1JfX4/XX38dZ86cQXJyssXek4iIyFE9Mj4er++4CI2G9Qg9gUgkwrCI7ncrMtf0fsF4Z/cleOumJneXSCTCbUMj8cG+KwAss5MAaAu67UEoWh7RQUepv84ejA3HM5FaUoN/HUrD09MGAAAyynreIDXAzCChqKgITz75pEUDBABYs2YNkpKS8NZbb+Hjjz/G8OHD9d2Nzpw5g6qqKkyaNAnvvvuuwceJRCJ8+umnFl0LERGRvUX7e2L9H8ahsqHZIq0wqXebOSAUr81PwvAIP8hllkndXpQUoQ8SRkV173t0kJ07HAntT4dH+rX7vLebDK8vGIrHvjmOV7dfwP2jY+Hn7qLvbsSdBGjrAhQKhaXXgg0bNuj/f2VlJfbu3XvDaw4ePIiDBw8aPMYggYiIeqoHx8bZewnUQ4jFIrw4d4hFjzkxPghPTkpAsJcr/D3k3TpWgq7DUU1jC/Kq6hHpZ7viZbVag7N5ne8kAMDScXH48OBVnM+vwivbz+OFWYloVKogEYtsul5bMCtIePLJJ/HGG2+gpKQEffr0sdhirl27ZrFjEREREZF1ScTidmcHmMNFKkFCHy9cLlLgUmG1TS+600prUNfcAjeZBP076TQlEYvxweKRmP6PPfj4cBqG6NqhRvt7dLsQ3NGYFSTceuutOHToEMaPH4+XXnoJI0aMgLd3+1/QqCjjC2O66mxERERERD1XYqgvLhcpkFJYjdk27HAkpBolhftC2sXF/rR+IViUFIEt5/Pw7I+nAfS8zkZAN9KNRCIRNBoNHn744Q5fJxKJ0NLSYvbiKioqUFdXh8jISLOPQURERETOwdQOR1nltfjXoVQ8eFMcEkN9zX7f64eodeXd20ZgW0oBapu017k9rR4BMDNIeOCBB6xWTFJdXY2XXnoJ3377LcrKygwCjePHj+PVV1/FqlWrMHLkSKu8PxERERHZR2Ko8UHCtov5uP+LI6isb0ZKYTW2PTnN7PftqrPR9foGeeGpqf2xZs9lAD2vsxFgZpDQtsDYkioqKjB+/HikpqZixIgRCAoKwuXLl/XPJyUlITk5GV999RWDBCIiIqIeRthJSCnsuMORSq3GS7+cx5s7U/SP7U0tRkNzC9xcTL+01Wg0+nQjY4MEAPjbHG1L1LLaJvQPtu7EbHtwqAqLV155Bampqfj2229x8uRJ3HnnnQbPu7m5YcqUKe12PSIiIiIi59avjxckYhEUjUr9tPG2ihUNmPXhXn2A8D+T+yHC1x2NShUOpJeY9Z45lXWorG+GVCxCoi5IMYaPmwu2PTEVr81PwvzEnjfHxKGChK1bt2LBggW46667OnxNTEwM8vLybLgqIiIiIrIFF6kECUFeALS7CW0dzijB8Ld/xd7UYni4SPH1QxPw4V2jMVdX4PzrpQKz3lNINRoc5mvy/IgxMYF4ce6QLoudnZHZn1FNTQ3eeustzJgxAwMHDkRcXNwN/8XHx5t0zMLCQgwaNKjT18jlctTV1Zm83vz8fHzwwQeYNWsWoqKi4OLigpCQECxevBjHjx9v92MUCgWWL1+O6OhoyOVyxMTE4Pnnn0dtba3J709EREREXbu+eFmj0eC9PZcx9e+7UahowMAQb5xYMQdLRsUAAOYm6oKEFHODBNNTjXoDs2oSSktLMX78eGRkZMDb2xsKhQI+Pj5obm5GQ4N2aygsLAwymcyk4wYEBCA3N7fT11y5cgWhoaEmr/mf//wnVq9ejfj4eMyaNQtBQUFIS0vDli1bsGXLFnz99de4++679a+vq6vDlClTcPbsWcyaNQtLlizBmTNnsGbNGhw4cAAHDx6Eq6uryesgIiIioo4lhvrgh3O5SCmsQnVDMx7+8hh+PKe9Prx3VAz+s2QMPOWt15gz+oVAKhYhrbQGGaU1iNftRBjrjDBpOaL9Scu9lVk7Ca+88goyMjLwxRdfoLJSu0XzzDPPoK6uDsePH8eYMWMQExODlJSULo5kaPLkyfjpp586TCe6dOkSfvvtN9x8880mr3nMmDHYv38/0tPTsW7dOrz11lv4/vvvsW/fPkgkEjz55JNoamrSv/6dd97B2bNnsXLlSuzYsQNvv/02duzYgZUrV+LEiRNYu3atyWsgIiIios4JOwn700ow6p3f8OO5XLhIxfjX3aPx5YPjDQIEAPB2k2FCXBAA81KOTO1s1FuYFSRs374dM2bMwH333XdD1fno0aPx66+/IisrC6+++qpJx/3rX/8KlUqFCRMm4KuvvkJZWRkA4PLly/j0008xffp0yOVyPP/88yav+fbbb8eUKVNueHzSpEmYNm0aKisrceHCBQDaba1169bB09MTL774osHrX3zxRXh6emLdunUmr4GIiIiIOie0QU0vrUF6aQ2i/Nxx+JlZeHJSvw5b8OtTjkwMEooUDShUNEAkAoaGcyehLbOChMLCQgwfPlz/b4lEok8zAgA/Pz/MnTsXmzZtMum4Q4YMwXfffYeqqio88MAD+Ne//gWNRoPBgwdj2bJlaGhowKZNm5CQkGDOsjskpEVJpdrsq7S0NBQUFGDChAnw8DAcCe7h4YEJEyYgMzOzy9QoIiIiIjJNvz7ecNUVEM8dFIbTK+dhdHRApx8jFC/vSy1Go1Jl9HsJqUb9+3jDQ25WFn6PZdZXw8fHB0qlUv9vPz+/G1KEvL29UVxcbPKxb7nlFly7dg2ff/45jh8/joqKCnh7e+Omm27Cww8/jMDAQHOW3KGcnBzs3r0boaGhGDJkCABtkACgw2AkISEBO3bsQFpaGqdBExEREVmQXCbBf5dOREltEx66KQ5icdcDfIeE+SLc1w35VQ04kFaM2bqgoStMNeqYWUFCXFwcsrKy9P8ePnw4du3ahfLycgQEBKChoQE///wzoqKizFqUv78/nnnmGbM+1hRKpRL3338/mpqasHr1akgk2qi1ulpbTe/j036vXG9vb4PXtaepqcmgxkGhUOjfs22AZQvC+9n6fcl4PEfOgefJ8fEcOQeeJ8dn73M0e0AwAEClaoHKyI2BWf1DsP74NWy7mIfpCUFGfcypnHIAwNAwH6f8fjTmPJn7eZkVJMyaNQtr165FfX093N3d8fjjj+OOO+7A0KFDMW7cOJw+fRpZWVl44403zFqULajVajz00EM4ePAgli1bhvvvv9+ix3/rrbfarcnYuXMn3N3dLfpextq1a5dd3peMx3PkHHieHB/PkXPgeXJ8znSOghpaAACbT6ZjhrzQqI85kqZtq9+UfxXbt2dYbW3W1tl5qq+vN+uYZgUJTzzxBAYNGqQPEm6//Xa8++67eP3117F582a4ublh+fLlZhUY24JarcbSpUvx9ddf47777sO///1vg+eFHYSOdgqEXYGOdhoA4IUXXsDy5csNPiYyMhKzZs3S70TYilKpxK5duzBz5kyT29KSbfAcOQeeJ8fHc+QceJ4cnzOeowkNzXj/4k8oqNdgwJjJiAv07PT1lfXNKN61BQDw+O1z4OfuYoNVWpYx50m4bjWVWUFCaGiowUwBAHj22Wfx9NNPo6ysDH369Omw+tze1Go1Hn74YXzxxRdYsmQJNmzYALHYsH5bqEUQahOu11XNAqAd+iaXy294XCaT2e2HzZ7vTcbhOXIOPE+Oj+fIOfA8OT5nOkeBMhnGxwXhYHoJdqeW4H9CO+9WlFKsTTWKDfBEHx+PTl/r6Do7T+aeP4vOkJZIJAgODnaKAOHuu+/Gxo0b9XUIbSUkJCAsLAzJyck3THeuq6tDcnIyYmNjWbRMRERE5ECELkfGtEIVipY5RK193QoSzpw5gxUrVuCWW24xGHCWnZ2NTZs2oaKiotsLtBQhxeiLL77AnXfeiS+//LLdAAEARCIRHn30UdTW1mLVqlUGz61atQq1tbVYtmyZLZZNREREREYSgoS9RrRCPa1rf8rORu0zuyHsihUr8N5770Gj0QCAwe6BRqPBvffei/feew9PPfVU91dpAa+99ho+//xzeHp6ol+/fnj99ddveM2iRYswbNgwANrP76effsLq1atx5swZjBgxAqdPn8bOnTsxevRoPP3007b9BIiIiIioU0nhvgjzcUNBdQMOpZdg5sDQDl97Jk8IEriT0B6zgoT169djzZo1WLhwId544w188803ePvtt/XPx8TEYMyYMdi6davDBAlCy9ba2toOuy7FxMTogwQPDw8cOHAAr7zyCjZv3ox9+/YhNDQUzz77LF5++WW4ubnZaOVEREREZAyRSIQ5g8Lw2dEM/HqpoMMgoa6pBVeKtQW9w7mT0C6z0o3+9a9/YeDAgdi8eTMGDx4MF5cbq8EHDBjQYeGvKcrLy7Fjxw7s3bvXYKqzqTZs2ACNRtPpfw899JDBx/j4+GDt2rXIyclBc3MzsrOzsWbNGnh5eXXzsyIiIiIiazCmLuFcfiU0GiDU2w0h3rzx2x6zgoRLly5h5syZkEo73ogIDg5GSUmJUcfbv38/brrpJgQEBGDq1Kk4c+YMAODbb79FbGws5s2bh5kzZyIyMhJbt241Z8lERERE1Avc3D8EErEIV4oVyCqvbfc1Z3KZatQVs4IEqVSK5ubmTl9TUFAAT8/O+9MC2naic+fOxYkTJwAABw8exNy5c3HmzBk8/PDDiI+Px1NPPYU77rgDNTU1uOuuu3D16lVzlk1EREREPZyvuwvGxwYC6Hg34XSerrMRU406ZFaQMGTIEOzduxeqDuZk19fXY/fu3Rg5cmSXx1q9ejVUKhX27NmD8vJyJCcno7q6Gvfccw9uvvlmnDx5Eu+//z6+++47bN26Fc3Nzfj73/9uzrKJiIiIqBfQpxyldBAksLNRl8wKEpYuXYrU1FQ88cQTaGpqMnhOoVDgoYceQlFRkVFtQpOTkzF37lxMmzYNADBu3DjMnz8f6enpePnllw3alM6ePRuTJk3Cvn37zFk2EREREfUCcxNbW6E2XdcKtUmpQkphNQCmG3XG7CDhnnvuwaeffoqgoCB8+umnAIAxY8YgPDwc33//PR588EHccccdXR4rJycHAwcONHisX79+AIBBgwbd8PqkpCTk5OSYs2wiIiIi6gWGhvshxNsVdc0tOJRhWCObUlQNpUoNP3cXRPk596RlazJ7mNrXX3+N//znP4iNjUV+fj40Gg1OnjyJqKgofPzxx/jss8+MOo6LiwuUSqXBY3K5HADg7u5+w+s9PDygVqvNXTYRERER9XAikQhzBrbf5ahtqlHbOV9kqFsTl5ctW4Zz586htrYWeXl5UCgUSElJweOPP270MUJCQlBQYHjyxo8fj+XLl7f7+ry8PAQFBXVn2URERETUwwkpR9cHCexsZByzJy635ebmZvZwsWHDhuH48eMGj82cORMzZ85s9/Xnz59vNw2JiIiIiEgwc0AIxCIRLhcpkF1Ri2h/bdfN07m6zkYRLFruTLd2EizhwQcfxIIFC7psqQoAp06dwsWLF/VFzkRERERE7fFzl2Oc0ApV1+VIpVbjXL42SGBno84ZvZMQFxdn8sFFIhEyMjI6fc2cOXMwZ84co443cuRI1iMQERERkVHmDgpDcmYpfr1UiCcm9cPV4ho0KFXwlEuREORl7+U5NKODhKysLEgkkk6nLBMREREROYq5iWH42y/nsOdqEZqUKn3R8tBwP4jFLFrujMnpRlOnTsUXX3wBhUKBhoaGLv8jIiIiIrKHYeF+CPbStkI9nFnKIWomMDpIuHTpEp566imcPXsW99xzD8LCwvDMM8/gwoULZr/5nDlzcOLECbM+tq6uDm+//TY++ugjs9+fiIiIiHousViEOW2mL5/JE+oR2NmoK0YHCQMGDMCaNWuQl5eHzZs3Y9y4cfjoo48wbNgwjBo1Ch9//DGqq6tNevPS0lKMHTsW06ZNw/r16436+GPHjuGPf/wjoqOjsWrVKgQHB5v0nkRERETUe8wdFAoA2H4pH2fytDsJ7GzUNZMLDCQSCRYtWoRFixahuLgYGzZswIYNG/A///M/eO6553DbbbfhzTffRFRUVJfHOnXqFD7//HO8+uqreOSRR7Bs2TL0798fI0eORHBwMHx9fdHY2IiKigpcvXoVJ0+eRE1NDSQSCe655x68/vrrRr0PEREREfVOMweE6luhAoCLVIxBoT52XpXj61YVcnBwMFauXImVK1diz549eOihh/DNN9/grrvuMvri/cEHH8QDDzyA7du3Y/369di/fz++/PLLG14nFouRlJSE2267DY8++ihCQ0O7s3QiIiIi6gX8PeQYGxuAI5llAICkMF/IJHafAuDwut2q6MSJE/jss8/w7bfforq6GuHh4YiIiDDpGCKRCPPnz8f8+fMBAJcvX0ZeXh7Ky8vh5uaGoKAgJCYmwseHUR8RERERmWbuoDB9kMBUI+OYFSSUlZVh48aNWL9+PVJSUiCVSrFw4UI88sgjmD17NsTi7kVnAwcOxMCBA7t1DCIiIiIiQBskvPjLeQAsWjaW0UGCWq3G9u3b8dlnn2Hbtm1QKpUYPHgw3nvvPdx3330IDAy05jqJiIiIiMwyPMIf4b5uyK9qwLjYIHsvxykYHSRERESguLgYPj4+eOSRR7B06VKMGjXKmmsjIiIiIuo2sViE7U9OQ1Z5HYZGcCfBGEYHCUVFRZDJZBg6dCiysrLw0ksvdfkxIpEI27Zt69YCiYiIiIi6KyncD0nhDBCMZVJNglKpxIEDB4x+vUjEcddERERERM7G6CDh2rVr1lwHERERERE5CKODhOjoaGuug4iIiIiIHAQnSRARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZ6FVBwpdffonHH38co0aNglwuh0gkwoYNGzp8vUKhwPLlyxEdHQ25XI6YmBg8//zzqK2ttd2iiYiIiIhsTGrvBdjS3/72N2RnZyMwMBChoaHIzs7u8LV1dXWYMmUKzp49i1mzZmHJkiU4c+YM1qxZgwMHDuDgwYNwdXW14eqJiIiIiGyjV+0krFu3DllZWSgtLcUTTzzR6WvfeecdnD17FitXrsSOHTvw9ttvY8eOHVi5ciVOnDiBtWvX2mjVRERERES21auChJtvvhnR0dFdvk6j0WDdunXw9PTEiy++aPDciy++CE9PT6xbt85ayyQiIiIisqteFSQYKy0tDQUFBZgwYQI8PDwMnvPw8MCECROQmZmJ3NxcO62QiIiIiMh6elVNgrHS0tIAAAkJCe0+n5CQgB07diAtLQ2RkZHtvqapqQlNTU36fysUCgCAUqmEUqm08Io7J7yfrd+XjMdz5Bx4nhwfz5Fz4HlyfDxHzsGY82TuOWSQ0I7q6moAgI+PT7vPe3t7G7yuPW+99RZeffXVGx7fuXMn3N3dLbBK0+3atcsu70vG4zlyDjxPjo/nyDnwPDk+niPn0Nl5qq+vN+uYDBKs5IUXXsDy5cv1/1YoFIiMjMSsWbP0QYatKJVK7Nq1CzNnzoRMJrPpe5NxeI6cA8+T4+M5cg48T46P58g5GHOehGwWUzFIaIewg9DRToHwxe5opwEA5HI55HL5DY/LZDK7/bDZ873JODxHzoHnyfHxHDkHnifHx3PkHDo7T+aePxYut0OoRRBqE67XVc0CEREREZEzY5DQjoSEBISFhSE5ORl1dXUGz9XV1SE5ORmxsbEdFi0TERERETkzBgntEIlEePTRR1FbW4tVq1YZPLdq1SrU1tZi2bJldlodEREREZF19aqahHXr1uHw4cMAgAsXLugf279/PwBg4sSJePTRRwEAK1aswE8//YTVq1fjzJkzGDFiBE6fPo2dO3di9OjRePrpp+3xKRARERERWV2vChIOHz6Mzz//3OCx5ORkJCcn6/8tBAkeHh44cOAAXnnlFWzevBn79u1DaGgonn32Wbz88stwc3Oz6dqJiIiIiGylVwUJGzZswIYNG4x+vY+PD9auXYu1a9dab1FERERERA6GNQlERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERERGWCQQEREREREBhgkEBERERGRAQYJRERERERkgEECEREREREZYJBAREREREQGGCQQEREREZEBBglERERERGSAQQIRERERERlgkEBERERERAYYJBARERERkQEGCUREREREZIBBAhERERH9//buPiiq6/4f+HsR2MVlF4ygK1PYZQkkaY0hKgQLCIoBQmaiUycBHIxQ0dQ+xUpSjY0BnBHrjDI2TSYGbfAhtXmoD80kNRAbHiJFNIJoHBNWKivVQVMEebAsC3u+f/hjf97sLmID7ALv18zOyDnn3vu5+5mrfrj3nEskwSLhHk6fPo2UlBT4+vpCqVQiKioKH3zwgbPDIiIiIiIaMe7ODsCVlZWVISkpCQqFAmlpaVCpVDh06BBSU1PR3NyMnJwcZ4dIRERERDTseCfBgb6+PqxatQpubm6orKxEUVERduzYgfr6eoSFhWHjxo0wGo3ODpOIiIiIaNixSHDg888/R2NjI5YtW4bw8HBru4+PDzZu3Ije3l7s27fPeQESEREREY0QFgkOlJeXAwASExNt+pKSkgAAFRUVoxkSEREREdGoYJHggMFgAACEhoba9Gk0Gnh7e1vHEBERERGNJ5y47MCtW7cA3Hm8yB61Wm0dY4/JZILJZLLZ382bN2E2m4cx0nszm824ffs2Wltb4eHhMarHpqFhjsYG5sn1MUdjA/Pk+pijsWEoeers7AQACCHua98sEkbI1q1bkZ+fb9MeHBzshGiIiIiIaCLr7Ox0+Mtve1gkODDwJTq6W9DR0YEpU6Y43P6VV17BunXrrD9bLBbcvHkTU6dOhUwmG95g76GjowOBgYFobm6GWq0e1WPT0DBHYwPz5PqYo7GBeXJ9zNHYMJQ8CSHQ2dmJgICA+9o3iwQHBuYiGAwGzJkzR9LX0tKCrq4uREZGOtxeLpdDLpdL2nx9fYc9zvuhVqt5obs45mhsYJ5cH3M0NjBPro85Ghvulaf7uYMwgBOXHYiLiwMAlJaW2vSVlJRIxhARERERjScsEhxISEiAXq/HwYMHcfbsWWv7rVu3UFBQAE9PTzz//PPOC5CIiIiIaITwcSMH3N3dsWfPHiQlJWH+/PlIS0uDSqXCoUOHYDQasX37duh0OmeHOSRyuRy5ubk2jz+R62COxgbmyfUxR2MD8+T6mKOxYSTzJBP3ux7SBHPq1Cnk5ubin//8J8xmMx599FGsW7cOqampzg6NiIiIiGhEsEggIiIiIiIJzkkgIiIiIiIJFglERERERCTBImEcO336NFJSUuDr6wulUomoqCh88MEHzg5rwnn33XfxwgsvYO7cuZDL5ZDJZNi7d6/D8R0dHVi3bh20Wi3kcjl0Oh1efvlldHV1jV7QE8zVq1exc+dOJCYmIigoCJ6entBoNFi6dClqamrsbsM8ja6enh6sW7cO8+fPR0BAABQKBTQaDaKjo1FcXAyz2WyzDXPkGrZt2waZTAaZTIaTJ0/a9DNPzqHT6ax5+e4nPj7eZrzJZMLmzZsRGhoKhUKBgIAArF69Gjdu3Bj94CeYI0eO4Mknn8TUqVOhUCgQHByM9PR0NDc3S8YN97XEOQnjVFlZGZKSkqBQKOyuzJSTk+PsECcMnU4Ho9EIPz8/KJVKGI1GFBcXIzMz02Zsd3c3YmJicPbsWSQmJuLxxx9HXV0dSktLERERgcrKSigUitE/iXFuw4YN2LZtG0JCQhAfHw9/f38YDAYcPXoUQggcPHhQslgB8zT6/vOf/yAwMBCRkZEICwuDv78/2tracOzYMRiNRiQmJuLYsWNwc7vzuy/myDV89dVXmDt3Ltzd3dHd3Y3q6mpERUVZ+5kn59HpdGhvb8fatWvt9t39b5TFYkFKSgpKSkoQFRWFuLg4GAwGHDlyBMHBwTh58iT8/f1HL/gJQgiBn/3sZygqKkJISAiSkpKgUqlw7do1VFRU4M9//jNiYmIAjNC1JGjcMZvNIiQkRMjlclFXV2dtb29vF2FhYcLT01M0NTU5L8AJ5rPPPrN+31u3bhUARHFxsd2xr732mgAg1q9fL2lfv369ACAKCgpGOtwJ6dChQ6K8vNymvbKyUnh4eIgpU6aInp4eazvzNPr6+/uFyWSyaTebzSI+Pl4AEB9//LG1nTlyvt7eXjF79mzxxBNPiIyMDAFAVFdXS8YwT86j1WqFVqsd0th33nlHABDp6enCYrFY29966y0BQKxevXqEopzYdu7cKQCIn//856Kvr8+m32w2W/88EtcSi4RxqKSkRAAQWVlZNn179+4VAER+fr4TIqPBigSLxSICAgKEt7e36OrqkvR1dXUJb29vodfrRylSGpCYmCgAiNOnTwshmCdX9Ic//EEAEDt37hRCMEeuIjc3V8jlcnHhwgWxYsUKmyKBeXKu+ykS5s2bJwDY/ILRYrEIvV4vlEqluH379ghEOXHdvn1bTJkyRej1ekkxYM9IXUuckzAOlZeXAwASExNt+pKSkgAAFRUVoxkSDYHBYMC1a9cQHR0NpVIp6VMqlYiOjsa//vUvm2cQaWR5eHgAuPOCRYB5cjUWiwWffvopAGDmzJkAmCNXUFtbiy1btiA3Nxc//OEP7Y5hnpzPZDJh7969KCgowBtvvGF3DlZPTw9qamrw0EMPQavVSvpkMhmefPJJdHd348svvxytsCeE0tJStLW1YcmSJejv78fhw4fx+9//Hrt27cKlS5ckY0fqWuIbl8chg8EAAAgNDbXp02g08Pb2to4h1zFY3gbaS0pKYDAYEBgYOJqhTVhXrlzB8ePHMWPGDDz66KMAmCdn6+3tRUFBAYQQaG1txT/+8Q98/fXXyMrKQkJCAgDmyNlMJhOef/55hIeH47e//a3DccyT87W0tCArK0vSFhERgb/85S8ICQkBADQ2NsJisQyaJ+BOPmNjY0c24AnkzJkzAIBJkyZh1qxZaGhosPa5ubnhN7/5DbZv3w5g5K4lFgnj0K1btwAAPj4+dvvVarV1DLmOoeTt7nE0ssxmM5YvXw6TyYRt27Zh0qRJAJgnZ+vt7UV+fr71Z5lMhpdeeglbt261tjFHzvXaa6/BYDDgzJkz1uvGHubJubKyshAbG4uZM2fC29sbDQ0NKCwsxIEDB5CQkIDz589DpVIxT04ysGpUYWEhZs+ejVOnTuGRRx5BXV0dVq9ejR07diAkJARr1qwZsRzxcSMiou+wWCzIzMxEZWUlVq1aheXLlzs7JPp/vL29IYRAf38/mpub8eabb2LPnj2Ij49HR0eHs8Ob8Kqrq7F9+3a8+uqr1se/yDXl5uZi4cKFmDZtGiZPnozw8HDs378fy5cvh9FoxO7du50d4oRmsVgAAJ6enjh69CgiIiLg7e2N2NhYfPjhh3Bzc8OOHTtGNAYWCePQQCXpqGLs6OhwWG2S8wwlb3ePo5FhsVjw05/+FAcPHkRGRgZ27dol6WeeXIObmxt+8IMfYM2aNSgqKkJVVRW2bNkCgDlylr6+PqxYsQKzZs3Chg0b7jmeeXJNL7zwAgCgqqoKAPPkLAPf59y5cxEQECDpmzlzJvR6PRobG9He3j5iOeLjRuPQ3c8HzpkzR9LX0tKCrq4uREZGOiM0GsTdebPnXs8c0vdnsViQlZWF/fv3Iz09HXv37rWuuz+AeXI9A4s0DCzawBw5R1dXl/W79fT0tDtm3rx5AO68HGpgQjPz5Fr8/PwA3Fl3HwD0ej3c3NyYp1H20EMPAQB8fX3t9g+0//e//x2xv/NYJIxDcXFx2Lp1K0pLS5GWlibpKykpsY4h1xIaGoqAgABUVVWhu7tbskJBd3c3qqqqEBwczAl8I+TuAiE1NRUHDhyw+zw18+R6rl27BuD/r0TFHDmHXC7HypUr7fZVVlbCYDDgmWeegb+/P3Q6HfPkogZWONLpdAAALy8vREZG4uTJkzAajZIVjoQQ+Oyzz6BUKjF37lxnhDtuLViwAABw8eJFmz6z2YxLly5BqVTC398fGo1mZK6l+140lVye2WwWer1+0JepXb582WnxTWR8mZpr6u/vt67j/uyzz95zTWrmafRduHBBdHd327R3d3eL5ORkAUBs2bLF2s4cuRZ770kQgnlylosXL9q9ni5evCg0Go0AICoqKqztfJmacwy8p2f37t2S9s2bNwsAIiMjw9o2EteSTAgh7q+soLGgrKwMSUlJUCgUSEtLg0qlwqFDh2A0GrF9+3bk5OQ4O8QJY8+ePThx4gQA4Pz586itrUV0dDQefPBBAEBMTAyys7MB3Kn4o6OjUV9fj8TERMyePRu1tbXW16pXVFTAy8vLaecyXuXl5SE/Px/e3t548cUXre9EuNuSJUsQHh4OgHlyhry8PBQWFiImJgY6nQ5qtRpXr17FsWPH0NraitjYWJSUlFi/d+bItWRmZmLfvn2orq5GVFSUtZ15co6B62n+/PnQarVQKpVoaGjA3//+d5jNZrzyyisoKCiwjrdYLEhJSUFJSQmioqIQFxeHS5cu4fDhw9DpdKipqYG/v78Tz2h8amxsxI9//GPcuHEDTz/9NB5++GHU1dXh888/h1arxcmTJ6HRaACM0LX0v1Q2NDbU1NSI5ORkoVarhZeXl4iMjBTvvfees8OacAZ+g+bos2LFCsn49vZ2sXbtWhEYGCg8PDxEUFCQyMnJER0dHc45gQngXjmCnbs/zNPoOn36tFi1apX40Y9+JHx9fYW7u7uYOnWqWLBggXj77bft3v1hjlyHozsJQjBPzlBeXi6ee+45ERoaKtRqtXB3dxcajUYsXrxYlJSU2N2mp6dH5OXliZCQEOHp6Sk0Go3Izs4WLS0toxz9xHLlyhWRmZkpNBqN8PDwEIGBgeIXv/iFuH79us3Y4b6WeCeBiIiIiIgkuAQqERERERFJsEggIiIiIiIJFglERERERCTBIoGIiIiIiCRYJBARERERkQSLBCIiIiIikmCRQEREREREEiwSiIiIiIhIgkUCERERERFJsEggIqJRl5eXB5lMhvLycmeHAp1OB51O5+wwiIhcCosEIiKy0dTUBJlMNuiH/7EmIhq/3J0dABERua6QkBBkZGTY7fP19f2f9/vLX/4SaWlpCAoK+p/3QUREI4dFAhEROfTggw8iLy9v2Pfr5+cHPz+/Yd8vERENDz5uRERE35tMJkN8fDz+/e9/Iz09HX5+fpg8eTKio6Nx/Phxm/GO5iSUlZXhqaeeQkBAAORyOaZPn47Y2FgUFRXZ7KOqqgpPP/00HnjgASgUCjz88MPIzc3F7du37cb4t7/9DREREfDy8sL06dOxatUqtLW1OTyn3t5eFBYWYvbs2VAqlVCpVIiNjcVHH310f18OEdEYxCKBiIiGRVtbG6Kjo2EwGJCdnY309HTU19cjOTkZR48evef2n3zyCRISElBTU4OkpCTk5OTgmWeegclkwoEDByRjP/zwQ8TFxaG8vBxLlizB2rVrMXnyZGzevBkLFy5ET0+PZPz+/fuxZMkSNDQ0YPny5VixYgWqqqqwaNEi9Pb22sRiMpmsMQghsHLlSmRkZMBoNGLx4sV44403vtd3RUTk6mRCCOHsIIiIyLU0NTUhODh40DkJUVFRSE5OBnDnTgIALFu2DO+++67153PnziEiIgI+Pj4wGo3w8vICcOdOQn5+PsrKyhAfHw8AWLp0KQ4fPoyzZ8/isccekxyrtbUVU6dOBQB0dHQgKCgIPT09OHXqFGbNmgUAsFgsWLZsGd5//31s3rwZmzZtso4PDAxEf38/amtrERYWBgAwm81YtGgRKisrodVq0dTUZD3e7373OxQUFGDTpk3Iz8+3nk9nZycWLlyIc+fO4fLlywgICPhe3zMRkcsSRERE33H58mUBYNDPiy++aB0PQEyaNEk0NTXZ7GvlypUCgPjrX/9qbcvNzRUARFlZmbXtJz/5iQAgvvnmm0Fj279/vwAg1qxZY9NnNBqFu7u70Ov11rZ9+/YJAOJXv/qVzfgvvvhCABBardba1t/fL6ZMmSJCQkKExWKx2eajjz4SAMQf//jHQeMkIhrLOHGZiIgcSkpKwqeffjqksUFBQdBqtTbtsbGx+NOf/oS6ujosXbrU4fZpaWk4fPgwoqKisGzZMiQkJCA2NtZmgnNdXR0AWO9AfDcGvV6PhoYGdHZ2QqVSob6+3hrHd82bNw/u7tJ/Cr/55hu0tbUhICAA+fn5Ntt8++23AICvv/7a4bkQEY11LBKIiGhYTJ8+fdD2W7duDbr9s88+i6NHj6KwsBC7du3Cm2++CZlMhgULFmDHjh0IDw8HcOfxocGON2PGDDQ0NKCjowMqlcp63GnTptmMnTRpkvUxpgE3b94EAFy4cAEXLlxwGG93d/eg50NENJZx4jIREQ2L69evD9ru4+Nzz30sXrwYFRUVaGtrw7Fjx5CdnY3y8nIkJyejvb0dAKBWqwc9XktLi2TcwHFv3LhhM7a/vx+tra2StoHtli5dCiGEw09xcfE9z4eIaKxikUBERMPiypUrMBqNNu1ffPEFAODxxx8f8r5UKhWSk5NRVFSEzMxMXL9+HTU1NZL9fHf5VABobm5GY2Mj9Ho9VCoVAFgnQQ/Ecbfq6mr09fVJ2h555BGo1Wp8+eWXMJvNQ46ZiGg8YZFARETDor+/Hxs3boS4a9G8c+fO4cCBA/D390dKSsqg21dWVqK/v9+mfeAOgEKhAHDnboOPjw+Ki4sljwMJIbB+/Xr09fUhMzPT2r548WKo1Wq88847aGhosLabzWa8+uqrNsdzd3fHmjVrYDQa8dJLL9ktFL766iu7dyaIiMYLzkkgIiKHLl26NOgblzds2GD9z/usWbNw4sQJREREYNGiRfj222/x/vvvo6+vD0VFRdblTx359a9/jWvXriEmJgY6nQ4ymQwnTpzAqVOnEBUVhZiYGAB3HgfavXs30tPT8cQTTyA1NRX+/v44fvw4zpw5g8jISLz88svW/fr4+OD1119HZmYmIiIikJaWBh8fH3z88cfw8vLCjBkzbGLJz89HbW0tXn/9dXzyySeYP38+pk2bhqtXr+L8+fOor69HdXW13XkORETjgtPWVSIiIpc1lCVQAYi2tjYhxJ0lUOPi4kRzc7NITU0VDzzwgFAoFGLevHmitLTUZv/2lkB97733xHPPPSdCQkLE5MmThY+Pj3jsscfEtm3bRGdnp80+KisrxVNPPSV8fX2Fp6enCAsLE5s2bRJdXV12z+nIkSNizpw5Qi6Xi2nTpons7Gxx8+ZNodVqJUugDujr6xNvv/22iI6OFmq1WsjlchEUFCSSk5PFW2+95fA4RETjAV+mRkRE35tMJrO+AZmIiMY+zkkgIiIiIiIJFglERERERCTBIoGIiIiIiCS4uhEREX1vnN5GRDS+8E4CERERERFJsEggIiIiIiIJFglERERERCTBIoGIiIiIiCRYJBARERERkQSLBCIiIiIikmCRQEREREREEiwSiIiIiIhIgkUCERERERFJ/B9leL6uGzFDcgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 800x450 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Make plots\n",
    "figs_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/\"\n",
    "model_name = 'result_advantage_critic_greedy'\n",
    "kernel_size = 3\n",
    "fig, ax = plot_episode_rewards(rewards, kernel_size, save_path=os.path.join(figs_path, f\"{model_name}_rewards.png\"))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device:  mps\n",
      "Starting training iteration 1/1\n",
      "Critic loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random2.pth\n",
      "BackBone loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random2.pth\n",
      "Actor loaded from:  /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random2.pth\n",
      "Number of trainable parameters in the Critic model: 15927297\n",
      "Number of trainable parameters in the BackBone model: 7474272\n",
      "Number of trainable parameters in the Actor model: 18055257\n",
      "Memory initialized with 12 episodes. Average steps per episode: 9.83, Average reward: 1.00\n",
      "Critic Loss: 0.9374122023582458\n",
      "Actor Loss: -2.980232238769531e-07\n",
      "Critic Loss: 0.4981852173805237\n",
      "Actor Loss: -1.3828277587890625e-05\n",
      "Critic Loss: 0.8570868968963623\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.8999881744384766\n",
      "Actor Loss: 0.6027507781982422\n",
      "Critic Loss: 0.9544755220413208\n",
      "Actor Loss: 4.681673999584746e-06\n",
      "Critic Loss: 0.8332691192626953\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8749406337738037\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.8570667505264282\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.8569973707199097\n",
      "Actor Loss: 1.498631149843277e-06\n",
      "Critic Loss: 0.9090728163719177\n",
      "Actor Loss: -6.935813416930614e-07\n",
      "Critic Loss: 0.9229977130889893\n",
      "Actor Loss: -1.4671912822450395e-06\n",
      "Critic Loss: 0.909020721912384\n",
      "Actor Loss: -6.935813416930614e-07\n",
      "Epoch 1: Average Steps per Episode: 7.00, Average Reward: 0.00, Average Loss: 0.9094\n",
      "Critic Loss: 0.9374120831489563\n",
      "Actor Loss: 3.5762786865234375e-07\n",
      "Critic Loss: 0.856997549533844\n",
      "Actor Loss: 6.811959565311554e-07\n",
      "Critic Loss: 0.9544757008552551\n",
      "Actor Loss: 1.1270697086729342e-06\n",
      "Critic Loss: 0.9090728163719177\n",
      "Actor Loss: -5.201860062697961e-07\n",
      "Critic Loss: 0.856997549533844\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.9229977130889893\n",
      "Actor Loss: -5.868765242666996e-07\n",
      "Critic Loss: 0.9544755816459656\n",
      "Actor Loss: 2.7743253667722456e-06\n",
      "Critic Loss: 0.909020721912384\n",
      "Actor Loss: 2.427534582238877e-06\n",
      "Critic Loss: 0.8570870161056519\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.9229977130889893\n",
      "Actor Loss: 2.3475060970667982e-06\n",
      "Critic Loss: 0.856997549533844\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.8749406933784485\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Epoch 2: Average Steps per Episode: 8.75, Average Reward: 1.50, Average Loss: 0.9011\n",
      "Critic Loss: 0.874981164932251\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.8570870161056519\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.9090206027030945\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.9544755816459656\n",
      "Actor Loss: 3.98809288526536e-06\n",
      "Critic Loss: 0.9374120235443115\n",
      "Actor Loss: 1.0728836059570312e-06\n",
      "Critic Loss: 0.9333094954490662\n",
      "Actor Loss: 1.144409225162235e-06\n",
      "Critic Loss: 0.498185396194458\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9544755816459656\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.8332189917564392\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8999882936477661\n",
      "Actor Loss: 0.6027507781982422\n",
      "Critic Loss: 0.8570868968963623\n",
      "Actor Loss: 2.7247838261246216e-06\n",
      "Critic Loss: 0.909020721912384\n",
      "Actor Loss: 1.0403720125395921e-06\n",
      "Epoch 3: Average Steps per Episode: 8.33, Average Reward: 1.50, Average Loss: 0.9184\n",
      "Critic Loss: 0.9374120831489563\n",
      "Actor Loss: -1.1920928955078125e-07\n",
      "Critic Loss: 0.909014105796814\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.8570810556411743\n",
      "Actor Loss: -3.056105136871338\n",
      "Critic Loss: 0.8570869565010071\n",
      "Actor Loss: 1.6348703866242431e-06\n",
      "Critic Loss: 0.8570868968963623\n",
      "Actor Loss: -5.449567765936081e-07\n",
      "Critic Loss: 0.856997549533844\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.8999881744384766\n",
      "Actor Loss: 0.6027523279190063\n",
      "Critic Loss: 0.874981164932251\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.49818533658981323\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8749812841415405\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.8332189917564392\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9090138673782349\n",
      "Actor Loss: -0.0\n",
      "Epoch 4: Average Steps per Episode: 8.50, Average Reward: 0.00, Average Loss: 0.6426\n",
      "Critic Loss: 0.8570870161056519\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.8332189917564392\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9523545503616333\n",
      "Actor Loss: 9.082612706379223e-08\n",
      "Critic Loss: 0.8749812245368958\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.8332189321517944\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9333096146583557\n",
      "Actor Loss: 2.5431316430513107e-07\n",
      "Critic Loss: 0.8999881148338318\n",
      "Actor Loss: 0.6027538180351257\n",
      "Critic Loss: 0.8570939302444458\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8570939898490906\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.8999881148338318\n",
      "Actor Loss: 0.6027536392211914\n",
      "Critic Loss: 0.9333095550537109\n",
      "Actor Loss: 1.0172526572205243e-06\n",
      "Critic Loss: 0.9090728759765625\n",
      "Actor Loss: -1.7339533542326535e-07\n",
      "Epoch 5: Average Steps per Episode: 10.30, Average Reward: 1.50, Average Loss: 0.9872\n",
      "Critic Loss: 0.8570808172225952\n",
      "Actor Loss: -3.05608868598938\n",
      "Critic Loss: 0.8570939898490906\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.8569974303245544\n",
      "Actor Loss: 1.3623919414840202e-07\n",
      "Critic Loss: 0.9523546099662781\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.85708087682724\n",
      "Actor Loss: -3.0560808181762695\n",
      "Critic Loss: 0.9090138673782349\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9523544311523438\n",
      "Actor Loss: 4.541306282135338e-07\n",
      "Critic Loss: 0.9284989833831787\n",
      "Actor Loss: -1.498631149843277e-06\n",
      "Critic Loss: 0.9411514401435852\n",
      "Actor Loss: 1.1219697881870161e-07\n",
      "Critic Loss: 0.9284989237785339\n",
      "Actor Loss: -3.065381974920456e-07\n",
      "Critic Loss: 0.856997549533844\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.9333094358444214\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Epoch 6: Average Steps per Episode: 11.17, Average Reward: 2.50, Average Loss: 0.3932\n",
      "Critic Loss: 0.9411514401435852\n",
      "Actor Loss: 3.365909435615322e-07\n",
      "Critic Loss: 0.9284988641738892\n",
      "Actor Loss: 2.588544703030493e-06\n",
      "Critic Loss: 0.9285174608230591\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9374754428863525\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Critic Loss: 0.9285174012184143\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.874981164932251\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9411500096321106\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8570939898490906\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.8570810556411743\n",
      "Actor Loss: -3.056044816970825\n",
      "Critic Loss: 0.9284991025924683\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9411513209342957\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9411500692367554\n",
      "Actor Loss: 6.731818871230644e-07\n",
      "Epoch 7: Average Steps per Episode: 11.93, Average Reward: 3.00, Average Loss: 0.6624\n",
      "Critic Loss: 0.9090139269828796\n",
      "Actor Loss: 1.7339533542326535e-06\n",
      "Critic Loss: 0.9285174012184143\n",
      "Actor Loss: 2.043587983280304e-06\n",
      "Critic Loss: 0.9523544311523438\n",
      "Actor Loss: -9.082612564270676e-07\n",
      "Critic Loss: 0.8570939898490906\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9090137481689453\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9523545503616333\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9411513805389404\n",
      "Actor Loss: 5.609849154097901e-07\n",
      "Critic Loss: 0.85708087682724\n",
      "Actor Loss: -3.056021213531494\n",
      "Critic Loss: 0.9230562448501587\n",
      "Actor Loss: -6.455641596403439e-06\n",
      "Critic Loss: 0.94115149974823\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8570935726165771\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.874981164932251\n",
      "Actor Loss: -0.0\n",
      "Epoch 8: Average Steps per Episode: 11.50, Average Reward: 1.50, Average Loss: 0.6539\n",
      "Critic Loss: 0.9090161919593811\n",
      "Actor Loss: 0.025262486189603806\n",
      "Critic Loss: 0.7498447299003601\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.8570936322212219\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9166135787963867\n",
      "Actor Loss: 7.947286349008209e-07\n",
      "Critic Loss: 0.9090180993080139\n",
      "Actor Loss: 0.06275124847888947\n",
      "Critic Loss: 0.7498459815979004\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Critic Loss: 0.9411501288414001\n",
      "Actor Loss: 2.2439395763740322e-07\n",
      "Critic Loss: 0.9230561256408691\n",
      "Actor Loss: -0.03649403527379036\n",
      "Critic Loss: 0.9090182781219482\n",
      "Actor Loss: 0.08487701416015625\n",
      "Critic Loss: 0.928500235080719\n",
      "Actor Loss: 9.877342108666198e-07\n",
      "Critic Loss: 0.9411500692367554\n",
      "Actor Loss: -4.4878791527480644e-07\n",
      "Critic Loss: 0.9374750852584839\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Epoch 9: Average Steps per Episode: 11.56, Average Reward: 0.50, Average Loss: 0.9007\n",
      "Critic Loss: 0.947313129901886\n",
      "Actor Loss: 2.409282387816347e-06\n",
      "Critic Loss: 0.7498445510864258\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.9411514401435852\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9285586476325989\n",
      "Actor Loss: 0.9032406806945801\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: 7.947286349008209e-07\n",
      "Critic Loss: 0.9166144132614136\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9285585284233093\n",
      "Actor Loss: 0.9033814072608948\n",
      "Critic Loss: 0.9285000562667847\n",
      "Actor Loss: 2.043587983280304e-06\n",
      "Critic Loss: 0.9473130106925964\n",
      "Actor Loss: -1.2046411939081736e-06\n",
      "Critic Loss: 0.9411500692367554\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.9411513209342957\n",
      "Actor Loss: -2.2439395763740322e-07\n",
      "Epoch 10: Average Steps per Episode: 12.05, Average Reward: 3.00, Average Loss: 1.0675\n",
      "Critic Loss: 0.923056423664093\n",
      "Actor Loss: -0.048575181514024734\n",
      "Critic Loss: 0.9374912977218628\n",
      "Actor Loss: 0.3049314022064209\n",
      "Critic Loss: 0.9473130106925964\n",
      "Actor Loss: 1.3050280358584132e-06\n",
      "Critic Loss: 0.9285587072372437\n",
      "Actor Loss: 0.9036124348640442\n",
      "Critic Loss: 0.8888380527496338\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9230562448501587\n",
      "Actor Loss: -0.04969083517789841\n",
      "Critic Loss: 0.9473130106925964\n",
      "Actor Loss: -5.019338686906849e-07\n",
      "Critic Loss: 0.9374911785125732\n",
      "Actor Loss: 0.3055868148803711\n",
      "Critic Loss: 0.9285181164741516\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9285584688186646\n",
      "Actor Loss: 0.9035061597824097\n",
      "Critic Loss: 0.9230563640594482\n",
      "Actor Loss: -0.05057789757847786\n",
      "Critic Loss: 0.8888378739356995\n",
      "Actor Loss: 2.119276274470394e-07\n",
      "Epoch 11: Average Steps per Episode: 12.09, Average Reward: 5.00, Average Loss: 1.1142\n",
      "Critic Loss: 0.9374911785125732\n",
      "Actor Loss: 0.30588388442993164\n",
      "Critic Loss: 0.9166142344474792\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9374752640724182\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Critic Loss: 0.9166136980056763\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9374743103981018\n",
      "Actor Loss: 0.03177070617675781\n",
      "Critic Loss: 0.9230562448501587\n",
      "Actor Loss: -0.05134876072406769\n",
      "Critic Loss: 0.9374752044677734\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Critic Loss: 0.8888379335403442\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9166136980056763\n",
      "Actor Loss: -1.5894572413799324e-07\n",
      "Critic Loss: 0.8888378143310547\n",
      "Actor Loss: 2.1192763597355224e-06\n",
      "Critic Loss: 0.937475323677063\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.7498456835746765\n",
      "Actor Loss: -1.1920928955078125e-07\n",
      "Epoch 12: Average Steps per Episode: 12.50, Average Reward: 2.00, Average Loss: 0.9312\n",
      "Critic Loss: 0.9374746084213257\n",
      "Actor Loss: 0.031949520111083984\n",
      "Critic Loss: 0.9285585880279541\n",
      "Actor Loss: 0.9033846855163574\n",
      "Critic Loss: 0.8888151049613953\n",
      "Actor Loss: 1.6954210195763153e-06\n",
      "Critic Loss: 0.9374909996986389\n",
      "Actor Loss: 0.3061819076538086\n",
      "Critic Loss: 0.9285587072372437\n",
      "Actor Loss: 0.9033740162849426\n",
      "Critic Loss: 0.888837993144989\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.923056423664093\n",
      "Actor Loss: -0.05188692733645439\n",
      "Critic Loss: 0.9443934559822083\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.8888379335403442\n",
      "Actor Loss: 3.178914312229608e-06\n",
      "Critic Loss: 0.9374913573265076\n",
      "Actor Loss: 0.30621814727783203\n",
      "Critic Loss: 0.7498458027839661\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.9230562448501587\n",
      "Actor Loss: -0.05196058005094528\n",
      "Epoch 13: Average Steps per Episode: 12.23, Average Reward: 0.50, Average Loss: 1.1020\n",
      "Critic Loss: 0.9166142344474792\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.9443934559822083\n",
      "Actor Loss: -1.059638137235197e-07\n",
      "Critic Loss: 0.88881516456604\n",
      "Actor Loss: -1.0596381798677612e-06\n",
      "Critic Loss: 0.9166135787963867\n",
      "Actor Loss: 2.384185791015625e-06\n",
      "Critic Loss: 0.9166141748428345\n",
      "Actor Loss: -1.1126201115985168e-06\n",
      "Critic Loss: 0.937491238117218\n",
      "Actor Loss: 0.3062434196472168\n",
      "Critic Loss: 0.88881516456604\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.857125461101532\n",
      "Actor Loss: 1.559680700302124\n",
      "Critic Loss: 0.8888280987739563\n",
      "Actor Loss: -4.238552548940788e-07\n",
      "Critic Loss: 0.9285586476325989\n",
      "Actor Loss: 0.9033529162406921\n",
      "Critic Loss: 0.9374911785125732\n",
      "Actor Loss: 0.30625081062316895\n",
      "Critic Loss: 0.6665067076683044\n",
      "Actor Loss: 3.814697265625e-06\n",
      "Epoch 14: Average Steps per Episode: 11.71, Average Reward: 1.50, Average Loss: 1.1469\n",
      "Critic Loss: 0.8888379335403442\n",
      "Actor Loss: -1.4834934063401306e-06\n",
      "Critic Loss: 0.8332698345184326\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9473130106925964\n",
      "Actor Loss: -2.0077354179193208e-07\n",
      "Critic Loss: 0.88881516456604\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.8571256399154663\n",
      "Actor Loss: 1.559646725654602\n",
      "Critic Loss: 0.8888280391693115\n",
      "Actor Loss: -5.298190899338806e-07\n",
      "Critic Loss: 0.9374746084213257\n",
      "Actor Loss: 0.032088279724121094\n",
      "Critic Loss: 0.8332697749137878\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.9374746084213257\n",
      "Actor Loss: 0.0320897102355957\n",
      "Critic Loss: 0.9443935751914978\n",
      "Actor Loss: 8.477105097881577e-07\n",
      "Critic Loss: 0.9374911785125732\n",
      "Actor Loss: 0.3062584400177002\n",
      "Critic Loss: 0.8888378739356995\n",
      "Actor Loss: -4.238552548940788e-07\n",
      "Epoch 15: Average Steps per Episode: 11.33, Average Reward: 0.50, Average Loss: 1.0594\n",
      "Critic Loss: 0.8332780599594116\n",
      "Actor Loss: 2.2252402231970336e-06\n",
      "Critic Loss: 0.8571255803108215\n",
      "Actor Loss: 1.5596336126327515\n",
      "Critic Loss: 0.8332698345184326\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.7999438643455505\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888152241706848\n",
      "Actor Loss: 2.119276274470394e-07\n",
      "Critic Loss: 0.9374912977218628\n",
      "Actor Loss: 0.3062601089477539\n",
      "Critic Loss: 0.8888379335403442\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.7999439239501953\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8888281583786011\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.6665066480636597\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Critic Loss: 0.8332698345184326\n",
      "Actor Loss: -9.5367431640625e-07\n",
      "Critic Loss: 0.888837993144989\n",
      "Actor Loss: 2.1192763597355224e-06\n",
      "Epoch 16: Average Steps per Episode: 11.28, Average Reward: 0.00, Average Loss: 0.9985\n",
      "Critic Loss: 0.8571255803108215\n",
      "Actor Loss: 1.5596270561218262\n",
      "Critic Loss: 0.9374321699142456\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.8887320756912231\n",
      "Actor Loss: 0.8216955661773682\n",
      "Critic Loss: 0.8888280987739563\n",
      "Actor Loss: 7.417467031700653e-07\n",
      "Critic Loss: 0.9374321103096008\n",
      "Actor Loss: -2.384185791015625e-07\n",
      "Critic Loss: 0.8888151049613953\n",
      "Actor Loss: -1.4834934063401306e-06\n",
      "Critic Loss: 0.8332778811454773\n",
      "Actor Loss: 2.2252402231970336e-06\n",
      "Critic Loss: 0.857125461101532\n",
      "Actor Loss: 1.5596249103546143\n",
      "Critic Loss: 0.937432050704956\n",
      "Actor Loss: -2.6226043701171875e-06\n",
      "Critic Loss: 0.666506826877594\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.857125461101532\n",
      "Actor Loss: 1.5596249103546143\n",
      "Critic Loss: 0.7998987436294556\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Epoch 17: Average Steps per Episode: 11.03, Average Reward: 0.00, Average Loss: 1.3209\n",
      "Critic Loss: 0.9523110389709473\n",
      "Actor Loss: -0.371639609336853\n",
      "Critic Loss: 0.6665067672729492\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9166437387466431\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.7999439239501953\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888281583786011\n",
      "Actor Loss: -1.059638137235197e-07\n",
      "Critic Loss: 0.9374322891235352\n",
      "Actor Loss: -2.0265579223632812e-06\n",
      "Critic Loss: 0.857125461101532\n",
      "Actor Loss: 1.5596249103546143\n",
      "Critic Loss: 0.7999440431594849\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.8888150453567505\n",
      "Actor Loss: -8.477105097881577e-07\n",
      "Critic Loss: 0.9166436195373535\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.6665067672729492\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8332699537277222\n",
      "Actor Loss: -0.0\n",
      "Epoch 18: Average Steps per Episode: 11.33, Average Reward: 2.00, Average Loss: 0.9427\n",
      "Critic Loss: 0.857125461101532\n",
      "Actor Loss: 1.5596245527267456\n",
      "Critic Loss: 0.7999440431594849\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.6665067672729492\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9523110389709473\n",
      "Actor Loss: -0.3716403543949127\n",
      "Critic Loss: 0.952311098575592\n",
      "Actor Loss: -0.3716357946395874\n",
      "Critic Loss: 0.8570800423622131\n",
      "Actor Loss: -5.449567765936081e-07\n",
      "Critic Loss: 0.7999440431594849\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.7998986840248108\n",
      "Actor Loss: 1.5258789289873675e-06\n",
      "Critic Loss: 0.9166436195373535\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.7999440431594849\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.666506826877594\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8887320160865784\n",
      "Actor Loss: 0.8217048645019531\n",
      "Epoch 19: Average Steps per Episode: 11.32, Average Reward: 4.50, Average Loss: 0.9663\n",
      "Critic Loss: 0.8887320160865784\n",
      "Actor Loss: 0.8217018842697144\n",
      "Critic Loss: 0.9285176396369934\n",
      "Actor Loss: -2.3246681690216064\n",
      "Critic Loss: 0.8570799231529236\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.9166437387466431\n",
      "Actor Loss: 3.178914482759865e-07\n",
      "Critic Loss: 0.937432050704956\n",
      "Actor Loss: -1.1920928955078125e-06\n",
      "Critic Loss: 0.8332780599594116\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -0.37163618206977844\n",
      "Critic Loss: 0.7998988628387451\n",
      "Actor Loss: 2.670288040462765e-06\n",
      "Critic Loss: 0.9523108601570129\n",
      "Actor Loss: -0.3716330826282501\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9285176992416382\n",
      "Actor Loss: -2.324667453765869\n",
      "Critic Loss: 0.8332780599594116\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Epoch 20: Average Steps per Episode: 11.35, Average Reward: 1.50, Average Loss: 0.5131\n",
      "Critic Loss: 0.9285176396369934\n",
      "Actor Loss: -2.3246681690216064\n",
      "Critic Loss: 0.8570801019668579\n",
      "Actor Loss: 1.0899135531872162e-06\n",
      "Critic Loss: 0.9333208799362183\n",
      "Actor Loss: 0.6380903124809265\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9374322295188904\n",
      "Actor Loss: -1.9073486328125e-06\n",
      "Critic Loss: 0.9333208799362183\n",
      "Actor Loss: 0.6380900144577026\n",
      "Critic Loss: 0.8570801019668579\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9166436195373535\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.799898624420166\n",
      "Actor Loss: 4.57763690064894e-06\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: 1.9073486612342094e-07\n",
      "Critic Loss: 0.8570799231529236\n",
      "Actor Loss: -5.449567765936081e-07\n",
      "Epoch 21: Average Steps per Episode: 11.33, Average Reward: 2.50, Average Loss: 0.8060\n",
      "Critic Loss: 0.9333208203315735\n",
      "Actor Loss: 0.6380892992019653\n",
      "Critic Loss: 0.9523108601570129\n",
      "Actor Loss: -0.3716394305229187\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.8887319564819336\n",
      "Actor Loss: 0.8217048645019531\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: 1.9073486612342094e-07\n",
      "Critic Loss: 0.9374450445175171\n",
      "Actor Loss: -0.3133854866027832\n",
      "Critic Loss: 0.7998988032341003\n",
      "Actor Loss: 2.28881845032447e-06\n",
      "Critic Loss: 0.9374450445175171\n",
      "Actor Loss: -0.3133854866027832\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8887320756912231\n",
      "Actor Loss: 0.82170569896698\n",
      "Critic Loss: 0.9285175204277039\n",
      "Actor Loss: -2.3246665000915527\n",
      "Epoch 22: Average Steps per Episode: 11.75, Average Reward: 1.00, Average Loss: 0.8187\n",
      "Critic Loss: 0.9166436195373535\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: 1.9073486612342094e-07\n",
      "Critic Loss: 0.8999776244163513\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.9599122405052185\n",
      "Actor Loss: 0.3921829164028168\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.9285176396369934\n",
      "Actor Loss: -2.3246688842773438\n",
      "Critic Loss: 0.8749148845672607\n",
      "Actor Loss: -1.0728836059570312e-06\n",
      "Critic Loss: 0.9166436195373535\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.9285176396369934\n",
      "Actor Loss: -2.3246686458587646\n",
      "Critic Loss: 0.8570799231529236\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9374450445175171\n",
      "Actor Loss: -0.3133864402770996\n",
      "Epoch 23: Average Steps per Episode: 11.76, Average Reward: 4.50, Average Loss: 0.5291\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9333208799362183\n",
      "Actor Loss: 0.6380900144577026\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.944351851940155\n",
      "Actor Loss: 0.9997652173042297\n",
      "Critic Loss: 0.933320939540863\n",
      "Actor Loss: 0.6380897760391235\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: 1.9073486612342094e-07\n",
      "Critic Loss: 0.8749148845672607\n",
      "Actor Loss: -1.9073486328125e-06\n",
      "Critic Loss: 0.9599122405052185\n",
      "Actor Loss: 0.39218321442604065\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.9374918937683105\n",
      "Actor Loss: 0.4375520944595337\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: -1.9073486612342094e-07\n",
      "Critic Loss: 0.9285176396369934\n",
      "Actor Loss: -2.324666976928711\n",
      "Epoch 24: Average Steps per Episode: 11.85, Average Reward: 2.50, Average Loss: 0.9827\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: -1.9073486612342094e-07\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.9599122405052185\n",
      "Actor Loss: 0.39218321442604065\n",
      "Critic Loss: 0.9166141748428345\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9374918937683105\n",
      "Actor Loss: 0.4375520944595337\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.9285176396369934\n",
      "Actor Loss: -2.324666976928711\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.95644211769104\n",
      "Actor Loss: 1.7308940887451172\n",
      "Critic Loss: 0.8999802470207214\n",
      "Actor Loss: -1.9073486612342094e-07\n",
      "Critic Loss: 0.9374918937683105\n",
      "Actor Loss: 0.4375520944595337\n",
      "Epoch 25: Average Steps per Episode: 12.20, Average Reward: 2.00, Average Loss: 1.1053\n",
      "Critic Loss: 0.9599122405052185\n",
      "Actor Loss: 0.3921829164028168\n",
      "Critic Loss: 0.9374927282333374\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.95644211769104\n",
      "Actor Loss: 1.7308940887451172\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.8999775052070618\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.9374450445175171\n",
      "Actor Loss: -0.3133859634399414\n",
      "Critic Loss: 0.9374927282333374\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.8749148845672607\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.9599122405052185\n",
      "Actor Loss: 0.3921829164028168\n",
      "Critic Loss: 0.9374918937683105\n",
      "Actor Loss: 0.4375520944595337\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9166141748428345\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Epoch 26: Average Steps per Episode: 12.29, Average Reward: 7.50, Average Loss: 1.1007\n",
      "Critic Loss: 0.95644211769104\n",
      "Actor Loss: 1.7308940887451172\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.9374450445175171\n",
      "Actor Loss: -0.3133859634399414\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.8571234345436096\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.8888716101646423\n",
      "Actor Loss: 1.5341137647628784\n",
      "Epoch 27: Average Steps per Episode: 12.13, Average Reward: 3.00, Average Loss: 1.1263\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.95644211769104\n",
      "Actor Loss: 1.7308940887451172\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.8571234345436096\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.8749148845672607\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Epoch 28: Average Steps per Episode: 12.18, Average Reward: 4.00, Average Loss: 1.2279\n",
      "Critic Loss: 0.9443520307540894\n",
      "Actor Loss: 0.9997660517692566\n",
      "Critic Loss: 0.95644211769104\n",
      "Actor Loss: 1.7308940887451172\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Critic Loss: 0.749846875667572\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Critic Loss: 0.8999900221824646\n",
      "Actor Loss: -0.5491361618041992\n",
      "Critic Loss: 0.9090362191200256\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Epoch 29: Average Steps per Episode: 12.02, Average Reward: 0.50, Average Loss: 1.0839\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.749846875667572\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.9374927282333374\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.9473403096199036\n",
      "Actor Loss: -0.12217150628566742\n",
      "Critic Loss: 0.9444153308868408\n",
      "Actor Loss: 0.21017594635486603\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.8571234345436096\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Critic Loss: 0.9229913949966431\n",
      "Actor Loss: -1.163171410560608\n",
      "Critic Loss: 0.9090362191200256\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.8571234345436096\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Epoch 30: Average Steps per Episode: 12.22, Average Reward: 2.00, Average Loss: 0.9832\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.9499713778495789\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888716101646423\n",
      "Actor Loss: 1.5341137647628784\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Critic Loss: 0.9499713778495789\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9229913949966431\n",
      "Actor Loss: -1.163171410560608\n",
      "Critic Loss: 0.8888716101646423\n",
      "Actor Loss: 1.5341137647628784\n",
      "Critic Loss: 0.9374927282333374\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Critic Loss: 0.9230265021324158\n",
      "Actor Loss: -0.5298362374305725\n",
      "Critic Loss: 0.9499713778495789\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8571234345436096\n",
      "Actor Loss: -8.174351933121216e-07\n",
      "Epoch 31: Average Steps per Episode: 12.31, Average Reward: 2.00, Average Loss: 1.1951\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.9499713778495789\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Critic Loss: 0.9473443627357483\n",
      "Actor Loss: 0.4902086853981018\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.8888716101646423\n",
      "Actor Loss: 1.5341137647628784\n",
      "Critic Loss: 0.9090362191200256\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.749846875667572\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Epoch 32: Average Steps per Episode: 12.28, Average Reward: 1.50, Average Loss: 2.1964\n",
      "Critic Loss: 0.749846875667572\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.9499713778495789\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.749846875667572\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.9229913949966431\n",
      "Actor Loss: -1.163171410560608\n",
      "Critic Loss: 0.8888136744499207\n",
      "Actor Loss: -3.56895112991333\n",
      "Critic Loss: 0.9285582304000854\n",
      "Actor Loss: 0.15006910264492035\n",
      "Epoch 33: Average Steps per Episode: 12.18, Average Reward: 0.00, Average Loss: 1.8626\n",
      "Critic Loss: 0.9473443627357483\n",
      "Actor Loss: 0.4902086853981018\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9473403096199036\n",
      "Actor Loss: -0.12217150628566742\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.9090369343757629\n",
      "Actor Loss: -2.835993766784668\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Critic Loss: 0.9473443627357483\n",
      "Actor Loss: 0.4902086853981018\n",
      "Critic Loss: 0.749846875667572\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.9473403096199036\n",
      "Actor Loss: -0.12217150628566742\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Epoch 34: Average Steps per Episode: 12.12, Average Reward: 0.00, Average Loss: 1.4807\n",
      "Critic Loss: 0.9090369343757629\n",
      "Actor Loss: -2.835993766784668\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.9285488724708557\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9285488724708557\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.9411044716835022\n",
      "Actor Loss: 0.8541905879974365\n",
      "Critic Loss: 0.9473443627357483\n",
      "Actor Loss: 0.4902086853981018\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Epoch 35: Average Steps per Episode: 12.21, Average Reward: 1.50, Average Loss: 1.3287\n",
      "Critic Loss: 0.9499713778495789\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9090369343757629\n",
      "Actor Loss: -2.835993766784668\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.9285488724708557\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.9410958290100098\n",
      "Actor Loss: 0.446310818195343\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8999301791191101\n",
      "Actor Loss: 1.2523090839385986\n",
      "Epoch 36: Average Steps per Episode: 12.22, Average Reward: 0.50, Average Loss: 0.6258\n",
      "Critic Loss: 0.9090369343757629\n",
      "Actor Loss: -2.835993766784668\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9473443627357483\n",
      "Actor Loss: 0.4902086853981018\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9090369343757629\n",
      "Actor Loss: -2.835993766784668\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.7498314380645752\n",
      "Actor Loss: 4.885769844055176\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9164389371871948\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9473443627357483\n",
      "Actor Loss: 0.4902086853981018\n",
      "Epoch 37: Average Steps per Episode: 12.50, Average Reward: 0.50, Average Loss: 0.7158\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.9285488724708557\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9164389371871948\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9410958290100098\n",
      "Actor Loss: 0.446310818195343\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Critic Loss: 0.8888077139854431\n",
      "Actor Loss: 4.238552548940788e-07\n",
      "Critic Loss: 0.8888211846351624\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9696341156959534\n",
      "Actor Loss: -0.8760658502578735\n",
      "Epoch 38: Average Steps per Episode: 12.43, Average Reward: 6.50, Average Loss: 0.8734\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.7999626994132996\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.7999626994132996\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.9410958290100098\n",
      "Actor Loss: 0.446310818195343\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.9090369343757629\n",
      "Actor Loss: -2.835993766784668\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Epoch 39: Average Steps per Episode: 12.60, Average Reward: 1.50, Average Loss: 0.2269\n",
      "Critic Loss: 0.9285488724708557\n",
      "Actor Loss: -2.7247838829680404e-07\n",
      "Critic Loss: 0.9410958290100098\n",
      "Actor Loss: 0.446310818195343\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8999413847923279\n",
      "Actor Loss: 2.292255401611328\n",
      "Critic Loss: 0.9410958290100098\n",
      "Actor Loss: 0.446310818195343\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9696341156959534\n",
      "Actor Loss: -0.8760658502578735\n",
      "Critic Loss: 0.7999626994132996\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.9696341156959534\n",
      "Actor Loss: -0.8760658502578735\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Epoch 40: Average Steps per Episode: 12.57, Average Reward: 0.50, Average Loss: 1.0785\n",
      "Critic Loss: 0.9374468326568604\n",
      "Actor Loss: -2.365510940551758\n",
      "Critic Loss: 0.7999626994132996\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.9164389371871948\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.8888463377952576\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9696341156959534\n",
      "Actor Loss: -0.8760658502578735\n",
      "Epoch 41: Average Steps per Episode: 12.46, Average Reward: 0.50, Average Loss: 0.6417\n",
      "Critic Loss: 0.7498602867126465\n",
      "Actor Loss: -4.52082633972168\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Critic Loss: 0.7498602867126465\n",
      "Actor Loss: -4.52082633972168\n",
      "Critic Loss: 0.8999413847923279\n",
      "Actor Loss: 2.292255401611328\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.7999626994132996\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9164389371871948\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9332321882247925\n",
      "Actor Loss: 6.357829079206567e-06\n",
      "Epoch 42: Average Steps per Episode: 12.39, Average Reward: 0.00, Average Loss: 0.2817\n",
      "Critic Loss: 0.833299994468689\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8999413847923279\n",
      "Actor Loss: 2.292255401611328\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.7498602867126465\n",
      "Actor Loss: -4.52082633972168\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9333248138427734\n",
      "Actor Loss: -5.086263286102621e-07\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.7999626994132996\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9332321882247925\n",
      "Actor Loss: 6.357829079206567e-06\n",
      "Critic Loss: 0.8999413847923279\n",
      "Actor Loss: 2.292255401611328\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Epoch 43: Average Steps per Episode: 12.36, Average Reward: 0.00, Average Loss: 0.8841\n",
      "Critic Loss: 0.9284801483154297\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.9374516010284424\n",
      "Actor Loss: 5.364418029785156e-07\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9545163512229919\n",
      "Actor Loss: -0.5216799974441528\n",
      "Critic Loss: 0.8999413847923279\n",
      "Actor Loss: 2.292255401611328\n",
      "Epoch 44: Average Steps per Episode: 12.28, Average Reward: 0.00, Average Loss: 0.8896\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.9284801483154297\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.7498602867126465\n",
      "Actor Loss: -4.52082633972168\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: 2.098083541568485e-06\n",
      "Critic Loss: 0.8570517301559448\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9229836463928223\n",
      "Actor Loss: 0.48758184909820557\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.833299994468689\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Epoch 45: Average Steps per Episode: 12.14, Average Reward: 2.50, Average Loss: 0.1487\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: 2.098083541568485e-06\n",
      "Critic Loss: 0.8332480788230896\n",
      "Actor Loss: 1.748402951307071e-06\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.833299994468689\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.8332480788230896\n",
      "Actor Loss: 1.748402951307071e-06\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9284801483154297\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.8332480788230896\n",
      "Actor Loss: 1.748402951307071e-06\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Epoch 46: Average Steps per Episode: 12.37, Average Reward: 2.00, Average Loss: 0.6735\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.9332321882247925\n",
      "Actor Loss: 6.357829079206567e-06\n",
      "Critic Loss: 0.9089850187301636\n",
      "Actor Loss: -0.20854517817497253\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.8570517301559448\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.8570517301559448\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.9089850187301636\n",
      "Actor Loss: -0.20854517817497253\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Epoch 47: Average Steps per Episode: 12.32, Average Reward: 0.00, Average Loss: 0.5739\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8570517301559448\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9284801483154297\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.874943196773529\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Epoch 48: Average Steps per Episode: 12.21, Average Reward: 1.50, Average Loss: 0.3700\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.8333245515823364\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8332480788230896\n",
      "Actor Loss: 1.748402951307071e-06\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Critic Loss: 0.8570517301559448\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9089850187301636\n",
      "Actor Loss: -0.20854517817497253\n",
      "Critic Loss: 0.9090018272399902\n",
      "Actor Loss: -2.571981430053711\n",
      "Critic Loss: 0.8570517301559448\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.9090368151664734\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Epoch 49: Average Steps per Episode: 12.23, Average Reward: 0.50, Average Loss: -0.2408\n",
      "Critic Loss: 0.9374477863311768\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.8749526739120483\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Critic Loss: 0.874940037727356\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Critic Loss: 0.8749526739120483\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.9089850187301636\n",
      "Actor Loss: -0.20854517817497253\n",
      "Critic Loss: 0.749894917011261\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.874940037727356\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Epoch 50: Average Steps per Episode: 12.11, Average Reward: 0.50, Average Loss: 0.1179\n",
      "Critic Loss: 0.9523242712020874\n",
      "Actor Loss: -0.7930454015731812\n",
      "Critic Loss: 0.8749526739120483\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.9090368151664734\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.9089850187301636\n",
      "Actor Loss: -0.20854517817497253\n",
      "Critic Loss: 0.874940037727356\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.970533013343811\n",
      "Actor Loss: 0.05062310770153999\n",
      "Critic Loss: 0.749894917011261\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9090709686279297\n",
      "Actor Loss: 3.467906708465307e-07\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Epoch 51: Average Steps per Episode: 12.18, Average Reward: 0.50, Average Loss: 0.4532\n",
      "Critic Loss: 0.9583030939102173\n",
      "Actor Loss: 0.12885475158691406\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Critic Loss: 0.8749526739120483\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.9443577527999878\n",
      "Actor Loss: 0.7454535961151123\n",
      "Critic Loss: 0.9583030939102173\n",
      "Actor Loss: 0.12885475158691406\n",
      "Critic Loss: 0.9089850187301636\n",
      "Actor Loss: -0.20854517817497253\n",
      "Critic Loss: 0.8749526739120483\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.749894917011261\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9583030939102173\n",
      "Actor Loss: 0.12885475158691406\n",
      "Critic Loss: 0.8887643814086914\n",
      "Actor Loss: -3.6610138416290283\n",
      "Critic Loss: 0.8749526739120483\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Epoch 52: Average Steps per Episode: 12.35, Average Reward: 2.50, Average Loss: 0.2970\n",
      "Critic Loss: 0.9090368151664734\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.8999027609825134\n",
      "Actor Loss: 3.2269065380096436\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.8333198428153992\n",
      "Actor Loss: -0.7518510818481445\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.9090368151664734\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9443577527999878\n",
      "Actor Loss: 0.7454535961151123\n",
      "Critic Loss: 0.749894917011261\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9374477863311768\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.874940037727356\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.9523242712020874\n",
      "Actor Loss: -0.7930454015731812\n",
      "Epoch 53: Average Steps per Episode: 12.51, Average Reward: 4.00, Average Loss: 1.1070\n",
      "Critic Loss: 0.9090368151664734\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9285494089126587\n",
      "Actor Loss: -0.37103110551834106\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.874940037727356\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9583030939102173\n",
      "Actor Loss: 0.12885475158691406\n",
      "Critic Loss: 0.9523242712020874\n",
      "Actor Loss: -0.7930454015731812\n",
      "Critic Loss: 0.9374477863311768\n",
      "Actor Loss: -3.5762786865234375e-07\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.9443577527999878\n",
      "Actor Loss: 0.7454535961151123\n",
      "Critic Loss: 0.8999027609825134\n",
      "Actor Loss: 3.2269065380096436\n",
      "Epoch 54: Average Steps per Episode: 12.60, Average Reward: 0.00, Average Loss: 1.2098\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8999027609825134\n",
      "Actor Loss: 3.2269065380096436\n",
      "Critic Loss: 0.749894917011261\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.8999027609825134\n",
      "Actor Loss: 3.2269065380096436\n",
      "Critic Loss: 0.874940037727356\n",
      "Actor Loss: 7.152557373046875e-07\n",
      "Critic Loss: 0.8999027609825134\n",
      "Actor Loss: 3.2269065380096436\n",
      "Critic Loss: 0.9473221302032471\n",
      "Actor Loss: -0.5837003588676453\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Epoch 55: Average Steps per Episode: 12.60, Average Reward: 0.00, Average Loss: 1.7209\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.9443577527999878\n",
      "Actor Loss: 0.7454535961151123\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.8999027609825134\n",
      "Actor Loss: 3.2269065380096436\n",
      "Critic Loss: 0.9285494089126587\n",
      "Actor Loss: -0.37103110551834106\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.9523242712020874\n",
      "Actor Loss: -0.7930454015731812\n",
      "Critic Loss: 0.9583030939102173\n",
      "Actor Loss: 0.12885475158691406\n",
      "Epoch 56: Average Steps per Episode: 12.61, Average Reward: 1.50, Average Loss: 1.2360\n",
      "Critic Loss: 0.923023521900177\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.9285494089126587\n",
      "Actor Loss: -0.37103110551834106\n",
      "Critic Loss: 0.923023521900177\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.9583030939102173\n",
      "Actor Loss: 0.12885475158691406\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.8748599290847778\n",
      "Actor Loss: -1.5497207641601562e-06\n",
      "Epoch 57: Average Steps per Episode: 12.52, Average Reward: 0.00, Average Loss: 0.8967\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.857074499130249\n",
      "Actor Loss: 0.5017024874687195\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.8748599290847778\n",
      "Actor Loss: -1.5497207641601562e-06\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9642660617828369\n",
      "Actor Loss: 0.4179600477218628\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9285494089126587\n",
      "Actor Loss: -0.37103110551834106\n",
      "Critic Loss: 0.9473221302032471\n",
      "Actor Loss: -0.5837003588676453\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Epoch 58: Average Steps per Episode: 12.37, Average Reward: 0.00, Average Loss: 0.8453\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8748599290847778\n",
      "Actor Loss: -1.5497207641601562e-06\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.9090032577514648\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9582613706588745\n",
      "Actor Loss: -0.01587589643895626\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.6665109395980835\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.8997349143028259\n",
      "Actor Loss: -5.245208853921213e-07\n",
      "Critic Loss: 0.8748599290847778\n",
      "Actor Loss: -1.5497207641601562e-06\n",
      "Critic Loss: 0.857074499130249\n",
      "Actor Loss: 0.5017024874687195\n",
      "Epoch 59: Average Steps per Episode: 12.35, Average Reward: 0.50, Average Loss: 0.7577\n",
      "Critic Loss: 0.8749309778213501\n",
      "Actor Loss: -3.9175801277160645\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.8332616090774536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9230581521987915\n",
      "Actor Loss: -0.46718189120292664\n",
      "Critic Loss: 0.857074499130249\n",
      "Actor Loss: 0.5017024874687195\n",
      "Critic Loss: 0.923023521900177\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.8748599290847778\n",
      "Actor Loss: -1.5497207641601562e-06\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.8997349143028259\n",
      "Actor Loss: -5.245208853921213e-07\n",
      "Critic Loss: 0.6665109395980835\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Epoch 60: Average Steps per Episode: 12.29, Average Reward: 0.00, Average Loss: 0.2196\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8748599290847778\n",
      "Actor Loss: -1.5497207641601562e-06\n",
      "Critic Loss: 0.923023521900177\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.6665109395980835\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.8749309778213501\n",
      "Actor Loss: -3.9175801277160645\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.7498843669891357\n",
      "Actor Loss: 0.045932769775390625\n",
      "Critic Loss: 0.8749309778213501\n",
      "Actor Loss: -3.9175801277160645\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.6665109395980835\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Epoch 61: Average Steps per Episode: 12.16, Average Reward: 0.50, Average Loss: -0.0005\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.9166170358657837\n",
      "Actor Loss: -0.7324090003967285\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Critic Loss: 0.7498843669891357\n",
      "Actor Loss: 0.045932769775390625\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.7499715089797974\n",
      "Actor Loss: 2.270752429962158\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.7498843669891357\n",
      "Actor Loss: 0.045932769775390625\n",
      "Critic Loss: 0.6665109395980835\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.8749309778213501\n",
      "Actor Loss: -3.9175801277160645\n",
      "Epoch 62: Average Steps per Episode: 12.10, Average Reward: 0.00, Average Loss: 0.0318\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.8748971819877625\n",
      "Actor Loss: 1.0657451152801514\n",
      "Critic Loss: 0.799884021282196\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.8749309778213501\n",
      "Actor Loss: -3.9175801277160645\n",
      "Epoch 63: Average Steps per Episode: 12.09, Average Reward: 1.50, Average Loss: 0.0425\n",
      "Critic Loss: 0.7498843669891357\n",
      "Actor Loss: 0.045932769775390625\n",
      "Critic Loss: 0.9374757409095764\n",
      "Actor Loss: -0.36582183837890625\n",
      "Critic Loss: 0.7499715089797974\n",
      "Actor Loss: 2.270752429962158\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.9374757409095764\n",
      "Actor Loss: -0.36582183837890625\n",
      "Critic Loss: 0.8997349143028259\n",
      "Actor Loss: -5.245208853921213e-07\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.8997349143028259\n",
      "Actor Loss: -5.245208853921213e-07\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.7499715089797974\n",
      "Actor Loss: 2.270752429962158\n",
      "Epoch 64: Average Steps per Episode: 12.07, Average Reward: 1.50, Average Loss: 0.7459\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Critic Loss: 0.7498843669891357\n",
      "Actor Loss: 0.045932769775390625\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.7499715089797974\n",
      "Actor Loss: 2.270752429962158\n",
      "Critic Loss: 0.6665799021720886\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8332681655883789\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.8748971819877625\n",
      "Actor Loss: 1.0657451152801514\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Critic Loss: 0.8999202847480774\n",
      "Actor Loss: -2.6742897033691406\n",
      "Critic Loss: 0.6665799021720886\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9374757409095764\n",
      "Actor Loss: -0.36582183837890625\n",
      "Epoch 65: Average Steps per Episode: 12.02, Average Reward: 0.00, Average Loss: 0.5452\n",
      "Critic Loss: 0.7498843669891357\n",
      "Actor Loss: 0.045932769775390625\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.8332681655883789\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.7499715089797974\n",
      "Actor Loss: 2.270752429962158\n",
      "Critic Loss: 0.9284937381744385\n",
      "Actor Loss: -1.5380772352218628\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.6665799021720886\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.7499715089797974\n",
      "Actor Loss: 2.270752429962158\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Epoch 66: Average Steps per Episode: 12.08, Average Reward: 1.50, Average Loss: 0.6267\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.8332681655883789\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Critic Loss: 0.9374757409095764\n",
      "Actor Loss: -0.36582183837890625\n",
      "Critic Loss: 0.8570764064788818\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.9499484896659851\n",
      "Actor Loss: 0.33568739891052246\n",
      "Critic Loss: 0.8999497294425964\n",
      "Actor Loss: 5.722046125811175e-07\n",
      "Critic Loss: 0.8998816609382629\n",
      "Actor Loss: -1.8465029001235962\n",
      "Epoch 67: Average Steps per Episode: 12.12, Average Reward: 1.00, Average Loss: 0.2205\n",
      "Critic Loss: 0.9499484896659851\n",
      "Actor Loss: 0.33568739891052246\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.6665799021720886\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Critic Loss: 0.8999497294425964\n",
      "Actor Loss: 5.722046125811175e-07\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.9284937381744385\n",
      "Actor Loss: -1.5380772352218628\n",
      "Critic Loss: 0.9499484896659851\n",
      "Actor Loss: 0.33568739891052246\n",
      "Critic Loss: 0.8332681655883789\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9333118796348572\n",
      "Actor Loss: -0.0021578471641987562\n",
      "Critic Loss: 0.8748971819877625\n",
      "Actor Loss: 1.0657451152801514\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Epoch 68: Average Steps per Episode: 12.14, Average Reward: 1.50, Average Loss: 0.5004\n",
      "Critic Loss: 0.9374757409095764\n",
      "Actor Loss: -0.36582183837890625\n",
      "Critic Loss: 0.7498945593833923\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.8332681655883789\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9499484896659851\n",
      "Actor Loss: 0.33568739891052246\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.7498945593833923\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9499484896659851\n",
      "Actor Loss: 0.33568739891052246\n",
      "Critic Loss: 0.9164321422576904\n",
      "Actor Loss: -2.2514305114746094\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43936434388160706\n",
      "Critic Loss: 0.8999497294425964\n",
      "Actor Loss: 5.722046125811175e-07\n",
      "Critic Loss: 0.666492223739624\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Epoch 69: Average Steps per Episode: 12.01, Average Reward: 0.00, Average Loss: 0.4495\n",
      "Critic Loss: 0.8887770771980286\n",
      "Actor Loss: -2.8189408779144287\n",
      "Critic Loss: 0.9583052396774292\n",
      "Actor Loss: -0.3885149359703064\n",
      "Critic Loss: 0.7498945593833923\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.6664923429489136\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.7498944997787476\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.6665560007095337\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8570865392684937\n",
      "Actor Loss: 1.3623919130623108e-06\n",
      "Critic Loss: 0.6665798425674438\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9499740600585938\n",
      "Actor Loss: -0.4393559396266937\n",
      "Critic Loss: 0.899949848651886\n",
      "Actor Loss: 1.9073486612342094e-07\n",
      "Critic Loss: 0.9164320826530457\n",
      "Actor Loss: -2.251479148864746\n",
      "Critic Loss: 0.6665799021720886\n",
      "Actor Loss: -0.0\n",
      "Epoch 70: Average Steps per Episode: 11.96, Average Reward: 0.00, Average Loss: 0.3115\n",
      "Critic Loss: 0.8570786118507385\n",
      "Actor Loss: -0.31397002935409546\n",
      "Critic Loss: 0.9499740600585938\n",
      "Actor Loss: -0.4393535554409027\n",
      "Critic Loss: 0.6664924025535583\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.7999211549758911\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.6664922833442688\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.857078492641449\n",
      "Actor Loss: -0.3139907419681549\n",
      "Critic Loss: 0.949974000453949\n",
      "Actor Loss: -0.43935129046440125\n",
      "Critic Loss: 0.8570864796638489\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.8887768387794495\n",
      "Actor Loss: -2.818981647491455\n",
      "Critic Loss: 0.6665558815002441\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9499484896659851\n",
      "Actor Loss: 0.33566662669181824\n",
      "Critic Loss: 0.6664922833442688\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Epoch 71: Average Steps per Episode: 11.87, Average Reward: 0.50, Average Loss: 0.4822\n",
      "Critic Loss: 0.6665558218955994\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.7498945593833923\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.7999213337898254\n",
      "Actor Loss: 1.144409225162235e-06\n",
      "Critic Loss: 0.9583050608634949\n",
      "Actor Loss: -0.388502299785614\n",
      "Critic Loss: 0.8570784330368042\n",
      "Actor Loss: -0.31403160095214844\n",
      "Critic Loss: 0.8570863604545593\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.749894380569458\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.7999212741851807\n",
      "Actor Loss: 7.629394644936838e-07\n",
      "Critic Loss: 0.8999496698379517\n",
      "Actor Loss: 1.5258789289873675e-06\n",
      "Critic Loss: 0.8570863604545593\n",
      "Actor Loss: 2.7247838829680404e-07\n",
      "Critic Loss: 0.666492223739624\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.6665557622909546\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Epoch 72: Average Steps per Episode: 11.75, Average Reward: 0.00, Average Loss: 0.7355\n",
      "Critic Loss: 0.6665558815002441\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.6665341854095459\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.8570783734321594\n",
      "Actor Loss: -0.3141714036464691\n",
      "Critic Loss: 0.6663838624954224\n",
      "Actor Loss: 4.450480446394067e-06\n",
      "Critic Loss: 0.8887770771980286\n",
      "Actor Loss: -2.819044828414917\n",
      "Critic Loss: 0.7999212741851807\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.958305299282074\n",
      "Actor Loss: -0.3884863257408142\n",
      "Critic Loss: 0.666492223739624\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9583052396774292\n",
      "Actor Loss: -0.3884807527065277\n",
      "Critic Loss: 0.6663839221000671\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.6664922833442688\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9332680702209473\n",
      "Actor Loss: 1.4177404642105103\n",
      "Epoch 73: Average Steps per Episode: 11.86, Average Reward: 0.00, Average Loss: 0.5752\n",
      "Critic Loss: 0.8570864200592041\n",
      "Actor Loss: -1.0899135531872162e-06\n",
      "Critic Loss: 0.6665341854095459\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.857078492641449\n",
      "Actor Loss: -0.31456702947616577\n",
      "Critic Loss: 0.933268129825592\n",
      "Actor Loss: 1.4177364110946655\n",
      "Critic Loss: 0.9599272012710571\n",
      "Actor Loss: -0.5416377186775208\n",
      "Critic Loss: 0.8570863604545593\n",
      "Actor Loss: 8.174351933121216e-07\n",
      "Critic Loss: 0.749894380569458\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.6664921641349792\n",
      "Actor Loss: 3.814697265625e-06\n",
      "Critic Loss: 0.6663837432861328\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.6665342450141907\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.888776957988739\n",
      "Actor Loss: -2.819268226623535\n",
      "Critic Loss: 0.933268129825592\n",
      "Actor Loss: 1.4177325963974\n",
      "Epoch 74: Average Steps per Episode: 11.87, Average Reward: 3.50, Average Loss: 0.7385\n",
      "Critic Loss: 0.9090354442596436\n",
      "Actor Loss: 0.9323956370353699\n",
      "Critic Loss: 0.857078492641449\n",
      "Actor Loss: -0.31480517983436584\n",
      "Critic Loss: 0.7999212145805359\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.8887770175933838\n",
      "Actor Loss: -2.8192923069000244\n",
      "Critic Loss: 0.888776957988739\n",
      "Actor Loss: -2.819293975830078\n",
      "Critic Loss: 0.9090354442596436\n",
      "Actor Loss: 0.9324315190315247\n",
      "Critic Loss: 0.8570863604545593\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.6665341854095459\n",
      "Actor Loss: 1.9073486328125e-06\n",
      "Critic Loss: 0.7999210953712463\n",
      "Actor Loss: 3.814697322468419e-07\n",
      "Critic Loss: 0.6665341854095459\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.933268129825592\n",
      "Actor Loss: 1.4177312850952148\n",
      "Critic Loss: 0.8570786118507385\n",
      "Actor Loss: -0.3148762881755829\n",
      "Epoch 75: Average Steps per Episode: 11.87, Average Reward: 0.00, Average Loss: 0.5873\n",
      "Critic Loss: 0.933268129825592\n",
      "Actor Loss: 1.4177294969558716\n",
      "Critic Loss: 0.9230315685272217\n",
      "Actor Loss: 1.719269871711731\n",
      "Critic Loss: 0.8570784330368042\n",
      "Actor Loss: -0.3148871958255768\n",
      "Critic Loss: 0.9090352654457092\n",
      "Actor Loss: 0.9324644804000854\n",
      "Critic Loss: 0.9332681894302368\n",
      "Actor Loss: 1.4177297353744507\n",
      "Critic Loss: 0.9090673327445984\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9599272608757019\n",
      "Actor Loss: -0.5416967868804932\n",
      "Critic Loss: 0.9411599636077881\n",
      "Actor Loss: 0.2284119576215744\n",
      "Critic Loss: 0.6665343046188354\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9230106472969055\n",
      "Actor Loss: 1.35431969165802\n",
      "Critic Loss: 0.6663838624954224\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9599272608757019\n",
      "Actor Loss: -0.5416944026947021\n",
      "Epoch 76: Average Steps per Episode: 11.91, Average Reward: 2.50, Average Loss: 1.3544\n",
      "Critic Loss: 0.9285508394241333\n",
      "Actor Loss: 1.2258074283599854\n",
      "Critic Loss: 0.9090352654457092\n",
      "Actor Loss: 0.9324779510498047\n",
      "Critic Loss: 0.9332680702209473\n",
      "Actor Loss: 1.4177294969558716\n",
      "Critic Loss: 0.9230105876922607\n",
      "Actor Loss: 1.3543223142623901\n",
      "Critic Loss: 0.9090671539306641\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.6663837432861328\n",
      "Actor Loss: 3.1789145396032836e-06\n",
      "Critic Loss: 0.9411598443984985\n",
      "Actor Loss: 0.2284121811389923\n",
      "Critic Loss: 0.6665341854095459\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.9230106472969055\n",
      "Actor Loss: 1.3543236255645752\n",
      "Critic Loss: 0.9411598443984985\n",
      "Actor Loss: 0.22841262817382812\n",
      "Critic Loss: 0.9285507202148438\n",
      "Actor Loss: 1.2258068323135376\n",
      "Critic Loss: 0.9230314493179321\n",
      "Actor Loss: 1.7192680835723877\n",
      "Epoch 77: Average Steps per Episode: 11.84, Average Reward: 0.00, Average Loss: 1.6899\n",
      "Critic Loss: 0.9090672731399536\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9599273204803467\n",
      "Actor Loss: -0.5416997671127319\n",
      "Critic Loss: 0.9285507202148438\n",
      "Actor Loss: 1.225806713104248\n",
      "Critic Loss: 0.9090352654457092\n",
      "Actor Loss: 0.9324809312820435\n",
      "Critic Loss: 0.9284983277320862\n",
      "Actor Loss: -2.2696878910064697\n",
      "Critic Loss: 0.9285507202148438\n",
      "Actor Loss: 1.2258070707321167\n",
      "Critic Loss: 0.9230314493179321\n",
      "Actor Loss: 1.7192671298980713\n",
      "Critic Loss: 0.9090672731399536\n",
      "Actor Loss: 1.7339533542326535e-07\n",
      "Critic Loss: 0.9599274396896362\n",
      "Actor Loss: -0.5416951775550842\n",
      "Critic Loss: 0.9332680106163025\n",
      "Actor Loss: 1.4177290201187134\n",
      "Critic Loss: 0.9284982681274414\n",
      "Actor Loss: -2.269686222076416\n",
      "Critic Loss: 0.9090352654457092\n",
      "Actor Loss: 0.9324809312820435\n",
      "Epoch 78: Average Steps per Episode: 11.89, Average Reward: 1.50, Average Loss: 1.0798\n",
      "Critic Loss: 0.9230106472969055\n",
      "Actor Loss: 1.3543239831924438\n",
      "Critic Loss: 0.8888237476348877\n",
      "Actor Loss: -1.4208965301513672\n",
      "Critic Loss: 0.9285507798194885\n",
      "Actor Loss: 1.2258069515228271\n",
      "Critic Loss: 0.9230313897132874\n",
      "Actor Loss: 1.7192670106887817\n",
      "Critic Loss: 0.9090352654457092\n",
      "Actor Loss: 0.9324795603752136\n",
      "Critic Loss: 0.7498915195465088\n",
      "Actor Loss: 9.5367431640625e-07\n",
      "Critic Loss: 0.9284983277320862\n",
      "Actor Loss: -2.2696878910064697\n",
      "Critic Loss: 0.9285638928413391\n",
      "Actor Loss: 0.45281466841697693\n",
      "Critic Loss: 0.9411598443984985\n",
      "Actor Loss: 0.2284121811389923\n",
      "Critic Loss: 0.9285638928413391\n",
      "Actor Loss: 0.4528142511844635\n",
      "Critic Loss: 0.9285507798194885\n",
      "Actor Loss: 1.2258070707321167\n",
      "Critic Loss: 0.9284982681274414\n",
      "Actor Loss: -2.269688367843628\n",
      "Epoch 79: Average Steps per Episode: 11.97, Average Reward: 7.50, Average Loss: 1.0448\n",
      "Critic Loss: 0.9411178827285767\n",
      "Actor Loss: -1.1984872817993164\n",
      "Critic Loss: 0.8332313895225525\n",
      "Actor Loss: 2.3645293712615967\n",
      "Critic Loss: 0.9230314493179321\n",
      "Actor Loss: 1.71926748752594\n",
      "Critic Loss: 0.9284983277320862\n",
      "Actor Loss: -2.269688129425049\n",
      "Critic Loss: 0.8332314491271973\n",
      "Actor Loss: 2.3645272254943848\n",
      "Critic Loss: 0.8888236284255981\n",
      "Actor Loss: -1.4208976030349731\n",
      "Critic Loss: 0.9230107069015503\n",
      "Actor Loss: 1.3543246984481812\n",
      "Critic Loss: 0.9411598443984985\n",
      "Actor Loss: 0.22841240465641022\n",
      "Critic Loss: 0.941152036190033\n",
      "Actor Loss: 0.3215457797050476\n",
      "Critic Loss: 0.8888236284255981\n",
      "Actor Loss: -1.420896291732788\n",
      "Critic Loss: 0.928498387336731\n",
      "Actor Loss: -2.269688606262207\n",
      "Critic Loss: 0.7498915195465088\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Epoch 80: Average Steps per Episode: 11.96, Average Reward: 0.50, Average Loss: 0.8745\n",
      "Critic Loss: 0.928498387336731\n",
      "Actor Loss: -2.2696876525878906\n",
      "Critic Loss: 0.9285638928413391\n",
      "Actor Loss: 0.4528142511844635\n",
      "Critic Loss: 0.8332314491271973\n",
      "Actor Loss: 2.364528179168701\n",
      "Critic Loss: 0.9411598443984985\n",
      "Actor Loss: 0.2284119576215744\n",
      "Critic Loss: 0.7498915195465088\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.9284981489181519\n",
      "Actor Loss: -2.269688844680786\n",
      "Critic Loss: 0.9411522150039673\n",
      "Actor Loss: 0.321544885635376\n",
      "Critic Loss: 0.8888236284255981\n",
      "Actor Loss: -1.4208980798721313\n",
      "Critic Loss: 0.9230106472969055\n",
      "Actor Loss: 1.354325532913208\n",
      "Critic Loss: 0.9411522150039673\n",
      "Actor Loss: 0.3215457797050476\n",
      "Critic Loss: 0.9284982681274414\n",
      "Actor Loss: -2.2696876525878906\n",
      "Critic Loss: 0.9545251727104187\n",
      "Actor Loss: 0.2574707567691803\n",
      "Epoch 81: Average Steps per Episode: 11.98, Average Reward: 0.50, Average Loss: 0.6631\n",
      "Critic Loss: 0.9411522150039673\n",
      "Actor Loss: 0.3215457797050476\n",
      "Critic Loss: 0.9284983277320862\n",
      "Actor Loss: -2.2696878910064697\n",
      "Critic Loss: 0.9166011214256287\n",
      "Actor Loss: -0.7767524719238281\n",
      "Critic Loss: 0.8332030177116394\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.8332028388977051\n",
      "Actor Loss: 2.384185791015625e-07\n",
      "Critic Loss: 0.9545251131057739\n",
      "Actor Loss: 0.25747039914131165\n",
      "Critic Loss: 0.8332313299179077\n",
      "Actor Loss: 2.3645284175872803\n",
      "Critic Loss: 0.9411178827285767\n",
      "Actor Loss: -1.198485016822815\n",
      "Critic Loss: 0.9411520957946777\n",
      "Actor Loss: 0.3215455412864685\n",
      "Critic Loss: 0.9582866430282593\n",
      "Actor Loss: 0.7902346849441528\n",
      "Critic Loss: 0.8332030177116394\n",
      "Actor Loss: -7.947286206899662e-08\n",
      "Critic Loss: 0.8332313299179077\n",
      "Actor Loss: 2.364527702331543\n",
      "Epoch 82: Average Steps per Episode: 12.02, Average Reward: 1.00, Average Loss: 1.0769\n",
      "Critic Loss: 0.9284983277320862\n",
      "Actor Loss: -2.269686698913574\n",
      "Critic Loss: 0.9545249938964844\n",
      "Actor Loss: 0.25747036933898926\n",
      "Critic Loss: 0.9285638928413391\n",
      "Actor Loss: 0.4528142511844635\n",
      "Critic Loss: 0.8332030177116394\n",
      "Actor Loss: 1.5894572413799324e-07\n",
      "Critic Loss: 0.8332643508911133\n",
      "Actor Loss: 4.76837158203125e-07\n",
      "Critic Loss: 0.9332576394081116\n",
      "Actor Loss: 8.900960892788135e-07\n",
      "Critic Loss: 0.8332030177116394\n",
      "Actor Loss: 6.35782896551973e-07\n",
      "Critic Loss: 0.8749898076057434\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8749898076057434\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9411178827285767\n",
      "Actor Loss: -1.1984870433807373\n",
      "Critic Loss: 0.9332576394081116\n",
      "Actor Loss: 7.629395213371026e-07\n",
      "Critic Loss: 0.941152036190033\n",
      "Actor Loss: 0.3215455412864685\n",
      "Epoch 83: Average Steps per Episode: 11.96, Average Reward: 3.00, Average Loss: 0.6978\n",
      "Critic Loss: 0.9411178231239319\n",
      "Actor Loss: -1.1984847784042358\n",
      "Critic Loss: 0.916601300239563\n",
      "Actor Loss: -0.7767518758773804\n",
      "Critic Loss: 0.874870777130127\n",
      "Actor Loss: -2.2112936973571777\n",
      "Critic Loss: 0.8332028388977051\n",
      "Actor Loss: -1.0331472140023834e-06\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.8332314491271973\n",
      "Actor Loss: 2.3645272254943848\n",
      "Critic Loss: 0.9411178231239319\n",
      "Actor Loss: -1.1984859704971313\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: 3.814697606685513e-07\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.9166449904441833\n",
      "Actor Loss: 1.3586251735687256\n",
      "Critic Loss: 0.916601300239563\n",
      "Actor Loss: -0.7767521739006042\n",
      "Critic Loss: 0.9411178231239319\n",
      "Actor Loss: -1.1984859704971313\n",
      "Epoch 84: Average Steps per Episode: 11.93, Average Reward: 1.50, Average Loss: 0.2282\n",
      "Critic Loss: 0.9582866430282593\n",
      "Actor Loss: 0.7902345657348633\n",
      "Critic Loss: 0.916601300239563\n",
      "Actor Loss: -0.7767521739006042\n",
      "Critic Loss: 0.9411178231239319\n",
      "Actor Loss: -1.1984859704971313\n",
      "Critic Loss: 0.8332642912864685\n",
      "Actor Loss: 1.9868215872520523e-07\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: 3.814697606685513e-07\n",
      "Critic Loss: 0.8332314491271973\n",
      "Actor Loss: 2.3645272254943848\n",
      "Critic Loss: 0.8749896287918091\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.916601300239563\n",
      "Actor Loss: -0.7767521739006042\n",
      "Critic Loss: 0.8749896287918091\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: 3.814697606685513e-07\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Epoch 85: Average Steps per Episode: 11.97, Average Reward: 2.00, Average Loss: 0.5621\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: 3.814697606685513e-07\n",
      "Critic Loss: 0.8749896287918091\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9582866430282593\n",
      "Actor Loss: 0.7902345657348633\n",
      "Critic Loss: 0.916601300239563\n",
      "Actor Loss: -0.7767521739006042\n",
      "Critic Loss: 0.8332642912864685\n",
      "Actor Loss: 1.9868215872520523e-07\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: 3.814697606685513e-07\n",
      "Critic Loss: 0.9582866430282593\n",
      "Actor Loss: 0.7902345657348633\n",
      "Critic Loss: 0.9523157477378845\n",
      "Actor Loss: -0.6943508386611938\n",
      "Critic Loss: 0.9089846611022949\n",
      "Actor Loss: 1.500400185585022\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: 3.814697606685513e-07\n",
      "Critic Loss: 0.916601300239563\n",
      "Actor Loss: -0.7767521739006042\n",
      "Critic Loss: 0.9166449308395386\n",
      "Actor Loss: 1.3586251735687256\n",
      "Epoch 86: Average Steps per Episode: 11.98, Average Reward: 0.00, Average Loss: 1.1023\n",
      "Critic Loss: 0.8749896287918091\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.9332109689712524\n",
      "Actor Loss: -1.1801631450653076\n",
      "Critic Loss: 0.888871431350708\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.9166449308395386\n",
      "Actor Loss: 1.3586251735687256\n",
      "Critic Loss: 0.9582866430282593\n",
      "Actor Loss: 0.7902345657348633\n",
      "Critic Loss: 0.8332642912864685\n",
      "Actor Loss: 1.9868215872520523e-07\n",
      "Critic Loss: 0.9582866430282593\n",
      "Actor Loss: 0.7902345657348633\n",
      "Critic Loss: 0.9332109689712524\n",
      "Actor Loss: -1.1801631450653076\n",
      "Critic Loss: 0.888871431350708\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Epoch 87: Average Steps per Episode: 12.03, Average Reward: 5.00, Average Loss: 0.6156\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.9166449308395386\n",
      "Actor Loss: 1.3586251735687256\n",
      "Critic Loss: 0.9332109689712524\n",
      "Actor Loss: -1.1801631450653076\n",
      "Critic Loss: 0.8749896287918091\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9374361634254456\n",
      "Actor Loss: -1.0856341123580933\n",
      "Critic Loss: 0.9332109689712524\n",
      "Actor Loss: -1.1801631450653076\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.8749896287918091\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9089846611022949\n",
      "Actor Loss: 1.500400185585022\n",
      "Critic Loss: 0.888871431350708\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9332109689712524\n",
      "Actor Loss: -1.1801631450653076\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Epoch 88: Average Steps per Episode: 12.02, Average Reward: 2.00, Average Loss: 0.4256\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Critic Loss: 0.9374361634254456\n",
      "Actor Loss: -1.0856341123580933\n",
      "Critic Loss: 0.9166449308395386\n",
      "Actor Loss: 1.3586251735687256\n",
      "Critic Loss: 0.8748708367347717\n",
      "Actor Loss: -2.2112934589385986\n",
      "Critic Loss: 0.9333085417747498\n",
      "Actor Loss: 1.0303198099136353\n",
      "Critic Loss: 0.9523157477378845\n",
      "Actor Loss: -0.6943508386611938\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Critic Loss: 0.888871431350708\n",
      "Actor Loss: -2.119276274470394e-07\n",
      "Critic Loss: 0.9166449308395386\n",
      "Actor Loss: 1.3586251735687256\n",
      "Critic Loss: 0.9523157477378845\n",
      "Actor Loss: -0.6943508386611938\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Epoch 89: Average Steps per Episode: 12.04, Average Reward: 3.50, Average Loss: 0.9448\n",
      "Critic Loss: 0.9165624380111694\n",
      "Actor Loss: -1.1782082319259644\n",
      "Critic Loss: 0.9165884256362915\n",
      "Actor Loss: 0.05216614529490471\n",
      "Critic Loss: 0.9523157477378845\n",
      "Actor Loss: -0.6943508386611938\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9089846611022949\n",
      "Actor Loss: 1.500400185585022\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Critic Loss: 0.9165624380111694\n",
      "Actor Loss: -1.1782082319259644\n",
      "Critic Loss: 0.9165624380111694\n",
      "Actor Loss: -1.1782082319259644\n",
      "Critic Loss: 0.9165884256362915\n",
      "Actor Loss: 0.05216614529490471\n",
      "Critic Loss: 0.9089846611022949\n",
      "Actor Loss: 1.500400185585022\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Epoch 90: Average Steps per Episode: 12.04, Average Reward: 0.00, Average Loss: 0.8879\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Critic Loss: 0.9374361634254456\n",
      "Actor Loss: -1.0856341123580933\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -1.7862237691879272\n",
      "Critic Loss: 0.9332109689712524\n",
      "Actor Loss: -1.1801631450653076\n",
      "Critic Loss: 0.9374361634254456\n",
      "Actor Loss: -1.0856341123580933\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9165624380111694\n",
      "Actor Loss: -1.1782082319259644\n",
      "Critic Loss: 0.9374361634254456\n",
      "Actor Loss: -1.0856341123580933\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Epoch 91: Average Steps per Episode: 12.02, Average Reward: 0.00, Average Loss: 0.3232\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Critic Loss: 0.9499449133872986\n",
      "Actor Loss: -1.3739519119262695\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9165884256362915\n",
      "Actor Loss: 0.05216614529490471\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9374914169311523\n",
      "Actor Loss: 0.3585158586502075\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9333085417747498\n",
      "Actor Loss: 1.0303198099136353\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Epoch 92: Average Steps per Episode: 12.04, Average Reward: 0.00, Average Loss: 0.8196\n",
      "Critic Loss: 0.9165884256362915\n",
      "Actor Loss: 0.05216614529490471\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.8749451041221619\n",
      "Actor Loss: 2.5786380767822266\n",
      "Critic Loss: 0.9230273365974426\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.9166440963745117\n",
      "Actor Loss: 0.42835792899131775\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9333085417747498\n",
      "Actor Loss: 1.0303198099136353\n",
      "Critic Loss: 0.9165884256362915\n",
      "Actor Loss: 0.05216614529490471\n",
      "Epoch 93: Average Steps per Episode: 12.03, Average Reward: 0.00, Average Loss: 1.0446\n",
      "Critic Loss: 0.9166461825370789\n",
      "Actor Loss: -4.76837158203125e-07\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.9230273365974426\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.9499449133872986\n",
      "Actor Loss: -1.3739519119262695\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Epoch 94: Average Steps per Episode: 12.10, Average Reward: 1.50, Average Loss: 0.4553\n",
      "Critic Loss: 0.9230273365974426\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9165624380111694\n",
      "Actor Loss: -1.1782082319259644\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9333106279373169\n",
      "Actor Loss: -0.4517424404621124\n",
      "Critic Loss: 0.9499449133872986\n",
      "Actor Loss: -1.3739519119262695\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.9165624380111694\n",
      "Actor Loss: -1.1782082319259644\n",
      "Critic Loss: 0.9333106279373169\n",
      "Actor Loss: -0.4517424404621124\n",
      "Critic Loss: 0.8747920989990234\n",
      "Actor Loss: -0.0\n",
      "Epoch 95: Average Steps per Episode: 12.09, Average Reward: 1.50, Average Loss: 0.3431\n",
      "Critic Loss: 0.8747920989990234\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Critic Loss: 0.9333106279373169\n",
      "Actor Loss: -0.4517424404621124\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9523003101348877\n",
      "Actor Loss: -1.3439276218414307\n",
      "Critic Loss: 0.8747920989990234\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Critic Loss: 0.9166011810302734\n",
      "Actor Loss: -0.804701030254364\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Critic Loss: 0.9230273365974426\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.8749126195907593\n",
      "Actor Loss: -0.44739866256713867\n",
      "Epoch 96: Average Steps per Episode: 12.19, Average Reward: 2.50, Average Loss: 0.4342\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Critic Loss: 0.9523261785507202\n",
      "Actor Loss: -1.4979400634765625\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.8749451041221619\n",
      "Actor Loss: 2.5786380767822266\n",
      "Critic Loss: 0.9333106279373169\n",
      "Actor Loss: -0.4517424404621124\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.9523261785507202\n",
      "Actor Loss: -1.4979400634765625\n",
      "Critic Loss: 0.8747920989990234\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Epoch 97: Average Steps per Episode: 12.21, Average Reward: 0.00, Average Loss: 0.6913\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Critic Loss: 0.9230273365974426\n",
      "Actor Loss: 7.335956411225197e-07\n",
      "Critic Loss: 0.9523003101348877\n",
      "Actor Loss: -1.3439276218414307\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.8999894261360168\n",
      "Actor Loss: 0.6450775265693665\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Critic Loss: 0.9333106279373169\n",
      "Actor Loss: -0.4517424404621124\n",
      "Critic Loss: 0.8999298214912415\n",
      "Actor Loss: -0.91848224401474\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Epoch 98: Average Steps per Episode: 12.26, Average Reward: 0.00, Average Loss: 0.8548\n",
      "Critic Loss: 0.8570247888565063\n",
      "Actor Loss: 3.913853883743286\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Critic Loss: 0.9166082143783569\n",
      "Actor Loss: -0.09905274957418442\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.9523261785507202\n",
      "Actor Loss: -1.4979400634765625\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.9523261785507202\n",
      "Actor Loss: -1.4979400634765625\n",
      "Critic Loss: 0.9333106279373169\n",
      "Actor Loss: -0.4517424404621124\n",
      "Critic Loss: 0.9333065748214722\n",
      "Actor Loss: 0.29097291827201843\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Epoch 99: Average Steps per Episode: 12.28, Average Reward: 2.00, Average Loss: 1.1745\n",
      "Critic Loss: 0.8999894261360168\n",
      "Actor Loss: 0.6450775265693665\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.8570247888565063\n",
      "Actor Loss: 3.913853883743286\n",
      "Critic Loss: 0.8999894261360168\n",
      "Actor Loss: 0.6450775265693665\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.8570247888565063\n",
      "Actor Loss: 3.913853883743286\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Epoch 100: Average Steps per Episode: 12.31, Average Reward: 2.50, Average Loss: 1.5544\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.8999894261360168\n",
      "Actor Loss: 0.6450775265693665\n",
      "Critic Loss: 0.9696120619773865\n",
      "Actor Loss: -0.009989420883357525\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.8999894261360168\n",
      "Actor Loss: 0.6450775265693665\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Critic Loss: 0.9523261785507202\n",
      "Actor Loss: -1.4979400634765625\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Epoch 101: Average Steps per Episode: 12.28, Average Reward: 0.00, Average Loss: 0.4095\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Critic Loss: 0.9374769926071167\n",
      "Actor Loss: -0.49573850631713867\n",
      "Critic Loss: 0.9523261785507202\n",
      "Actor Loss: -1.4979400634765625\n",
      "Critic Loss: 0.9166082143783569\n",
      "Actor Loss: -0.09905274957418442\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9166082143783569\n",
      "Actor Loss: -0.09905274957418442\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Epoch 102: Average Steps per Episode: 12.25, Average Reward: 0.00, Average Loss: -0.6390\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Critic Loss: 0.9285032153129578\n",
      "Actor Loss: 1.5240181684494019\n",
      "Critic Loss: 0.9374769926071167\n",
      "Actor Loss: -0.49573850631713867\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.9523109197616577\n",
      "Actor Loss: -1.375142216682434\n",
      "Epoch 103: Average Steps per Episode: 12.18, Average Reward: 0.00, Average Loss: 0.6932\n",
      "Critic Loss: 0.9374769926071167\n",
      "Actor Loss: -0.49573850631713867\n",
      "Critic Loss: 0.874984860420227\n",
      "Actor Loss: -1.0201568603515625\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9332579970359802\n",
      "Actor Loss: -1.3543444871902466\n",
      "Critic Loss: 0.9374769926071167\n",
      "Actor Loss: -0.49573850631713867\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.9332579970359802\n",
      "Actor Loss: -1.3543444871902466\n",
      "Critic Loss: 0.8570419549942017\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.85714191198349\n",
      "Actor Loss: 1.4623245000839233\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9166082143783569\n",
      "Actor Loss: -0.09905274957418442\n",
      "Epoch 104: Average Steps per Episode: 12.16, Average Reward: 22.00, Average Loss: 1.0101\n",
      "Critic Loss: 0.8570419549942017\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Critic Loss: 0.9564611315727234\n",
      "Actor Loss: 0.1875326782464981\n",
      "Critic Loss: 0.85714191198349\n",
      "Actor Loss: 1.4623245000839233\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Critic Loss: 0.9332579970359802\n",
      "Actor Loss: -1.3543444871902466\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.8570419549942017\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.9332579970359802\n",
      "Actor Loss: -1.3543444871902466\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Epoch 105: Average Steps per Episode: 12.10, Average Reward: 0.00, Average Loss: -0.1104\n",
      "Critic Loss: 0.8999813199043274\n",
      "Actor Loss: -0.14477424323558807\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.6664996147155762\n",
      "Actor Loss: -0.0\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.8570419549942017\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.85714191198349\n",
      "Actor Loss: 1.4623245000839233\n",
      "Critic Loss: 0.9332579970359802\n",
      "Actor Loss: -1.3543444871902466\n",
      "Critic Loss: 0.9332576990127563\n",
      "Actor Loss: -1.7644329071044922\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Epoch 106: Average Steps per Episode: 12.10, Average Reward: 1.50, Average Loss: -1.1174\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.8999813199043274\n",
      "Actor Loss: -0.14477424323558807\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Critic Loss: 0.6665620803833008\n",
      "Actor Loss: -6.663676738739014\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Epoch 107: Average Steps per Episode: 12.10, Average Reward: 1.50, Average Loss: 0.2325\n",
      "Critic Loss: 0.85714191198349\n",
      "Actor Loss: 1.4623245000839233\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Critic Loss: 0.8570419549942017\n",
      "Actor Loss: 2.1798271063744323e-06\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Critic Loss: 0.9522990584373474\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.8749818205833435\n",
      "Actor Loss: 0.720806360244751\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Critic Loss: 0.7998062968254089\n",
      "Actor Loss: 4.954711437225342\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Epoch 108: Average Steps per Episode: 12.19, Average Reward: 5.00, Average Loss: 0.7845\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.9522990584373474\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Critic Loss: 0.8749818205833435\n",
      "Actor Loss: 0.720806360244751\n",
      "Critic Loss: 0.9545356035232544\n",
      "Actor Loss: 0.35567301511764526\n",
      "Critic Loss: 0.8749818205833435\n",
      "Actor Loss: 0.720806360244751\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Critic Loss: 0.85714191198349\n",
      "Actor Loss: 1.4623245000839233\n",
      "Critic Loss: 0.7498800754547119\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.916580855846405\n",
      "Actor Loss: -1.3512811660766602\n",
      "Epoch 109: Average Steps per Episode: 12.12, Average Reward: 0.00, Average Loss: 0.6751\n",
      "Critic Loss: 0.8332481384277344\n",
      "Actor Loss: -3.496807098388672\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.9545356035232544\n",
      "Actor Loss: 0.35567301511764526\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.8999813199043274\n",
      "Actor Loss: -0.14477424323558807\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.7998815178871155\n",
      "Actor Loss: 0.6496269106864929\n",
      "Critic Loss: 0.7498800754547119\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Epoch 110: Average Steps per Episode: 12.12, Average Reward: 0.50, Average Loss: -0.2980\n",
      "Critic Loss: 0.7498800754547119\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.8888170123100281\n",
      "Actor Loss: -3.162891387939453\n",
      "Critic Loss: 0.9522990584373474\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Critic Loss: 0.9545356035232544\n",
      "Actor Loss: 0.35567301511764526\n",
      "Critic Loss: 0.9332515001296997\n",
      "Actor Loss: -1.7447841167449951\n",
      "Critic Loss: 0.9090730547904968\n",
      "Actor Loss: 0.538161039352417\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Epoch 111: Average Steps per Episode: 12.12, Average Reward: 1.50, Average Loss: 0.2780\n",
      "Critic Loss: 0.9545356035232544\n",
      "Actor Loss: 0.35567301511764526\n",
      "Critic Loss: 0.9411014914512634\n",
      "Actor Loss: 1.3099035024642944\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.9522990584373474\n",
      "Actor Loss: 2.543131586207892e-06\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Critic Loss: 0.8749818205833435\n",
      "Actor Loss: 0.720806360244751\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Epoch 112: Average Steps per Episode: 12.14, Average Reward: 0.50, Average Loss: -0.5777\n",
      "Critic Loss: 0.9545356035232544\n",
      "Actor Loss: 0.35567301511764526\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.9599716663360596\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9090730547904968\n",
      "Actor Loss: 0.538161039352417\n",
      "Critic Loss: 0.8888170123100281\n",
      "Actor Loss: -3.162891387939453\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.7498800754547119\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.8888170123100281\n",
      "Actor Loss: -3.162891387939453\n",
      "Epoch 113: Average Steps per Episode: 12.21, Average Reward: 2.00, Average Loss: -0.9521\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.8888170123100281\n",
      "Actor Loss: -3.162891387939453\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.7498800754547119\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.9090566039085388\n",
      "Actor Loss: -1.5692720413208008\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Critic Loss: 0.7498800754547119\n",
      "Actor Loss: 5.960464477539062e-07\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.108155250549316\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.9090566039085388\n",
      "Actor Loss: -1.5692720413208008\n",
      "Critic Loss: 0.9332814812660217\n",
      "Actor Loss: 0.8201599717140198\n",
      "Epoch 114: Average Steps per Episode: 12.16, Average Reward: 1.50, Average Loss: -0.5746\n",
      "Critic Loss: 0.8999791145324707\n",
      "Actor Loss: 0.6243885159492493\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.9090730547904968\n",
      "Actor Loss: 0.538161039352417\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Critic Loss: 0.916634738445282\n",
      "Actor Loss: -0.7926160097122192\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.9285182952880859\n",
      "Actor Loss: -1.5047861337661743\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Epoch 115: Average Steps per Episode: 12.15, Average Reward: 2.50, Average Loss: 0.1255\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.6638094186782837\n",
      "Actor Loss: -7.9384894371032715\n",
      "Critic Loss: 0.9090566039085388\n",
      "Actor Loss: -1.5692720413208008\n",
      "Critic Loss: 0.9090730547904968\n",
      "Actor Loss: 0.538161039352417\n",
      "Critic Loss: 0.9332814812660217\n",
      "Actor Loss: 0.8201599717140198\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.928501307964325\n",
      "Actor Loss: -1.7770249843597412\n",
      "Critic Loss: 0.9705364108085632\n",
      "Actor Loss: -1.1714266538619995\n",
      "Critic Loss: 0.8999791145324707\n",
      "Actor Loss: 0.6243885159492493\n",
      "Critic Loss: 0.9090730547904968\n",
      "Actor Loss: 0.538161039352417\n",
      "Critic Loss: 0.9090566039085388\n",
      "Actor Loss: -1.5692720413208008\n",
      "Critic Loss: 0.916634738445282\n",
      "Actor Loss: -0.7926160097122192\n",
      "Epoch 116: Average Steps per Episode: 12.21, Average Reward: 0.00, Average Loss: -0.5826\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.9332814812660217\n",
      "Actor Loss: 0.8201599717140198\n",
      "Critic Loss: 0.6638094186782837\n",
      "Actor Loss: -7.9384894371032715\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.749863862991333\n",
      "Actor Loss: -6.595875263214111\n",
      "Critic Loss: 0.9599716663360596\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9230128526687622\n",
      "Actor Loss: -7.335956553333745e-08\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.9410930275917053\n",
      "Actor Loss: -2.762382984161377\n",
      "Critic Loss: 0.9599716663360596\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.9332814812660217\n",
      "Actor Loss: 0.8201599717140198\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Epoch 117: Average Steps per Episode: 12.18, Average Reward: 0.00, Average Loss: -0.4365\n",
      "Critic Loss: 0.9332814812660217\n",
      "Actor Loss: 0.8201599717140198\n",
      "Critic Loss: 0.9705364108085632\n",
      "Actor Loss: -1.1714266538619995\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.6638094186782837\n",
      "Actor Loss: -7.9384894371032715\n",
      "Critic Loss: 0.9599716663360596\n",
      "Actor Loss: 1.5258788721439487e-07\n",
      "Critic Loss: 0.916609525680542\n",
      "Actor Loss: -2.140868663787842\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.916634738445282\n",
      "Actor Loss: -0.7926160097122192\n",
      "Critic Loss: 0.9705364108085632\n",
      "Actor Loss: -1.1714266538619995\n",
      "Critic Loss: 0.9230128526687622\n",
      "Actor Loss: -7.335956553333745e-08\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.916634738445282\n",
      "Actor Loss: -0.7926160097122192\n",
      "Epoch 118: Average Steps per Episode: 12.15, Average Reward: 0.00, Average Loss: -0.4124\n",
      "Critic Loss: 0.9705364108085632\n",
      "Actor Loss: -1.1714266538619995\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.749863862991333\n",
      "Actor Loss: -6.595875263214111\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.8331426382064819\n",
      "Actor Loss: -1.2160844802856445\n",
      "Critic Loss: 0.916634738445282\n",
      "Actor Loss: -0.7926160097122192\n",
      "Critic Loss: 0.8749814033508301\n",
      "Actor Loss: 1.1849966049194336\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.9499189257621765\n",
      "Actor Loss: 1.021126627922058\n",
      "Critic Loss: 0.9230128526687622\n",
      "Actor Loss: -7.335956553333745e-08\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.749863862991333\n",
      "Actor Loss: -6.595875263214111\n",
      "Epoch 119: Average Steps per Episode: 12.17, Average Reward: 0.00, Average Loss: -0.4234\n",
      "Critic Loss: 0.8332375884056091\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.916634738445282\n",
      "Actor Loss: -0.7926160097122192\n",
      "Critic Loss: 0.6638094186782837\n",
      "Actor Loss: -7.9384894371032715\n",
      "Critic Loss: 0.9410697817802429\n",
      "Actor Loss: 0.7198127508163452\n",
      "Critic Loss: 0.6638094186782837\n",
      "Actor Loss: -7.9384894371032715\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.916609525680542\n",
      "Actor Loss: -2.140868663787842\n",
      "Critic Loss: 0.8999791145324707\n",
      "Actor Loss: 0.6243885159492493\n",
      "Critic Loss: 0.916609525680542\n",
      "Actor Loss: -2.140868663787842\n",
      "Critic Loss: 0.8998773694038391\n",
      "Actor Loss: 0.45393791794776917\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.9705364108085632\n",
      "Actor Loss: -1.1714266538619995\n",
      "Epoch 120: Average Steps per Episode: 12.17, Average Reward: 0.00, Average Loss: -0.8363\n",
      "Critic Loss: 0.8998773694038391\n",
      "Actor Loss: 0.45393791794776917\n",
      "Critic Loss: 0.9410697817802429\n",
      "Actor Loss: 0.7198127508163452\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.916609525680542\n",
      "Actor Loss: -2.140868663787842\n",
      "Critic Loss: 0.6638094186782837\n",
      "Actor Loss: -7.9384894371032715\n",
      "Critic Loss: 0.9499189257621765\n",
      "Actor Loss: 1.021126627922058\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.8332375884056091\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.9230128526687622\n",
      "Actor Loss: -7.335956553333745e-08\n",
      "Critic Loss: 0.749863862991333\n",
      "Actor Loss: -6.595875263214111\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Epoch 121: Average Steps per Episode: 12.14, Average Reward: 0.00, Average Loss: -0.9569\n",
      "Critic Loss: 0.8998773694038391\n",
      "Actor Loss: 0.45393791794776917\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Critic Loss: 0.9499189257621765\n",
      "Actor Loss: 1.021126627922058\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.9499189257621765\n",
      "Actor Loss: 1.021126627922058\n",
      "Critic Loss: 0.9410697817802429\n",
      "Actor Loss: 0.7198127508163452\n",
      "Critic Loss: 0.9230128526687622\n",
      "Actor Loss: -7.335956553333745e-08\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Critic Loss: 0.9410697817802429\n",
      "Actor Loss: 0.7198127508163452\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Epoch 122: Average Steps per Episode: 12.11, Average Reward: 0.50, Average Loss: 0.8331\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.8998773694038391\n",
      "Actor Loss: 0.45393791794776917\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.8332403898239136\n",
      "Actor Loss: 1.1920928955078125e-06\n",
      "Critic Loss: 0.9230166673660278\n",
      "Actor Loss: -1.0554075241088867\n",
      "Critic Loss: 0.8998773694038391\n",
      "Actor Loss: 0.45393791794776917\n",
      "Critic Loss: 0.9499189257621765\n",
      "Actor Loss: 1.021126627922058\n",
      "Critic Loss: 0.9410697817802429\n",
      "Actor Loss: 0.7198127508163452\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Epoch 123: Average Steps per Episode: 12.08, Average Reward: 0.00, Average Loss: 0.3482\n",
      "Critic Loss: 0.8332375884056091\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.8749803304672241\n",
      "Actor Loss: 0.5628583431243896\n",
      "Critic Loss: 0.8888280987739563\n",
      "Actor Loss: -0.9813902378082275\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.8332375884056091\n",
      "Actor Loss: -3.178914482759865e-07\n",
      "Critic Loss: 0.9230166673660278\n",
      "Actor Loss: -1.0554075241088867\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Epoch 124: Average Steps per Episode: 12.05, Average Reward: 2.00, Average Loss: 0.1555\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.8332704305648804\n",
      "Actor Loss: -2.5838751792907715\n",
      "Critic Loss: 0.8749803304672241\n",
      "Actor Loss: 0.5628583431243896\n",
      "Critic Loss: 0.8888280987739563\n",
      "Actor Loss: -0.9813902378082275\n",
      "Critic Loss: 0.9332672357559204\n",
      "Actor Loss: -1.235254168510437\n",
      "Critic Loss: 0.8332704305648804\n",
      "Actor Loss: -2.5838751792907715\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.9230166673660278\n",
      "Actor Loss: -1.0554075241088867\n",
      "Critic Loss: 0.9166108965873718\n",
      "Actor Loss: -2.487497329711914\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.8332704305648804\n",
      "Actor Loss: -2.5838751792907715\n",
      "Epoch 125: Average Steps per Episode: 12.04, Average Reward: 0.50, Average Loss: -0.8201\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.9374526739120483\n",
      "Actor Loss: -1.8018580675125122\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.9374526739120483\n",
      "Actor Loss: -1.8018580675125122\n",
      "Critic Loss: 0.8569296598434448\n",
      "Actor Loss: -1.4874627590179443\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.8749803304672241\n",
      "Actor Loss: 0.5628583431243896\n",
      "Critic Loss: 0.9230166673660278\n",
      "Actor Loss: -1.0554075241088867\n",
      "Epoch 126: Average Steps per Episode: 12.08, Average Reward: 0.00, Average Loss: -0.1830\n",
      "Critic Loss: 0.9230166673660278\n",
      "Actor Loss: -1.0554075241088867\n",
      "Critic Loss: 0.9332672357559204\n",
      "Actor Loss: -1.235254168510437\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.7998650074005127\n",
      "Actor Loss: -1.62124638336536e-06\n",
      "Critic Loss: 0.749875545501709\n",
      "Actor Loss: 0.007445603609085083\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.9090210795402527\n",
      "Actor Loss: -2.1575450897216797\n",
      "Critic Loss: 0.8749803304672241\n",
      "Actor Loss: 0.5628583431243896\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.9374526739120483\n",
      "Actor Loss: -1.8018580675125122\n",
      "Critic Loss: 0.9332672357559204\n",
      "Actor Loss: -1.235254168510437\n",
      "Epoch 127: Average Steps per Episode: 12.04, Average Reward: 0.00, Average Loss: -0.0805\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.8749803304672241\n",
      "Actor Loss: 0.5628583431243896\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -2.8776557445526123\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.7994189262390137\n",
      "Actor Loss: -0.33836936950683594\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -2.8776557445526123\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.8888280987739563\n",
      "Actor Loss: -0.9813902378082275\n",
      "Critic Loss: 0.8332704305648804\n",
      "Actor Loss: -2.5838751792907715\n",
      "Critic Loss: 0.9444201588630676\n",
      "Actor Loss: 0.7919771671295166\n",
      "Epoch 128: Average Steps per Episode: 12.05, Average Reward: 2.00, Average Loss: 0.6252\n",
      "Critic Loss: 0.9374526739120483\n",
      "Actor Loss: -1.8018580675125122\n",
      "Critic Loss: 0.8749803304672241\n",
      "Actor Loss: 0.5628583431243896\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -2.8776557445526123\n",
      "Critic Loss: 0.9332672357559204\n",
      "Actor Loss: -1.235254168510437\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.9332672357559204\n",
      "Actor Loss: -1.235254168510437\n",
      "Critic Loss: 0.8332704305648804\n",
      "Actor Loss: -2.5838751792907715\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Epoch 129: Average Steps per Episode: 12.07, Average Reward: 0.50, Average Loss: 0.0747\n",
      "Critic Loss: 0.9444201588630676\n",
      "Actor Loss: 0.7919771671295166\n",
      "Critic Loss: 0.8332704305648804\n",
      "Actor Loss: -2.5838751792907715\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.9332672357559204\n",
      "Actor Loss: -1.235254168510437\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -2.8776557445526123\n",
      "Critic Loss: 0.9410873651504517\n",
      "Actor Loss: -2.0226175785064697\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Epoch 130: Average Steps per Episode: 12.05, Average Reward: 1.50, Average Loss: 0.1059\n",
      "Critic Loss: 0.8570119738578796\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -2.8776557445526123\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.8999781608581543\n",
      "Actor Loss: 0.16566582024097443\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.8570119738578796\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9374526739120483\n",
      "Actor Loss: -1.8018580675125122\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Epoch 131: Average Steps per Episode: 12.01, Average Reward: 0.00, Average Loss: 1.2140\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.7497962713241577\n",
      "Actor Loss: -9.5367431640625e-07\n",
      "Critic Loss: 0.9090425372123718\n",
      "Actor Loss: -2.8776557445526123\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.88881915807724\n",
      "Actor Loss: 2.533782720565796\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Critic Loss: 0.7497962713241577\n",
      "Actor Loss: -9.5367431640625e-07\n",
      "Critic Loss: 0.7998650074005127\n",
      "Actor Loss: -1.62124638336536e-06\n",
      "Critic Loss: 0.9444201588630676\n",
      "Actor Loss: 0.7919771671295166\n",
      "Epoch 132: Average Steps per Episode: 11.99, Average Reward: 0.00, Average Loss: 0.8163\n",
      "Critic Loss: 0.8999781608581543\n",
      "Actor Loss: 0.16566582024097443\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.8999781608581543\n",
      "Actor Loss: 0.16566582024097443\n",
      "Critic Loss: 0.9444201588630676\n",
      "Actor Loss: 0.7919771671295166\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.9444201588630676\n",
      "Actor Loss: 0.7919771671295166\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Epoch 133: Average Steps per Episode: 12.02, Average Reward: 8.00, Average Loss: 0.8997\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.7497962713241577\n",
      "Actor Loss: -9.5367431640625e-07\n",
      "Critic Loss: 0.8999781608581543\n",
      "Actor Loss: 0.16566582024097443\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.923052966594696\n",
      "Actor Loss: 0.07408846914768219\n",
      "Critic Loss: 0.8570119738578796\n",
      "Actor Loss: 5.449567765936081e-07\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Critic Loss: 0.8999364972114563\n",
      "Actor Loss: -3.159740686416626\n",
      "Critic Loss: 0.899932324886322\n",
      "Actor Loss: -1.4457108974456787\n",
      "Critic Loss: 0.7497962713241577\n",
      "Actor Loss: -9.5367431640625e-07\n",
      "Critic Loss: 0.9523240923881531\n",
      "Actor Loss: -0.9700915813446045\n",
      "Epoch 134: Average Steps per Episode: 12.09, Average Reward: 0.00, Average Loss: 0.4190\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.923052966594696\n",
      "Actor Loss: 0.07408846914768219\n",
      "Critic Loss: 0.9473168253898621\n",
      "Actor Loss: -1.019601583480835\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: 0.35408681631088257\n",
      "Critic Loss: 0.8999781608581543\n",
      "Actor Loss: 0.16566582024097443\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.923052966594696\n",
      "Actor Loss: 0.07408846914768219\n",
      "Critic Loss: 0.8999781608581543\n",
      "Actor Loss: 0.16566582024097443\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Epoch 135: Average Steps per Episode: 12.12, Average Reward: 1.00, Average Loss: 1.0004\n",
      "Critic Loss: 0.8888084888458252\n",
      "Actor Loss: 1.2276560068130493\n",
      "Critic Loss: 0.8999364972114563\n",
      "Actor Loss: -3.159740686416626\n",
      "Critic Loss: 0.7497962713241577\n",
      "Actor Loss: -9.5367431640625e-07\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.941119372844696\n",
      "Actor Loss: -1.9935877323150635\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.6665360331535339\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.8332330584526062\n",
      "Actor Loss: 3.5286202430725098\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.923052966594696\n",
      "Actor Loss: 0.07408846914768219\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: 0.35408681631088257\n",
      "Epoch 136: Average Steps per Episode: 12.11, Average Reward: 0.00, Average Loss: 0.8905\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.933216392993927\n",
      "Actor Loss: 0.4866529107093811\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: 0.35408681631088257\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.923052966594696\n",
      "Actor Loss: 0.07408846914768219\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.8999364972114563\n",
      "Actor Loss: -3.159740686416626\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.8332330584526062\n",
      "Actor Loss: 3.5286202430725098\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: 0.35408681631088257\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Epoch 137: Average Steps per Episode: 12.12, Average Reward: 0.00, Average Loss: 0.2182\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.8332330584526062\n",
      "Actor Loss: 3.5286202430725098\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.941119372844696\n",
      "Actor Loss: -1.9935877323150635\n",
      "Critic Loss: 0.8332330584526062\n",
      "Actor Loss: 3.5286202430725098\n",
      "Critic Loss: 0.8999364972114563\n",
      "Actor Loss: -3.159740686416626\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Epoch 138: Average Steps per Episode: 12.13, Average Reward: 0.00, Average Loss: 0.0724\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.9284892082214355\n",
      "Actor Loss: 2.242387294769287\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.8332330584526062\n",
      "Actor Loss: 3.5286202430725098\n",
      "Critic Loss: 0.9473168253898621\n",
      "Actor Loss: -1.019601583480835\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.941119372844696\n",
      "Actor Loss: -1.9935877323150635\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9686720967292786\n",
      "Actor Loss: -1.4227142333984375\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.7999299764633179\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Epoch 139: Average Steps per Episode: 12.10, Average Reward: 0.00, Average Loss: 0.3580\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.9284892082214355\n",
      "Actor Loss: 2.242387294769287\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.941119372844696\n",
      "Actor Loss: -1.9935877323150635\n",
      "Critic Loss: 0.8332330584526062\n",
      "Actor Loss: 3.5286202430725098\n",
      "Critic Loss: 0.6665196418762207\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9285286068916321\n",
      "Actor Loss: -0.89700847864151\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.9284892082214355\n",
      "Actor Loss: 2.242387294769287\n",
      "Epoch 140: Average Steps per Episode: 12.07, Average Reward: 0.00, Average Loss: 0.5023\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.7999299764633179\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Critic Loss: 0.9229869246482849\n",
      "Actor Loss: -2.2936487197875977\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.9284892082214355\n",
      "Actor Loss: 2.242387294769287\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.9284892082214355\n",
      "Actor Loss: 2.242387294769287\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Epoch 141: Average Steps per Episode: 12.08, Average Reward: 0.00, Average Loss: 0.0653\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.9166075587272644\n",
      "Actor Loss: -0.7675735354423523\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.7999134063720703\n",
      "Actor Loss: -4.7408447265625\n",
      "Critic Loss: 0.9166075587272644\n",
      "Actor Loss: -0.7675735354423523\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.7999299764633179\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Epoch 142: Average Steps per Episode: 12.05, Average Reward: 2.00, Average Loss: -0.1232\n",
      "Critic Loss: 0.8888247013092041\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.9165996313095093\n",
      "Actor Loss: -2.315026044845581\n",
      "Critic Loss: 0.8570592403411865\n",
      "Actor Loss: -2.368870496749878\n",
      "Critic Loss: 0.6665196418762207\n",
      "Actor Loss: -6.35782896551973e-07\n",
      "Critic Loss: 0.9165996313095093\n",
      "Actor Loss: -2.315026044845581\n",
      "Critic Loss: 0.9582841396331787\n",
      "Actor Loss: -0.5433993339538574\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Epoch 143: Average Steps per Episode: 12.09, Average Reward: 4.50, Average Loss: -0.0841\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.8888247013092041\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9166075587272644\n",
      "Actor Loss: -0.7675735354423523\n",
      "Critic Loss: 0.899928867816925\n",
      "Actor Loss: -1.1418144702911377\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Critic Loss: 0.7999134063720703\n",
      "Actor Loss: -4.7408447265625\n",
      "Critic Loss: 0.9166075587272644\n",
      "Actor Loss: -0.7675735354423523\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.7999134063720703\n",
      "Actor Loss: -4.7408447265625\n",
      "Critic Loss: 0.7999299764633179\n",
      "Actor Loss: -7.629394644936838e-07\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Critic Loss: 0.9165996313095093\n",
      "Actor Loss: -2.315026044845581\n",
      "Epoch 144: Average Steps per Episode: 12.13, Average Reward: 8.50, Average Loss: -0.3211\n",
      "Critic Loss: 0.8888247013092041\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.7999134063720703\n",
      "Actor Loss: -4.7408447265625\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Critic Loss: 0.9166075587272644\n",
      "Actor Loss: -0.7675735354423523\n",
      "Critic Loss: 0.7998653054237366\n",
      "Actor Loss: -5.024833679199219\n",
      "Critic Loss: 0.7999134063720703\n",
      "Actor Loss: -4.7408447265625\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.9090408086776733\n",
      "Actor Loss: -2.4855527877807617\n",
      "Epoch 145: Average Steps per Episode: 12.09, Average Reward: 0.00, Average Loss: -0.9885\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Critic Loss: 0.9284762144088745\n",
      "Actor Loss: -0.5975624322891235\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.9583250880241394\n",
      "Actor Loss: 0.11482938379049301\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: -1.0253300666809082\n",
      "Critic Loss: 0.9583250880241394\n",
      "Actor Loss: 0.11482938379049301\n",
      "Critic Loss: 0.9165996313095093\n",
      "Actor Loss: -2.315026044845581\n",
      "Critic Loss: 0.7998653054237366\n",
      "Actor Loss: -5.024833679199219\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.9583250880241394\n",
      "Actor Loss: 0.11482938379049301\n",
      "Epoch 146: Average Steps per Episode: 12.12, Average Reward: 0.50, Average Loss: -0.3148\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.7999134063720703\n",
      "Actor Loss: -4.7408447265625\n",
      "Critic Loss: 0.9090408086776733\n",
      "Actor Loss: -2.4855527877807617\n",
      "Critic Loss: 0.9090667366981506\n",
      "Actor Loss: 0.19039519131183624\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9583250880241394\n",
      "Actor Loss: 0.11482938379049301\n",
      "Critic Loss: 0.7998653054237366\n",
      "Actor Loss: -5.024833679199219\n",
      "Critic Loss: 0.9285500049591064\n",
      "Actor Loss: 0.4442569315433502\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9599841833114624\n",
      "Actor Loss: 0.1657751500606537\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Epoch 147: Average Steps per Episode: 12.12, Average Reward: 1.50, Average Loss: -0.7794\n",
      "Critic Loss: 0.9499467015266418\n",
      "Actor Loss: -0.27121296525001526\n",
      "Critic Loss: 0.9285061359405518\n",
      "Actor Loss: -2.342787265777588\n",
      "Critic Loss: 0.9166456460952759\n",
      "Actor Loss: -0.48848819732666016\n",
      "Critic Loss: 0.9090408086776733\n",
      "Actor Loss: -2.4855527877807617\n",
      "Critic Loss: 0.9499467015266418\n",
      "Actor Loss: -0.27121296525001526\n",
      "Critic Loss: 0.7998653054237366\n",
      "Actor Loss: -5.024833679199219\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9285061359405518\n",
      "Actor Loss: -2.342787265777588\n",
      "Critic Loss: 0.7998653054237366\n",
      "Actor Loss: -5.024833679199219\n",
      "Critic Loss: 0.9285500049591064\n",
      "Actor Loss: 0.4442569315433502\n",
      "Critic Loss: 0.9090408086776733\n",
      "Actor Loss: -2.4855527877807617\n",
      "Critic Loss: 0.9285061359405518\n",
      "Actor Loss: -2.342787265777588\n",
      "Epoch 148: Average Steps per Episode: 12.12, Average Reward: 1.50, Average Loss: -1.1384\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: -1.0253300666809082\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Critic Loss: 0.7998653054237366\n",
      "Actor Loss: -5.024833679199219\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9166456460952759\n",
      "Actor Loss: -0.48848819732666016\n",
      "Critic Loss: 0.9285061359405518\n",
      "Actor Loss: -2.342787265777588\n",
      "Critic Loss: 0.9090408086776733\n",
      "Actor Loss: -2.4855527877807617\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: -1.0253300666809082\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Epoch 149: Average Steps per Episode: 12.13, Average Reward: 1.50, Average Loss: -0.6049\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9285500049591064\n",
      "Actor Loss: 0.4442569315433502\n",
      "Critic Loss: 0.9499467015266418\n",
      "Actor Loss: -0.27121296525001526\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.9285166263580322\n",
      "Actor Loss: -2.3094868659973145\n",
      "Critic Loss: 0.9499467015266418\n",
      "Actor Loss: -0.27121296525001526\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.8570834398269653\n",
      "Actor Loss: -3.3174521923065186\n",
      "Critic Loss: 0.9285166263580322\n",
      "Actor Loss: -2.3094868659973145\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: -1.0253300666809082\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Epoch 150: Average Steps per Episode: 12.14, Average Reward: 4.50, Average Loss: 1.0922\n",
      "Critic Loss: 0.9285500049591064\n",
      "Actor Loss: 0.4442569315433502\n",
      "Critic Loss: 0.9499467015266418\n",
      "Actor Loss: -0.27121296525001526\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: -1.0253300666809082\n",
      "Critic Loss: 0.8999199271202087\n",
      "Actor Loss: -1.8733932971954346\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.9166134595870972\n",
      "Actor Loss: -1.0253300666809082\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.8999199271202087\n",
      "Actor Loss: -1.8733932971954346\n",
      "Epoch 151: Average Steps per Episode: 12.13, Average Reward: 0.50, Average Loss: 1.1112\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.9285061359405518\n",
      "Actor Loss: -2.342787265777588\n",
      "Critic Loss: 0.9166456460952759\n",
      "Actor Loss: -0.48848819732666016\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9285634160041809\n",
      "Actor Loss: 0.2565825581550598\n",
      "Critic Loss: 0.9090179204940796\n",
      "Actor Loss: -1.8825327157974243\n",
      "Critic Loss: 0.9411209225654602\n",
      "Actor Loss: 0.34607428312301636\n",
      "Epoch 152: Average Steps per Episode: 12.13, Average Reward: 0.50, Average Loss: 0.8442\n",
      "Critic Loss: 0.9285166263580322\n",
      "Actor Loss: -2.3094868659973145\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.9333163499832153\n",
      "Actor Loss: 0.7715974450111389\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.9285634160041809\n",
      "Actor Loss: 0.2565825581550598\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.9411209225654602\n",
      "Actor Loss: 0.34607428312301636\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8999199271202087\n",
      "Actor Loss: -1.8733932971954346\n",
      "Critic Loss: 0.9411209225654602\n",
      "Actor Loss: 0.34607428312301636\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Epoch 153: Average Steps per Episode: 12.12, Average Reward: 3.00, Average Loss: 1.5515\n",
      "Critic Loss: 0.9333163499832153\n",
      "Actor Loss: 0.7715974450111389\n",
      "Critic Loss: 0.9285634160041809\n",
      "Actor Loss: 0.2565825581550598\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8569705486297607\n",
      "Actor Loss: 4.176507949829102\n",
      "Critic Loss: 0.9444172382354736\n",
      "Actor Loss: 0.6363218426704407\n",
      "Critic Loss: 0.9599518179893494\n",
      "Actor Loss: 0.1674545258283615\n",
      "Critic Loss: 0.9285634160041809\n",
      "Actor Loss: 0.2565825581550598\n",
      "Critic Loss: 0.9285166263580322\n",
      "Actor Loss: -2.3094868659973145\n",
      "Critic Loss: 0.9285166263580322\n",
      "Actor Loss: -2.3094868659973145\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.9499074816703796\n",
      "Actor Loss: 0.7876238226890564\n",
      "Epoch 154: Average Steps per Episode: 12.19, Average Reward: 0.50, Average Loss: 0.9221\n",
      "Critic Loss: 0.9599518179893494\n",
      "Actor Loss: 0.1674545258283615\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.8999199271202087\n",
      "Actor Loss: -1.8733932971954346\n",
      "Critic Loss: 0.7998995780944824\n",
      "Actor Loss: 4.731819152832031\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8999199271202087\n",
      "Actor Loss: -1.8733932971954346\n",
      "Critic Loss: 0.9411209225654602\n",
      "Actor Loss: 0.34607428312301636\n",
      "Critic Loss: 0.9333163499832153\n",
      "Actor Loss: 0.7715974450111389\n",
      "Critic Loss: 0.9285634160041809\n",
      "Actor Loss: 0.2565825581550598\n",
      "Critic Loss: 0.9499074816703796\n",
      "Actor Loss: 0.7876238226890564\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Epoch 155: Average Steps per Episode: 12.18, Average Reward: 0.00, Average Loss: 1.0769\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.9333163499832153\n",
      "Actor Loss: 0.7715974450111389\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.9166033267974854\n",
      "Actor Loss: -0.31694287061691284\n",
      "Critic Loss: 0.9499074816703796\n",
      "Actor Loss: 0.7876238226890564\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.7998837828636169\n",
      "Actor Loss: 2.1009202003479004\n",
      "Critic Loss: 0.9333163499832153\n",
      "Actor Loss: 0.7715974450111389\n",
      "Epoch 156: Average Steps per Episode: 12.14, Average Reward: 0.00, Average Loss: 1.0550\n",
      "Critic Loss: 0.9599518179893494\n",
      "Actor Loss: 0.1674545258283615\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8570528626441956\n",
      "Actor Loss: -0.8622682094573975\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.9499074816703796\n",
      "Actor Loss: 0.7876238226890564\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.9599518179893494\n",
      "Actor Loss: 0.1674545258283615\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.8570443391799927\n",
      "Actor Loss: -3.112338066101074\n",
      "Critic Loss: 0.8332504034042358\n",
      "Actor Loss: 3.950709104537964\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Epoch 157: Average Steps per Episode: 12.13, Average Reward: 0.00, Average Loss: -0.3442\n",
      "Critic Loss: 0.7998995780944824\n",
      "Actor Loss: 4.731819152832031\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.9599518179893494\n",
      "Actor Loss: 0.1674545258283615\n",
      "Critic Loss: 0.9285171627998352\n",
      "Actor Loss: -2.5232670307159424\n",
      "Critic Loss: 0.7998995780944824\n",
      "Actor Loss: 4.731819152832031\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.9499074816703796\n",
      "Actor Loss: 0.7876238226890564\n",
      "Critic Loss: 0.7998995780944824\n",
      "Actor Loss: 4.731819152832031\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.9499074816703796\n",
      "Actor Loss: 0.7876238226890564\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Epoch 158: Average Steps per Episode: 12.10, Average Reward: 0.00, Average Loss: 1.0733\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Critic Loss: 0.9285171627998352\n",
      "Actor Loss: -2.5232670307159424\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.7998995780944824\n",
      "Actor Loss: 4.731819152832031\n",
      "Critic Loss: 0.8570528626441956\n",
      "Actor Loss: -0.8622682094573975\n",
      "Critic Loss: 0.9599518179893494\n",
      "Actor Loss: 0.1674545258283615\n",
      "Epoch 159: Average Steps per Episode: 12.12, Average Reward: 1.50, Average Loss: 0.0332\n",
      "Critic Loss: 0.7998837828636169\n",
      "Actor Loss: 2.1009202003479004\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.8570528626441956\n",
      "Actor Loss: -0.8622682094573975\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Critic Loss: 0.9374394416809082\n",
      "Actor Loss: -1.7696034908294678\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.9285171627998352\n",
      "Actor Loss: -2.5232670307159424\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.7998837828636169\n",
      "Actor Loss: 2.1009202003479004\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Epoch 160: Average Steps per Episode: 12.11, Average Reward: 0.00, Average Loss: 0.1717\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Critic Loss: 0.8570528626441956\n",
      "Actor Loss: -0.8622682094573975\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Critic Loss: 0.9333092570304871\n",
      "Actor Loss: -0.2515925168991089\n",
      "Critic Loss: 0.9165222644805908\n",
      "Actor Loss: -1.753626823425293\n",
      "Critic Loss: 0.856945276260376\n",
      "Actor Loss: -3.09248948097229\n",
      "Critic Loss: 0.9333092570304871\n",
      "Actor Loss: -0.2515925168991089\n",
      "Critic Loss: 0.9165222644805908\n",
      "Actor Loss: -1.753626823425293\n",
      "Critic Loss: 0.9285171627998352\n",
      "Actor Loss: -2.5232670307159424\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Epoch 161: Average Steps per Episode: 12.11, Average Reward: 2.00, Average Loss: -0.3522\n",
      "Critic Loss: 0.8570528626441956\n",
      "Actor Loss: -0.8622682094573975\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Critic Loss: 0.9089950323104858\n",
      "Actor Loss: -2.6525657176971436\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.8570528626441956\n",
      "Actor Loss: -0.8622682094573975\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.9230126142501831\n",
      "Actor Loss: -1.7858408689498901\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Epoch 162: Average Steps per Episode: 12.10, Average Reward: 0.00, Average Loss: -0.5961\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Critic Loss: 0.9229995012283325\n",
      "Actor Loss: -1.4733694791793823\n",
      "Critic Loss: 0.9230126142501831\n",
      "Actor Loss: -1.7858408689498901\n",
      "Critic Loss: 0.8749303817749023\n",
      "Actor Loss: -1.4629783630371094\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.8570486307144165\n",
      "Actor Loss: -0.6540287733078003\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Epoch 163: Average Steps per Episode: 12.09, Average Reward: 0.00, Average Loss: -0.5881\n",
      "Critic Loss: 0.9089950323104858\n",
      "Actor Loss: -2.6525657176971436\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Critic Loss: 0.9229995012283325\n",
      "Actor Loss: -1.4733694791793823\n",
      "Critic Loss: 0.9089950323104858\n",
      "Actor Loss: -2.6525657176971436\n",
      "Critic Loss: 0.9443791508674622\n",
      "Actor Loss: -1.5089459419250488\n",
      "Critic Loss: 0.9473409056663513\n",
      "Actor Loss: -0.13934145867824554\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.9089950323104858\n",
      "Actor Loss: -2.6525657176971436\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Epoch 164: Average Steps per Episode: 12.08, Average Reward: 0.00, Average Loss: -1.4733\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9229995012283325\n",
      "Actor Loss: -1.4733694791793823\n",
      "Critic Loss: 0.9333092570304871\n",
      "Actor Loss: -0.2515925168991089\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9443791508674622\n",
      "Actor Loss: -1.5089459419250488\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9285461902618408\n",
      "Actor Loss: -0.0011343274964019656\n",
      "Critic Loss: 0.8888244032859802\n",
      "Actor Loss: 1.0596381798677612e-06\n",
      "Critic Loss: 0.9333092570304871\n",
      "Actor Loss: -0.2515925168991089\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Epoch 165: Average Steps per Episode: 12.07, Average Reward: 2.00, Average Loss: -0.3078\n",
      "Critic Loss: 0.9285461902618408\n",
      "Actor Loss: -0.0011343274964019656\n",
      "Critic Loss: 0.8999830484390259\n",
      "Actor Loss: 1.760446548461914\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9443791508674622\n",
      "Actor Loss: -1.5089459419250488\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9333092570304871\n",
      "Actor Loss: -0.2515925168991089\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9230126142501831\n",
      "Actor Loss: -1.7858408689498901\n",
      "Critic Loss: 0.9089915752410889\n",
      "Actor Loss: -2.219904899597168\n",
      "Critic Loss: 0.9285461902618408\n",
      "Actor Loss: -0.0011343274964019656\n",
      "Epoch 166: Average Steps per Episode: 12.09, Average Reward: 4.00, Average Loss: -0.5874\n",
      "Critic Loss: 0.9090160131454468\n",
      "Actor Loss: -1.8086612224578857\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9473453164100647\n",
      "Actor Loss: -1.0634201765060425\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.8332406878471375\n",
      "Actor Loss: -4.303012847900391\n",
      "Critic Loss: 0.9229995012283325\n",
      "Actor Loss: -1.4733694791793823\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9230126142501831\n",
      "Actor Loss: -1.7858408689498901\n",
      "Critic Loss: 0.9229995012283325\n",
      "Actor Loss: -1.4733694791793823\n",
      "Critic Loss: 0.9230126142501831\n",
      "Actor Loss: -1.7858408689498901\n",
      "Critic Loss: 0.8749090433120728\n",
      "Actor Loss: -1.7037086486816406\n",
      "Critic Loss: 0.9473453164100647\n",
      "Actor Loss: -1.0634201765060425\n",
      "Epoch 167: Average Steps per Episode: 12.12, Average Reward: 0.50, Average Loss: -1.0050\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.9090160131454468\n",
      "Actor Loss: -1.8086612224578857\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9285461902618408\n",
      "Actor Loss: -0.0011343274964019656\n",
      "Critic Loss: 0.8999830484390259\n",
      "Actor Loss: 1.760446548461914\n",
      "Critic Loss: 0.9443791508674622\n",
      "Actor Loss: -1.5089459419250488\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.959941029548645\n",
      "Actor Loss: -1.0527687072753906\n",
      "Critic Loss: 0.9473453164100647\n",
      "Actor Loss: -1.0634201765060425\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9443791508674622\n",
      "Actor Loss: -1.5089459419250488\n",
      "Epoch 168: Average Steps per Episode: 12.12, Average Reward: 1.00, Average Loss: 0.4064\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.9473453164100647\n",
      "Actor Loss: -1.0634201765060425\n",
      "Critic Loss: 0.9285461902618408\n",
      "Actor Loss: -0.0011343274964019656\n",
      "Critic Loss: 0.6665414571762085\n",
      "Actor Loss: -1.271565793103946e-06\n",
      "Critic Loss: 0.959941029548645\n",
      "Actor Loss: -1.0527687072753906\n",
      "Critic Loss: 0.7496458292007446\n",
      "Actor Loss: -3.3409433364868164\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9089906215667725\n",
      "Actor Loss: 0.0010670749470591545\n",
      "Critic Loss: 0.7496458292007446\n",
      "Actor Loss: -3.3409433364868164\n",
      "Epoch 169: Average Steps per Episode: 12.07, Average Reward: 0.00, Average Loss: -0.4242\n",
      "Critic Loss: 0.959941029548645\n",
      "Actor Loss: -1.0527687072753906\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: -0.23066391050815582\n",
      "Critic Loss: 0.8999830484390259\n",
      "Actor Loss: 1.760446548461914\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: -0.23066391050815582\n",
      "Critic Loss: 0.7496458292007446\n",
      "Actor Loss: -3.3409433364868164\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.8888205289840698\n",
      "Actor Loss: -3.123634099960327\n",
      "Critic Loss: 0.9473453164100647\n",
      "Actor Loss: -1.0634201765060425\n",
      "Critic Loss: 0.7496458292007446\n",
      "Actor Loss: -3.3409433364868164\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Epoch 170: Average Steps per Episode: 12.10, Average Reward: 7.50, Average Loss: 0.2080\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.9090160131454468\n",
      "Actor Loss: -1.8086612224578857\n",
      "Critic Loss: 0.959941029548645\n",
      "Actor Loss: -1.0527687072753906\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.7999103665351868\n",
      "Actor Loss: -4.824582576751709\n",
      "Critic Loss: 0.9090160131454468\n",
      "Actor Loss: -1.8086612224578857\n",
      "Critic Loss: 0.7496458292007446\n",
      "Actor Loss: -3.3409433364868164\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.9473453164100647\n",
      "Actor Loss: -1.0634201765060425\n",
      "Critic Loss: 0.7999103665351868\n",
      "Actor Loss: -4.824582576751709\n",
      "Epoch 171: Average Steps per Episode: 12.08, Average Reward: 1.50, Average Loss: -0.4760\n",
      "Critic Loss: 0.9090160131454468\n",
      "Actor Loss: -1.8086612224578857\n",
      "Critic Loss: 0.9285522103309631\n",
      "Actor Loss: -1.3255313634872437\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.959941029548645\n",
      "Actor Loss: -1.0527687072753906\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.7496458292007446\n",
      "Actor Loss: -3.3409433364868164\n",
      "Critic Loss: 0.9285522103309631\n",
      "Actor Loss: -1.3255313634872437\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Critic Loss: 0.959941029548645\n",
      "Actor Loss: -1.0527687072753906\n",
      "Critic Loss: 0.9090160131454468\n",
      "Actor Loss: -1.8086612224578857\n",
      "Epoch 172: Average Steps per Episode: 12.06, Average Reward: 0.00, Average Loss: -0.4134\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.7999103665351868\n",
      "Actor Loss: -4.824582576751709\n",
      "Critic Loss: 0.9166092872619629\n",
      "Actor Loss: 1.297730803489685\n",
      "Critic Loss: 0.9089906215667725\n",
      "Actor Loss: 0.0010670749470591545\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.7999103665351868\n",
      "Actor Loss: -4.824582576751709\n",
      "Critic Loss: 0.9090323448181152\n",
      "Actor Loss: -2.0046865940093994\n",
      "Critic Loss: 0.9089906215667725\n",
      "Actor Loss: 0.0010670749470591545\n",
      "Critic Loss: 0.9473593831062317\n",
      "Actor Loss: -0.23066391050815582\n",
      "Epoch 173: Average Steps per Episode: 12.03, Average Reward: 0.00, Average Loss: -0.0710\n",
      "Critic Loss: 0.9374520778656006\n",
      "Actor Loss: -0.9774041175842285\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.9285522103309631\n",
      "Actor Loss: -1.3255313634872437\n",
      "Critic Loss: 0.799872100353241\n",
      "Actor Loss: -1.144409225162235e-06\n",
      "Critic Loss: 0.7999103665351868\n",
      "Actor Loss: -4.824582576751709\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Epoch 174: Average Steps per Episode: 12.05, Average Reward: 0.00, Average Loss: -0.1778\n",
      "Critic Loss: 0.8749210834503174\n",
      "Actor Loss: 0.6028305292129517\n",
      "Critic Loss: 0.9090323448181152\n",
      "Actor Loss: -2.0046865940093994\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Epoch 175: Average Steps per Episode: 12.05, Average Reward: 1.50, Average Loss: -0.3108\n",
      "Critic Loss: 0.9090323448181152\n",
      "Actor Loss: -2.0046865940093994\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.9230518937110901\n",
      "Actor Loss: -0.06898263841867447\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Critic Loss: 0.9090511202812195\n",
      "Actor Loss: -0.4948676824569702\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Epoch 176: Average Steps per Episode: 12.03, Average Reward: 0.50, Average Loss: -0.0371\n",
      "Critic Loss: 0.9090323448181152\n",
      "Actor Loss: -2.0046865940093994\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Critic Loss: 0.8749210834503174\n",
      "Actor Loss: 0.6028305292129517\n",
      "Critic Loss: 0.7999221086502075\n",
      "Actor Loss: -1.2470253705978394\n",
      "Critic Loss: 0.7998720407485962\n",
      "Actor Loss: -5.722046125811175e-07\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Critic Loss: 0.9165777564048767\n",
      "Actor Loss: -2.9461116790771484\n",
      "Critic Loss: 0.9374520778656006\n",
      "Actor Loss: -0.9774041175842285\n",
      "Critic Loss: 0.6664392948150635\n",
      "Actor Loss: 1.271565793103946e-06\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.7998678684234619\n",
      "Actor Loss: -0.8691596984863281\n",
      "Epoch 177: Average Steps per Episode: 12.00, Average Reward: 2.00, Average Loss: 0.2035\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.9090323448181152\n",
      "Actor Loss: -2.0046865940093994\n",
      "Critic Loss: 0.9374520778656006\n",
      "Actor Loss: -0.9774041175842285\n",
      "Critic Loss: 0.9582738280296326\n",
      "Actor Loss: -0.766330897808075\n",
      "Critic Loss: 0.7998720407485962\n",
      "Actor Loss: -5.722046125811175e-07\n",
      "Critic Loss: 0.9374520778656006\n",
      "Actor Loss: -0.9774041175842285\n",
      "Critic Loss: 0.9090511202812195\n",
      "Actor Loss: -0.4948676824569702\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.9582738280296326\n",
      "Actor Loss: -0.766330897808075\n",
      "Epoch 178: Average Steps per Episode: 12.01, Average Reward: 0.00, Average Loss: 0.0855\n",
      "Critic Loss: 0.9374520778656006\n",
      "Actor Loss: -0.9774041175842285\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.7498853206634521\n",
      "Actor Loss: 0.3278157711029053\n",
      "Critic Loss: 0.8888025879859924\n",
      "Actor Loss: -2.0885722637176514\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.8749210834503174\n",
      "Actor Loss: 0.6028305292129517\n",
      "Critic Loss: 0.7999221086502075\n",
      "Actor Loss: -1.2470253705978394\n",
      "Critic Loss: 0.9230113625526428\n",
      "Actor Loss: -1.5828359127044678\n",
      "Critic Loss: 0.7998720407485962\n",
      "Actor Loss: -5.722046125811175e-07\n",
      "Critic Loss: 0.7999221086502075\n",
      "Actor Loss: -1.2470253705978394\n",
      "Critic Loss: 0.8749210834503174\n",
      "Actor Loss: 0.6028305292129517\n",
      "Critic Loss: 0.7498853206634521\n",
      "Actor Loss: 0.3278157711029053\n",
      "Epoch 179: Average Steps per Episode: 12.00, Average Reward: 0.00, Average Loss: 0.2874\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.7999221086502075\n",
      "Actor Loss: -1.2470253705978394\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Critic Loss: 0.9444178342819214\n",
      "Actor Loss: 0.047022394835948944\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Critic Loss: 0.9090511202812195\n",
      "Actor Loss: -0.4948676824569702\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.9090511202812195\n",
      "Actor Loss: -0.4948676824569702\n",
      "Critic Loss: 0.8749210834503174\n",
      "Actor Loss: 0.6028305292129517\n",
      "Critic Loss: 0.7999221086502075\n",
      "Actor Loss: -1.2470253705978394\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Epoch 180: Average Steps per Episode: 12.02, Average Reward: 0.00, Average Loss: 0.4628\n",
      "Critic Loss: 0.9333102107048035\n",
      "Actor Loss: -0.054857637733221054\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.9472787976264954\n",
      "Actor Loss: -2.218297004699707\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.7998720407485962\n",
      "Actor Loss: -5.722046125811175e-07\n",
      "Critic Loss: 0.9582738280296326\n",
      "Actor Loss: -0.766330897808075\n",
      "Critic Loss: 0.7999221086502075\n",
      "Actor Loss: -1.2470253705978394\n",
      "Critic Loss: 0.9472787976264954\n",
      "Actor Loss: -2.218297004699707\n",
      "Critic Loss: 0.8888025879859924\n",
      "Actor Loss: -2.0885722637176514\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: -1.5107303857803345\n",
      "Critic Loss: 0.9090511202812195\n",
      "Actor Loss: -0.4948676824569702\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Epoch 181: Average Steps per Episode: 12.07, Average Reward: 3.00, Average Loss: -0.1625\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.899981677532196\n",
      "Actor Loss: 0.6913103461265564\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: -1.5107303857803345\n",
      "Critic Loss: 0.9333102107048035\n",
      "Actor Loss: -0.054857637733221054\n",
      "Critic Loss: 0.9472787976264954\n",
      "Actor Loss: -2.218297004699707\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: -1.5107303857803345\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.9582738280296326\n",
      "Actor Loss: -0.766330897808075\n",
      "Epoch 182: Average Steps per Episode: 12.05, Average Reward: 0.00, Average Loss: -0.8150\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.9472788572311401\n",
      "Actor Loss: -2.2182974815368652\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.9582738280296326\n",
      "Actor Loss: -0.766330897808075\n",
      "Critic Loss: 0.9472788572311401\n",
      "Actor Loss: -2.2182974815368652\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: -1.5107303857803345\n",
      "Critic Loss: 0.7498853206634521\n",
      "Actor Loss: 0.3278157711029053\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Epoch 183: Average Steps per Episode: 12.07, Average Reward: 0.50, Average Loss: -0.7609\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: -1.5107301473617554\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.9472788572311401\n",
      "Actor Loss: -2.2182974815368652\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.9333102107048035\n",
      "Actor Loss: -0.054857637733221054\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.9411238431930542\n",
      "Actor Loss: 1.7513266801834106\n",
      "Critic Loss: 0.8999471664428711\n",
      "Actor Loss: -1.5107301473617554\n",
      "Critic Loss: 0.9472788572311401\n",
      "Actor Loss: -2.2182974815368652\n",
      "Critic Loss: 0.9090316295623779\n",
      "Actor Loss: -2.216775894165039\n",
      "Epoch 184: Average Steps per Episode: 12.08, Average Reward: 2.50, Average Loss: -0.4129\n",
      "Critic Loss: 0.916645884513855\n",
      "Actor Loss: -0.7772606611251831\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.9472788572311401\n",
      "Actor Loss: -2.2182974815368652\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.9374215602874756\n",
      "Actor Loss: -1.7888509035110474\n",
      "Critic Loss: 0.9411238431930542\n",
      "Actor Loss: 1.7513266801834106\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9472788572311401\n",
      "Actor Loss: -2.2182974815368652\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Epoch 185: Average Steps per Episode: 12.05, Average Reward: 0.00, Average Loss: -0.7233\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9411238431930542\n",
      "Actor Loss: 1.7513266801834106\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.7498766183853149\n",
      "Actor Loss: 0.4414186477661133\n",
      "Critic Loss: 0.8888006210327148\n",
      "Actor Loss: -0.743682324886322\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9642546772956848\n",
      "Actor Loss: -0.6823540329933167\n",
      "Critic Loss: 0.9411238431930542\n",
      "Actor Loss: 1.7513266801834106\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616657495498657\n",
      "Epoch 186: Average Steps per Episode: 12.04, Average Reward: 0.50, Average Loss: 0.1016\n",
      "Critic Loss: 0.916645884513855\n",
      "Actor Loss: -0.7772606611251831\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.8888006210327148\n",
      "Actor Loss: -0.743682324886322\n",
      "Critic Loss: 0.8570848703384399\n",
      "Actor Loss: 1.3623919130623108e-06\n",
      "Critic Loss: 0.9166126251220703\n",
      "Actor Loss: -2.4053802490234375\n",
      "Critic Loss: 0.9374215602874756\n",
      "Actor Loss: -1.7888509035110474\n",
      "Critic Loss: 0.7497025728225708\n",
      "Actor Loss: -2.359057903289795\n",
      "Critic Loss: 0.7498766183853149\n",
      "Actor Loss: 0.4414186477661133\n",
      "Critic Loss: 0.916645884513855\n",
      "Actor Loss: -0.7772606611251831\n",
      "Critic Loss: 0.9166175127029419\n",
      "Actor Loss: -0.17095613479614258\n",
      "Critic Loss: 0.8570833802223206\n",
      "Actor Loss: -3.034874677658081\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616652727127075\n",
      "Epoch 187: Average Steps per Episode: 12.03, Average Reward: 0.00, Average Loss: -0.4684\n",
      "Critic Loss: 0.9411238431930542\n",
      "Actor Loss: 1.7513266801834106\n",
      "Critic Loss: 0.7497025728225708\n",
      "Actor Loss: -2.359057903289795\n",
      "Critic Loss: 0.916645884513855\n",
      "Actor Loss: -0.7772606611251831\n",
      "Critic Loss: 0.8570848703384399\n",
      "Actor Loss: 1.3623919130623108e-06\n",
      "Critic Loss: 0.9332536458969116\n",
      "Actor Loss: -0.6962200403213501\n",
      "Critic Loss: 0.7497025728225708\n",
      "Actor Loss: -2.359057903289795\n",
      "Critic Loss: 0.7999882102012634\n",
      "Actor Loss: -0.4345172941684723\n",
      "Critic Loss: 0.9166175127029419\n",
      "Actor Loss: -0.17095613479614258\n",
      "Critic Loss: 0.9285184741020203\n",
      "Actor Loss: -1.4616652727127075\n",
      "Critic Loss: 0.9374215602874756\n",
      "Actor Loss: -1.7888509035110474\n",
      "Critic Loss: 0.7498766183853149\n",
      "Actor Loss: 0.4414186477661133\n",
      "Critic Loss: 0.9166175127029419\n",
      "Actor Loss: -0.17095613479614258\n",
      "Epoch 188: Average Steps per Episode: 12.02, Average Reward: 2.00, Average Loss: 0.1976\n",
      "Critic Loss: 0.916645884513855\n",
      "Actor Loss: -0.7772606611251831\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Critic Loss: 0.7497025728225708\n",
      "Actor Loss: -2.359057903289795\n",
      "Critic Loss: 0.8570848703384399\n",
      "Actor Loss: 1.3623919130623108e-06\n",
      "Critic Loss: 0.8570848703384399\n",
      "Actor Loss: 1.3623919130623108e-06\n",
      "Critic Loss: 0.9166175127029419\n",
      "Actor Loss: -0.17095613479614258\n",
      "Critic Loss: 0.9332536458969116\n",
      "Actor Loss: -0.6962200403213501\n",
      "Critic Loss: 0.7497025728225708\n",
      "Actor Loss: -2.359057903289795\n",
      "Critic Loss: 0.916645884513855\n",
      "Actor Loss: -0.7772606611251831\n",
      "Critic Loss: 0.7999882102012634\n",
      "Actor Loss: -0.4345172941684723\n",
      "Critic Loss: 0.8888006210327148\n",
      "Actor Loss: -0.743682324886322\n",
      "Critic Loss: 0.9411238431930542\n",
      "Actor Loss: 1.7513266801834106\n",
      "Epoch 189: Average Steps per Episode: 12.01, Average Reward: 2.00, Average Loss: 0.1427\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Critic Loss: 0.9166175127029419\n",
      "Actor Loss: -0.17095613479614258\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.7498766183853149\n",
      "Actor Loss: 0.4414186477661133\n",
      "Critic Loss: 0.9332536458969116\n",
      "Actor Loss: -0.6962200403213501\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.7999882102012634\n",
      "Actor Loss: -0.4345172941684723\n",
      "Critic Loss: 0.9166175127029419\n",
      "Actor Loss: -0.17095613479614258\n",
      "Critic Loss: 0.9166488647460938\n",
      "Actor Loss: -0.6184656023979187\n",
      "Critic Loss: 0.9090433120727539\n",
      "Actor Loss: 1.5696229934692383\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Epoch 190: Average Steps per Episode: 11.99, Average Reward: 0.00, Average Loss: -0.0240\n",
      "Critic Loss: 0.7497025728225708\n",
      "Actor Loss: -2.359057903289795\n",
      "Critic Loss: 0.9090433120727539\n",
      "Actor Loss: 1.5696229934692383\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.7999882102012634\n",
      "Actor Loss: -0.4345172941684723\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.7498714923858643\n",
      "Actor Loss: 3.366912364959717\n",
      "Critic Loss: 0.9166488647460938\n",
      "Actor Loss: -0.6184656023979187\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Critic Loss: 0.9332536458969116\n",
      "Actor Loss: -0.6962200403213501\n",
      "Epoch 191: Average Steps per Episode: 11.99, Average Reward: 1.50, Average Loss: 0.2536\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Critic Loss: 0.7498714923858643\n",
      "Actor Loss: 3.366912364959717\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.7999882102012634\n",
      "Actor Loss: -0.4345172941684723\n",
      "Critic Loss: 0.9332536458969116\n",
      "Actor Loss: -0.6962200403213501\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.9166488647460938\n",
      "Actor Loss: -0.6184656023979187\n",
      "Critic Loss: 0.9166488647460938\n",
      "Actor Loss: -0.6184656023979187\n",
      "Critic Loss: 0.7498714923858643\n",
      "Actor Loss: 3.366912364959717\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Critic Loss: 0.933272659778595\n",
      "Actor Loss: -2.181015729904175\n",
      "Epoch 192: Average Steps per Episode: 12.01, Average Reward: 3.00, Average Loss: 0.6225\n",
      "Critic Loss: 0.9332536458969116\n",
      "Actor Loss: -0.6962200403213501\n",
      "Critic Loss: 0.8999444842338562\n",
      "Actor Loss: -2.3206660747528076\n",
      "Critic Loss: 0.7999882102012634\n",
      "Actor Loss: -0.4345172941684723\n",
      "Critic Loss: 0.7498714923858643\n",
      "Actor Loss: 3.366912364959717\n",
      "Critic Loss: 0.9166488647460938\n",
      "Actor Loss: -0.6184656023979187\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Critic Loss: 0.7498714923858643\n",
      "Actor Loss: 3.366912364959717\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Critic Loss: 0.8999444842338562\n",
      "Actor Loss: -2.3206660747528076\n",
      "Critic Loss: 0.9090527892112732\n",
      "Actor Loss: -1.1244295835494995\n",
      "Epoch 193: Average Steps per Episode: 12.00, Average Reward: 0.50, Average Loss: 0.9452\n",
      "Critic Loss: 0.8999444842338562\n",
      "Actor Loss: -2.3206660747528076\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.9090433120727539\n",
      "Actor Loss: 1.5696229934692383\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.9090527892112732\n",
      "Actor Loss: -1.1244295835494995\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.9654861688613892\n",
      "Actor Loss: -0.48220640420913696\n",
      "Critic Loss: 0.8999444842338562\n",
      "Actor Loss: -2.3206660747528076\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.959990918636322\n",
      "Actor Loss: 0.299955278635025\n",
      "Epoch 194: Average Steps per Episode: 12.08, Average Reward: 8.50, Average Loss: 0.4189\n",
      "Critic Loss: 0.9654861688613892\n",
      "Actor Loss: -0.48220640420913696\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.7498714923858643\n",
      "Actor Loss: 3.366912364959717\n",
      "Critic Loss: 0.8749245405197144\n",
      "Actor Loss: 0.8416241407394409\n",
      "Critic Loss: 0.8331891298294067\n",
      "Actor Loss: -2.1365866661071777\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Critic Loss: 0.9090527892112732\n",
      "Actor Loss: -1.1244295835494995\n",
      "Critic Loss: 0.959990918636322\n",
      "Actor Loss: 0.299955278635025\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Epoch 195: Average Steps per Episode: 12.08, Average Reward: 0.00, Average Loss: 0.6114\n",
      "Critic Loss: 0.9544764757156372\n",
      "Actor Loss: -1.019477367401123\n",
      "Critic Loss: 0.928497314453125\n",
      "Actor Loss: -2.5785574913024902\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.9654861688613892\n",
      "Actor Loss: -0.48220640420913696\n",
      "Critic Loss: 0.9090527892112732\n",
      "Actor Loss: -1.1244295835494995\n",
      "Critic Loss: 0.959990918636322\n",
      "Actor Loss: 0.299955278635025\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.9523515701293945\n",
      "Actor Loss: -0.6151755452156067\n",
      "Critic Loss: 0.8749245405197144\n",
      "Actor Loss: 0.8416241407394409\n",
      "Critic Loss: 0.959990918636322\n",
      "Actor Loss: 0.299955278635025\n",
      "Critic Loss: 0.9544764757156372\n",
      "Actor Loss: -1.019477367401123\n",
      "Critic Loss: 0.928497314453125\n",
      "Actor Loss: -2.5785574913024902\n",
      "Epoch 196: Average Steps per Episode: 12.11, Average Reward: 0.50, Average Loss: -0.4983\n",
      "Critic Loss: 0.8749245405197144\n",
      "Actor Loss: 0.8416241407394409\n",
      "Critic Loss: 0.9564723372459412\n",
      "Actor Loss: -0.3068675696849823\n",
      "Critic Loss: 0.9090527892112732\n",
      "Actor Loss: -1.1244295835494995\n",
      "Critic Loss: 0.9411510229110718\n",
      "Actor Loss: 0.4099614918231964\n",
      "Critic Loss: 0.9654861688613892\n",
      "Actor Loss: -0.48220640420913696\n",
      "Critic Loss: 0.9564723372459412\n",
      "Actor Loss: -0.3068675696849823\n",
      "Critic Loss: 0.8749245405197144\n",
      "Actor Loss: 0.8416241407394409\n",
      "Critic Loss: 0.9285489916801453\n",
      "Actor Loss: 0.8596216440200806\n",
      "Critic Loss: 0.8749282360076904\n",
      "Actor Loss: -2.5425503253936768\n",
      "Critic Loss: 0.9472841024398804\n",
      "Actor Loss: -1.8861353397369385\n",
      "Critic Loss: 0.9654861688613892\n",
      "Actor Loss: -0.48220640420913696\n",
      "Critic Loss: 0.928497314453125\n",
      "Actor Loss: -2.5785574913024902\n",
      "Epoch 197: Average Steps per Episode: 12.12, Average Reward: 0.00, Average Loss: 0.3639\n",
      "Critic Loss: 0.9472841024398804\n",
      "Actor Loss: -1.8861353397369385\n",
      "Critic Loss: 0.8749245405197144\n",
      "Actor Loss: 0.8416241407394409\n",
      "Critic Loss: 0.959990918636322\n",
      "Actor Loss: 0.299955278635025\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.8749245405197144\n",
      "Actor Loss: 0.8416241407394409\n",
      "Critic Loss: 0.9090527892112732\n",
      "Actor Loss: -1.1244295835494995\n",
      "Critic Loss: 0.9333093762397766\n",
      "Actor Loss: -4.953572750091553\n",
      "Critic Loss: 0.9544764757156372\n",
      "Actor Loss: -1.019477367401123\n",
      "Critic Loss: 0.9472841024398804\n",
      "Actor Loss: -1.8861353397369385\n",
      "Critic Loss: 0.9544764757156372\n",
      "Actor Loss: -1.019477367401123\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.8749282360076904\n",
      "Actor Loss: -2.5425503253936768\n",
      "Epoch 198: Average Steps per Episode: 12.14, Average Reward: 4.50, Average Loss: -0.8841\n",
      "Critic Loss: 0.9472841024398804\n",
      "Actor Loss: -1.8861353397369385\n",
      "Critic Loss: 0.9544764757156372\n",
      "Actor Loss: -1.019477367401123\n",
      "Critic Loss: 0.8749282360076904\n",
      "Actor Loss: -2.5425503253936768\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.6665101051330566\n",
      "Actor Loss: -4.362075328826904\n",
      "Critic Loss: 0.928497314453125\n",
      "Actor Loss: -2.5785574913024902\n",
      "Critic Loss: 0.916573166847229\n",
      "Actor Loss: 0.022290866822004318\n",
      "Critic Loss: 0.9284635782241821\n",
      "Actor Loss: -1.3623919414840202e-07\n",
      "Critic Loss: 0.9654861688613892\n",
      "Actor Loss: -0.48220640420913696\n",
      "Critic Loss: 0.9333093762397766\n",
      "Actor Loss: -4.953572750091553\n",
      "Critic Loss: 0.9564723372459412\n",
      "Actor Loss: -0.3068675696849823\n",
      "Critic Loss: 0.9284635782241821\n",
      "Actor Loss: -1.3623919414840202e-07\n",
      "Epoch 199: Average Steps per Episode: 12.14, Average Reward: 0.00, Average Loss: -0.9837\n",
      "Critic Loss: 0.9544764757156372\n",
      "Actor Loss: -1.019477367401123\n",
      "Critic Loss: 0.937480628490448\n",
      "Actor Loss: 0.4654252529144287\n",
      "Critic Loss: 0.8999500274658203\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.916573166847229\n",
      "Actor Loss: 0.022290866822004318\n",
      "Critic Loss: 0.8749282360076904\n",
      "Actor Loss: -2.5425503253936768\n",
      "Critic Loss: 0.9599725604057312\n",
      "Actor Loss: -0.21947966516017914\n",
      "Critic Loss: 0.8999500274658203\n",
      "Actor Loss: -3.814697322468419e-07\n",
      "Critic Loss: 0.9333093762397766\n",
      "Actor Loss: -4.953572750091553\n",
      "Critic Loss: 0.916573166847229\n",
      "Actor Loss: 0.022290866822004318\n",
      "Critic Loss: 0.9564723372459412\n",
      "Actor Loss: -0.3068675696849823\n",
      "Critic Loss: 0.9284635782241821\n",
      "Actor Loss: -1.3623919414840202e-07\n",
      "Critic Loss: 0.928497314453125\n",
      "Actor Loss: -2.5785574913024902\n",
      "Epoch 200: Average Steps per Episode: 12.17, Average Reward: 3.00, Average Loss: -0.0003\n",
      "Critic saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_greedy2.pth\n",
      "Backbone saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_greedy2.pth\n",
      "Actor saved to /Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_greedy2.pth\n",
      "Finished training iteration 1/1. Model saved.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "##############\n",
    "# Parameters\n",
    "##############\n",
    "\n",
    "# I/O parameters\n",
    "device = torch.device(\"mps\" if torch.backends.mps.is_available() else \"cpu\")\n",
    "print(\"Device: \", device)\n",
    "critic_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_random2.pth\"\n",
    "critic_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_critic_greedy2.pth\"\n",
    "backbone_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_random2.pth\"\n",
    "backbone_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_complete_backbone_greedy2.pth\"\n",
    "actor_load_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_random2.pth\"\n",
    "actor_save_path = \"/Users/tawab/Desktop/columbia/Courses/Spring2024/Reinforcement Learning/project/can_chess_with_hexagons_rl/results/advanced_ac_actor_greedy2.pth\"\n",
    "\n",
    "\n",
    "# Environment parameters\n",
    "opponent_class = GreedyPlayer\n",
    "opponent_is_white = True\n",
    "apply_negative_scores = False\n",
    "\n",
    "# Critic parameters\n",
    "input_dim = 256 \n",
    "hidden_dim = 256\n",
    "\n",
    "# Actor parameters\n",
    "input_dim = 256\n",
    "hidden_dim = 256\n",
    "\n",
    "# BackBone parameters\n",
    "input_channels = 6\n",
    "hidden_dimensions = 256\n",
    "residual_blocks = 6\n",
    "\n",
    "\n",
    "# Critic Learning parameters\n",
    "num_previous_states = 6\n",
    "epochs = 200\n",
    "episodes_per_epoch = 2\n",
    "batch_size = 4\n",
    "learning_rate = 1e-4\n",
    "gamma = 0.25\n",
    "max_steps = 100\n",
    "simple_critic = False\n",
    "\n",
    "\n",
    "# Prioritized Memory Replay parameters\n",
    "memory_size = 12\n",
    "sampling_priority_scale = 0.7\n",
    "replace_samples = False\n",
    "priority_offset = 1e-5\n",
    "update_buffer_after_forward = False\n",
    "importance_sampling_beta_start = -1\n",
    "\n",
    "# Model Training\n",
    "n_iterations = 1\n",
    "\n",
    "################\n",
    "# Training\n",
    "################\n",
    "\n",
    "for iteration in range(n_iterations):\n",
    "    print(f\"Starting training iteration {iteration + 1}/{n_iterations}\")\n",
    "\n",
    "    env = HexChessEnv(opponent_class, opponent_is_white, apply_negative_scores=apply_negative_scores)\n",
    "\n",
    "    if critic_load_path is not None:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim, model_path=critic_load_path)\n",
    "    else:\n",
    "        critic = Critic(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    if backbone_load_path is not None:\n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks, model_path=backbone_load_path)\n",
    "    else: \n",
    "        backbone = ResidualBackBone(input_channels=input_channels, hidden_dimensions=hidden_dimensions, residual_blocks=residual_blocks)\n",
    "    \n",
    "    if actor_load_path is not None:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim, model_path=actor_load_path)\n",
    "    else:\n",
    "        actor = Actor(input_dim=input_dim, hidden_dim=hidden_dim)\n",
    "\n",
    "    critic.to(device)\n",
    "    backbone.to(device)\n",
    "    actor.to(device)\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in critic.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in backbone.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "    \n",
    "    # Convert model weights to MPS if necessary\n",
    "    if torch.backends.mps.is_available():\n",
    "        for param in actor.parameters():\n",
    "            param.data = param.data.to(device='mps')\n",
    "\n",
    "    # Learning system initialization\n",
    "    learning_system = AdvancedA2CLearning(env=env, \n",
    "                                    critic=critic, \n",
    "                                    backbone=backbone,\n",
    "                                    actor=actor, \n",
    "                                    num_previous_states=num_previous_states, \n",
    "                                    device=device, \n",
    "                                    epochs=epochs,\n",
    "                                    episodes_per_epoch=episodes_per_epoch,\n",
    "                                    batch_size=batch_size,\n",
    "                                    learning_rate=learning_rate,\n",
    "                                    gamma=gamma,\n",
    "                                    max_steps=max_steps,\n",
    "                                    memory_size=memory_size,\n",
    "                                    simple_critic=simple_critic\n",
    "                                    )\n",
    "\n",
    "\n",
    "    \n",
    "    # Calculate the number of trainable parameters\n",
    "    critic_num_parameters = count_parameters(critic) \n",
    "    print(f\"Number of trainable parameters in the Critic model: {critic_num_parameters}\")\n",
    "\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    backbone_num_parameters = count_parameters(backbone) \n",
    "    print(f\"Number of trainable parameters in the BackBone model: {backbone_num_parameters}\")\n",
    "\n",
    "    # Calculate the number of trainable parameters\n",
    "    actor_num_parameters = count_parameters(actor)\n",
    "    print(f\"Number of trainable parameters in the Actor model: {actor_num_parameters}\")\n",
    "\n",
    "    # Training\n",
    "    rewards, losses = learning_system.train(sampling_prob_scale=sampling_priority_scale, \n",
    "                                            replace_samples=replace_samples, \n",
    "                                            priority_offset=priority_offset,  \n",
    "                                            importance_sampling_beta=importance_sampling_beta_start)\n",
    "\n",
    "    #Save Model\n",
    "    learning_system.save(critic_save_path, backbone_save_path, actor_save_path)\n",
    "\n",
    "    print(f\"Finished training iteration {iteration + 1}/{n_iterations}. Model saved.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Critic Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1_1_rewards, v1_1_losses = rewards, losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save results\n",
    "model_name = \"a2c_ps_random_conv_v1_1\"\n",
    "losses_df = pd.DataFrame(v1_1_losses)\n",
    "np.savetxt(os.path.join(base_path, f\"{model_name}_rewards.csv\"), v1_1_rewards, delimiter=\",\")\n",
    "losses_df.to_csv(os.path.join(base_path, f\"{model_name}_losses.csv\"), index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load Results\n",
    "model_name = \"a2c_ps_random_conv_v1_1\"\n",
    "csv_losses_path = os.path.join(base_path, f\"{model_name}_losses.csv\")\n",
    "csv_rewards_path = os.path.join(base_path, f\"{model_name}_rewards.csv\")\n",
    "losses_df = pd.read_csv(csv_losses_path)\n",
    "rewards = np.loadtxt(csv_rewards_path, delimiter=\",\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAG3CAYAAAApYcPFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAACQP0lEQVR4nO3dd3hTZfsH8G/SNmnTvQdtoey9VwEZsnGhOFBUhgMRFcVXBX0dKIqir+sn4kR8VURRkFdEsKyy915lFQrdezdNk/P7IzmnWR1p2qZpv5/r4qI5I3nSJ23PfZ7nuW+ZIAgCiIiIiIiI7CB3dAOIiIiIiMj5MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7OXVgsXz5cvTs2RM+Pj7w8fFBbGws/v77b2l/WVkZ5s6di8DAQHh5eWHKlClIT093YIuJiIiIiJonmSAIgqMbUVd//vknXFxc0KFDBwiCgO+//x7vv/8+jh07hm7dumHOnDn466+/sHLlSvj6+uKpp56CXC7Hnj17HN10IiIiIqJmxakDC2sCAgLw/vvv4+6770ZwcDBWrVqFu+++GwBw/vx5dOnSBfv27cPgwYMd3FIiIiIioubD1dENqC9arRZr1qxBcXExYmNjceTIEWg0GowZM0Y6pnPnzoiOjq42sFCr1VCr1dJjnU6HnJwcBAYGQiaTNfj7ICIiIiJqKgRBQGFhISIiIiCXV7+KwukDi1OnTiE2NhZlZWXw8vLCunXr0LVrVxw/fhwKhQJ+fn4mx4eGhiItLa3K51uyZAkWLVrUwK0mIiIiInIe169fR2RkZLXHOH1g0alTJxw/fhz5+fn47bffMH36dMTHx9f5+RYuXIj58+dLj/Pz8xEdHY3ExER4e3vXR5NtotFosH37dowaNQpubm6N/vpUe+wr58L+ch7sK+fBvnIe7Cvn4ei+KiwsRExMTK2ug50+sFAoFGjfvj0AoF+/fjh06BA++eQT3HfffSgvL0deXp7JqEV6ejrCwsKqfD6lUgmlUmmxPSAgAD4+PvXe/ppoNBqoVCoEBgbyB7+JY185F/aX82BfOQ/2lfNgXzkPR/eV+Jq1WRLg1OlmrdHpdFCr1ejXrx/c3NywdetWaV9CQgKSkpIQGxvrwBYSERERETU/Tj1isXDhQkycOBHR0dEoLCzEqlWrsGPHDmzevBm+vr545JFHMH/+fGm04emnn0ZsbCwzQhERERER1TOnDiwyMjLw8MMPIzU1Fb6+vujZsyc2b96MsWPHAgA++ugjyOVyTJkyBWq1GuPHj8fnn3/u4FYTERERETU/Th1YfPvtt9Xud3d3x7Jly7Bs2bJGahERERERUcvU7NZYEBERERFR42NgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNgQUREREREdmNg4WQupBfi0e8P43RyvqObAgBYsTsRm06nOroZRERERORgDCyczMPfHsSWc+l46NsDjm4KLmUU4s0NZ/HEj0dRWq51dHOIiIiIyIEYWDiZtIIyAEBuicbBLTFtw8GrOQ5sCRERERE5GgMLqrPc4nLp6+NJeY5rCBERERE5HAMLqrM8oxELcSSFiIiIiFomBhZOSuHasF2XV1KOcR/F46O4C1Uek1NSOWKRwcCCiIiIqEVjYOFEBEGQvvZwc2nQ1/puz1VcSC/CJ1svVnlMrlFgwRELIiIiopaNgYUTuZRRJH3d0IFFfmnNi8OziyoDi/QCdUM2h4iIiIiaOFdHN4Bqb9XBpEZ7rTJNZfpYnU6AXC6THheWafDUqmOIv5ApbcsuVqNCq4OrC2NVIiIiopaIV4FOJDWvcrpRcXlFg75WqVFgUag2fa09l7KkoKJXlB8AQBCAgrKGbRMRERERNV0MLJxIjlF615Jyrcmai/pmnPEp36xmxvWcUgDA0PaB+OPJIfBWuhrOKQcRERERtUxOHVgsWbIEAwYMgLe3N0JCQjB58mQkJCSYHDNy5EjIZDKTf0888YSDWmyfrOLKdQxanYByrc6u5/vnTBrmrjqKwjLL9RTpRouxzddbJOfpA4uekX6QyWTwVbkBaBpF+4iIiIjIMZw6sIiPj8fcuXOxf/9+xMXFQaPRYNy4cSguLjY57rHHHkNqaqr0b+nSpQ5qsX2MRywAoEStreLI6mm0OhSWafD4D0fw18lUfL7jssUxGYWVQUxeqenrXs8pAQBE+nsAAPxVCgBAfilHLIiIiIhaKqdevL1p0yaTxytXrkRISAiOHDmC4cOHS9tVKhXCwsJq9ZxqtRpqdeVFdUFBAQBAo9FAo2n8O/Lia56+kWsyPQkA8kvK4KWQWTutSjdyS/HIf48ivbByRCI5p8TkvakrdCZBTFZBqbQ/Ja8Uey9nAQDCfRTQaDTwcdd/jLILyxzyPWoqxPfekr8HzoT95TzYV86DfeU82FfOw9F9ZcvryoSGnKjfyC5duoQOHTrg1KlT6N69OwD9VKgzZ85AEASEhYXhtttuw6uvvgqVSmX1Od544w0sWrTIYvuqVauqPKehZZcBbx7TX7zLIMDdBSjVyrCwVwX8lMCyMy7o6q/DxKjqu/JQpgw/XrJMUzsgSIcHO1ROq8pRA4uOVsacU9poMTxc/9zfnJfjVK4ckZ4CnuuuhascWHlBjmPZckyI1NbYBiIiIiJyHiUlJXjggQeQn58PHx+fao9tNoGFTqfD7bffjry8POzevVva/tVXX6F169aIiIjAyZMn8dJLL2HgwIFYu3at1eexNmIRFRWFrKysGr+ZDeHNDWfxw4Eb0uMRHYNwKaMIyXll+G32IJxOKcAbf54DAFx8a1yVz1Oh1WHI0nir6yDu7B2OpVN6SI+PJuXhvq8PSo+fGB6D58d2QJG6Av3f2Q6tTsCGubHoFOYNAHj9z7NYdVDfxtOvjYaygWtsNFUajQZxcXEYO3Ys3NzcHN0cqgH7y3mwr5wH+8p5sK+ch6P7qqCgAEFBQbUKLJx6KpSxuXPn4vTp0yZBBQA8/vjj0tc9evRAeHg4Ro8ejcuXL6Ndu3YWz6NUKqFUKi22u7m5OaQz4y9mS1+P6hSMFTMGYNxHOwEA5ToAqJwK5erqCpnM+tSorJJSKai4pWc4/jqZWrlTJpfeW3pBGV74/bTpucUauLm54fjlXGh1AqICPNA9KkDa37d1oBRYZJZoERPkXuf32xw46rNCdcP+ch7sK+fBvnIe7Cvn4ai+suU1nXrxtuipp57Chg0bsH37dkRGRlZ77KBBgwDop005g3k3t5e+9vVwg0wmg8qQ3rVErTUZHSgorbqOhFglO9RHKS26FhWWaZBbXI74C5mY+MkuJBkWZ9/UIQgAcOJ6HgAgIb0QANA32t/k/Lv7RcLXQ/+hyypiBW4iIiKilsipAwtBEPDUU09h3bp12LZtG2JiYmo85/jx4wCA8PDwBm5d/bi9VziiPfWz1e7o0woAoDIEE8XlFdAYpZzdeDq1ytoW4gV/oKcSYT6mIwopeWW4+T87MH3FQeQUl0OlcMGH9/bCrGH67+fFjCJsOZuObMNzBHtZjuh0CPECAGQWMrAgIiIiaomceirU3LlzsWrVKqxfvx7e3t5IS0sDAPj6+sLDwwOXL1/GqlWrMGnSJAQGBuLkyZN47rnnMHz4cPTs2dPBra+9OV21aNMrFrHtQwAAnkp9YFFarkWRUVXshWtP4Y3/ncGul0YhxNs0eMgyjFgEeikQahZYnE0tkL4e3DYAnz3QF0FeSiRmVabt3XclW8pKFeClsGhjkCHY4IgFERERUcvk1CMWy5cvR35+PkaOHInw8HDp3y+//AIAUCgU2LJlC8aNG4fOnTvj+eefx5QpU/Dnn386uOW2UbkC/VtXTj9SKfTxYHG5FkVlptOf1BU6fL/3qsVziKMNQV5Ki8DC2KpHB0tBQkyQJwbG6NdSpOWXIadYHPWwElh467dlccSCiIiIqEVy6hGLmhJaRUVFIT4+vpFa03hUCv2IRYm6wmTEQpScW2qxLdtQlyLQU4EwX+uBxaCYAMjlpou/H7upLQ4m5iAppwTirgBPy6lQwV7650zMLsGHcRcwqlMw+pitxSAiIiKi5supA4uWShyxKNFYjlgAplWzReIUpSBvpckaCYWrHOUV+nUaIVZGMqID9LU7knJK4G0ohBdQzYjFnydSAAB7L2XhtzlDav+miIiIiMipOfVUqJZKXGNR1YhFekGZxTYxK1SgpwIK18puD/WpDDJCvC1HIsTAIr9UgxuGkRCrU6HMFnQfvpZb4/sgIiIiouaDgYUT8lCIWaG0VgOLakcsDAHAiI7BAICnRlWms23l52FxnofCxSKQME9Xa/y8xppJ7UUiIiIiqgVOhXJCoYaMT8m5pcgvtaykXVhWgTKNFu5GNS6yjbJCAcC30/ujoKxCmgYFAHca0tmaE9dnAPqsUa4ulvGotdGOgtIK+KpYdIeIiIioJWBg4YTaGWpGJKQXotgwYvHp/X0wpF0ghr67DeoKHTIK1IgO1E9jOpOSjzTD9KhAw8iCq4tcWiux5olYBHgq4G9lihMABHkppHS1CyZ2qeIYy8Aio7CMgQURERFRC8GpUE6obbAnACCnuBxqw4jDoJgABHkpEWJYM5FRWLnO4oU1J6Wvra2PGNAmAO2Cvap8vWUP9EWPVr7Y8PQw9I7ys3qMOD3LmLUpWUQtlU4nQKfj9EAiImq+GFg4IR93N4t1DuLog1gY72JGkbTPuACe8fSo2hrUNhB/Pj0M3Vv5VnvcJ1N7Y3TnEMgMaWmNgxuilkwQBEz5Yi8mfrILFVpdzScQERE5IQYWTurLh/qZPHYzrHsQU8kuXHsKWUVqaHUCFIZ9fz41rEHbdEfvVvh2xgDc0SsCAJBRwBELIkC/3uhYUh4S0gtxKbOo5hOIiIicEAMLJ9Utwhd394u02H41u1j6evOZNCTnlqJcq4PCRY6uET6N0jaxHganQhHpZRVX/iycTSmo5kgiIiLnxcDCic0f2xGt/Dwwf2xHaduMIW2kr19ZdxrTvt0PAOjeygcuZlW1G4qYIWrnhUxcyihCXkl5DWdUTaPVYe/lLGgM00dS8kqhrtDWSzuJGkuWUZA9/9cTuJpVXM3RREREzolZoZxYhJ8H9iy42WTb3f0icT6tECv3XgUAXM/RF7WbPaJdo7Ur2BBYXMwowpgP49HKzwNb5o+wusC7Jr8evo5X1p022dYr0hfrG3haF1F9Mk7ZDACf77iEpXf3clBriIiIGgZHLJoZVxc57uprWo9i2qBojOsa2mhtiAnyNHmcnFeKXw4l2fw8CWmFFkEFAJy4kV/lOaXlWhbmoyZHLFAp2nkhi59Tqle/HrqOe7/chzf+dwbXsjkiRkSOwcCiGQrzdZe+fn5sR7x9Zw/IZI0zDQoAekb64fNpfbH1+RF4a3J3AMBXO6/YnGpz1spDNR5jfHF25FouBr69BU//fMy2BhM1MHEq1ND2gXBzkSGtoAxXs0sc3CpqTj7fcQkHE3Owcu9VjP1wJ45cy3F0k4ioBWJg0QwFeVYWq1O6OaaLJ/UIR7tgL9xjWGCekl+GHBvXWiTnlUpf32O2UL1Mo8X28xno81Yc/jmThvxSDaYs34tCdQU2nEy1/w0Q1SMx/fPwDsHoE+0PABj1wQ6UabheiOxXodVJgaq30hXlWh32Xsp2cKuIqCViYNEMyY0WafdvE+DAlujrZvi465fy5JVobDpXZbQmIzpAZbIvu7gcM1ceQl6JBs+sPob9V0z/iGpYK4CaiOS8Uvx9Og0A0DvKD4NiKn8mdyRkOqpZ1EzodAJ+3H8NAKBwlWO6IYFHOusIEZEDMLBoprY+PwLfzRiAvoa7o47kbyjeZ2t2qACjKuHRgaaBRVp+5WiG0tUFRWUVJvvN57QDwOXMIpSUV1hsJ2ooOp2AJ344AgBQuMjRI9IXk/tUroFiEUmy1z9n0/DGn2cBAK0DVAg1TIVNL1BzHQ8RNToGFs1Uu2AvjOoc4uhmAAD8PNwA1G7EQqcTpAAk0CiwiDIbsZiyfJ/0detAlUXAYF6c7/cjNzD6P/GY+MkuFJTZNnJCVFcJ6YU4laxPNvD+PT2hUriiXbAXpg2KBsAikmS/vZcrR2v7Rvsj1JCVL+5sOmIWbsRHcRcYYBBRo2FgQQ3OT6UPEHJrMWKxcO0pDHxnK7YnZMBXVRlYxAR6VnmOj7sbistN56qbF+f7ZnciAOBadgm2nE2Xtn+3JxHzfzmOYrU+MBEEweZF5kRVuWRYW9G/tT/u6F05UhEmFZHkiAXZx9/o9+SLEzoh1MfdZP8nWy9KwS0RUUNjYEENzk9lOWLxy6EkHLmWa3HsL4evo7xCh5nfHUJ+qf74Fyd0kp4DALyUrgg3ynxVUl6BErXpiEW22VSojILKC7jU/MqvF/15FmuPJePldadwPacEw97bjv5vb8HBxNpnVDl1Ix+FHAUhKy5n6gOLdsFeJttDfPR3lVmdnuwljtbOHt4WgV5KRAeo4OZimgUwzuhmik4ncASDiBoMAwtqcNJUqFL9iMWZlHy89PspTFm+1yIlolhcDwBOXM8DALQO8DRJlzttUDR2vTgKn97fBwBQrNZajFgYP9bqBJOMVGKQoTUamfjfiRTctHQ7kvNKkVNcjpd+P1mrP76nk/Nx22e7MeqDHYi/kAkOdpCxc6kFAIC2waYjbuJd5d0Xs5BqtF6IyFZFav3vOpVCnyTD31OBtXOGYtvzI/D6bV0BVAa4p5Pz0W9xHIa9tx3n0woc02AiatYYWFCDE6c0iSMWWUWVF/kzvzOtVWHtWl7havoxFaAvBBjl7wEAKC6vsFhjYTyCkVNcbvK86YZ57QWllaMM5q+bmFWMj+IuSKMmVTmTki+9p0d/OIZVl/kjRXoZhWXYdj4DADCkXZDJvth2gWgTqEKFTsDao8nVPk9BmYbrgqhK4jROT2VlFr0ekb5oG+wFL6U+2Cgp1yItvwy3/t9u5JZokJxXig0nmJabiOofr4KowXkZ/uCVGEYRjC/6C8oqTHL5l1rJ2mQ+rC+mrzX+o1lsuGsnDmyUGJ4zs1CNh1ccNDlfTMOYV0PQ8Om2S/jPPwn6Y0vKEX8h06S2BmC5+PZolgwarY7rNFoorU7A3stZKNNosflMOjRaAb0ifdEj0tfkOKWrCx4ZFgMA2HWx6pSzZRotxn24Ez3f+Acv/naCU1jIgnhTxdPw+9CYuK1ErcWF9EKTfSl5HCkjovrHwIIanDhEL95ZKzRfD1GsH8HQ6QQpIDAmjlgsntwdQ9sHYsZQ/QWZyvBHs0hdOWIR7KWfSiUGLx/GJUjTUURiMFCb9LcHE3Pw18lUDH13G6avOIh7lu+FuqKyjWKQ8qjhIlEryDD+kz0Y+M4WVr5tgTacTMEDXx/A5GV7sN0wWjGhe7jVYwcY6lmcTal6SsqN3FKkGabu/Xr4Bq5kFddzi8nZFamrDiw8DLWAissrLJJnXOZniYgaAAMLanCeZiMWheY1JwwLWMsqtFanQikNgcWDg1vjp0cHSyMVnoY/muUVOmnKkrhGQ1xj8b/jKdLzdAn3AaCfoqLTCVWOWPSM9MV7U3oAAM6nFWLuqqPS86Xkl6HTvzdhz6UsAEBavr7tbYI84W6ocn49txRZReVY9OdZCAIXSrYkRw0JCc6nFUrToHq08rV6bISffipfQVkFSsutV+A2TjoAABfN7joTib9XvYymQok8DTd1Ssu1yDHcwBETX5y4nof1x61Pwyuv0KG8gkVGich2DCyowUkjFoZRBfNidtnF+otzcTqTOTcX6x9T8XkB4NBV/QWdGFhcTC/Es6uPSQHBW3d0w+fT+gIANFoBuSXlyK+iroaLXIZ7+0eZ1NGYMaQN3r2rh/R4y7l0VGh10gLIMB93k7SPAHDyRj6e/eU4hry7DWn5TCvaEpRaGXEzTkhgzFvpKlWXTyuw/vkwr56ckFZkZwupuRFHLIx/H4pUxiMWhsCip9G0vHXHLAMLrU7AhI93YtxH8SYJLoiIaoOBBTU48a5ZiSFwyCk2XZcwa+VhnEnJR7rh4spVbrqmwnzxtvF2pdm+cF/9XeATN/Lxh9FoxUOxbRAT5IkgL/3Ff3qBusq6GpO6h0Mmk2HFjAEY2SkYXz/cH2/c3g1TB0ZjtKHoYGpeGX4/egM3ckvhpXRF72g/+BulxBWtP56C1PwyfLcn0eprUfOSaSV9rPiZMyeTyaR6FlUFnuKImOhKFgMLqqSu0OJKpn5Kk5eVqVBiYFFSrpUy43UK80H7EH36Y2txQ3axGleyinE1u8Tq55mIqDoMLKjBqZSVd82u55Tg+33XLI655dPduPX/dgMAKnSCyYJtRRUjFgAQ4Gl60TasfVAVR+qJaT4nfboLvxy6DgC4qUMQ3FxkiG0biE/v74MZQ9sAAHpF+WHlzIEY2zVUOv/eAVEAgLOpBfh06yUAwDOj2yPISylNSQCAfq39TV5Xo+Wdv5bAOOMZoB/9Mh/JMiZ+HtOrGrEwbG9lmDbFkS8ytnLPVelrf0/Lz5k4ilFSrsWFdH1QGqByw2u36tPQplv5PBnXGypSMxsZEdmGgQU1OOPsTcapNe/q28oiMDA/B6h6KhRgWnX2mdEdTFIuWmNclfZ8mn6++s2dQ3Dw5TH4ftZA3N4rotrXizCMiCTllCA5rxTRASpMHRgNAKjQVs5J7hzmXW07qHnKMivM6OvhBrnZCJwxcZqU+XkiMcGAuD6oqgDE3OnkfLy87hR2JGTU6nhyTmLhu9GdQ6Tg05h4U0erE6Sin/6eCoQZ1llczy2xOEdciwGYBhlERLXBwIIanDTPV10B42t2f5UCR18di4GG7DjGjDOcmE93MmYcmIT5uFvNjGIcpNza0zJDT6S/Cv6eiiqnXBmL8KsMTHpF+uL3OUPg466fArXo9q5opRKw7onBaOVv+kc+o5B3mps7QRAsAoQwo0DWGk+joNsasfiZOHUlraCsVskAFv91FqsOJGHGd4fw7W5Ow2uONFodjhmKiL5xezerx6jcTG+0eLu74qYOwdINlpJyLX46YDqCzMCCiOzBwIIanLjGQl2hQ1JO5R2yO/u0AgA8fXN7k+PbBXvCw+gPYnUX/MbD/2G+SimIEXUO88Z/HxkoPb6rbyROvTFOqiEAAFEBlnf6qhLopcTsEW3xcGxrrH481mRh7vAOQXixlxbdW/lgeIdguBjdqd5wMpXTWBxkz6UsvPG/Myb1UmyxPSEDj//3MP53IgXrjydjzIfx+O++qxbH5ZdqLKa8zR7Rttrn9jRaXGuNmKJZrNxdptGhoNT6saKMgjLsv1KZ6vitDWdrPdJBziO7qBxanQBXuczqaAWgLyRq7IdHBiHAUyHVAgKAHQmmdVRMAosaav0QEZmzvL1LVM9URtOTfj18AwDw0oTO6G5Iw2k+B33VY4PxyPeVFbmrm5rkaRRIhPl4ICbIE/4qN+Qa7rStf2oolK7md+3c8OKETjibUoDi8gq0DfKy6f0snNilxmO6t/LF6scHY9PpNOmOcfyFDNw3INqm1yL7TfvmAAD96NYzozvU+rzU/FJM/GSXdNf2H8O0EwB4bf0ZLP7rHA4sHC0Ft+Joha+HG3a+MAo+Hq6QyaqeBgVU1mIpqSIjmhhwBHoqEOSlQFZRORKzi9Fb5Vflc/77j9MW23Yk1P6zV6HVWVyQUtMjLqwO9FJUO91ONKVvJHpH+QGAITlFf8xaeRjJuaaF8kxHLGqu9UNEZIx/PajBKVzkJnfvAdPc/oFGWXMGxgQg1MfdZMF2dSMWxjUxOoV5w0Phgr0LRmN6bGu8NKGzRVAhUrq6YNVjg/C/p4bVagpUXQxoE4BXb+0qpXcsVmvx331X8fK6UxYVvKnhHb+ehwobqqKvPni92qkg5RU6rDDK9pVhuNAL8lLAV+VWY1AB1DxiYVz8TAzEX1l3CjeszI0XiaOCr0zqgvsHGpINVFGE71hSLm7+YAfWHr2BpZvO494v9qHvW3F4dvWxGttOjiUGslWlMxbd3S8SbQJVeO22ribbI/1VAPSJKMSpmoIgYOOpVOmYfI5YEJGNGFhQg5PJZLijV4T0eMaQNohtFyg9Nl4nIRZvMl4rYZ5+1ticke0Q4q3Eh/f2koIXD4ULFt3RHXNGtquxXY2hS5h+4W12sRqvrT+DVQeS8FX85UZ5baqUXlCGsR/txL1f7qvx2IIyjZQ1DAD2LLgZH93Xy+I4sVAiUJkRKsir+gs9Y9ZGLD7ecgG3f7YbhWUaqeaLl9JVutt8JqUAL6w5WeVzihecQ9sHSaNxVV0g/rD/Gq5kFWP+ryfw+Y7LOHg1BwVlFSapmqlpypQC2eo/bx/c0wvb/zUSvh6m6bCNp0/9e51+lOt6TqmU1AIwHb0gIqoNToWiRvHe3T1xR59W6B3lZ/EHznhU4WZDnYhZw2KQnFeKUZ1Cqg0AurfyxcFXxjRMo+uJOBUso6ByYW9Vi3Wp4Zwx3LVPzCpGYZkG7maDWRfSC/HNriuYPqQNNp9JR1pBGUJ9lNj+r5FQKVwRHeApHbtgYme8+/d5ZBtdeIkV5Gu6g2zMeMTialYx5q46KrXznzPp0hoLT6Ur7u4XiR/2XUN2cTkOX8tBmUYLd7PFuVqdIF0MBnkrpJ+1qubKV7cOvEhdYbU2AjUNmUW1CywA6zdRjG/eiFXizWv7ZLCOBRHZiH81qFG4ucgxomNwlfv/c08vXM4swm099SMbozqFYFSnkMZqXoMSF69nGmUMKjdKTUsNp6oMSukFZWjtb5qx6a0NZ7HrYpa0DggAhrYLkmoBRAeopO1dDelfs43qVmTU8g6yMeM6Ayv3XpWCCgDQCoJUOd5T6YIQb3cc/vcYDF6yFekFapy4nodBbQNNni+3pBw6AZDJgACVfkoWUHV2n6qKRAL6mhliNipqemo7Fao6797VAwvWnkK7YH0/m38euOifiGzFqVDUJEzpF4kXJ3Su1SJEZyOOWBhXsVVrGFg0BnWF9e/zsu2XMeS9HdiXXvl5s7buxTjrWJCXAmO6hGBIu0D0ivQDoL+rv3jDWeh0gnR+VRl6rDFOxWxec8J4Ua04ciCTydAhxLvK9ooXm/4qBVxd5PAzjFgUWBmxeG/TeYuMQM/c3B4dxNS2zGLWpNV2KlR1ukXo1+2IU+XEANTbkDWKnwEishUDC6IGJo1YGAcWFZwK1RjExc/m1h1LRmZROQ5lVv4K9DObogcA/qrKbTKZDN9MH4BVjw2Gj0flYO83uxOx53IWkg0Lqs1rmFRHLOhYWFaBq9n68283rEcSi5fJZTBJvxxkSHaQaWWaSlZhuckx0oiFWWBxLbsYy3dYrvPxU1UWT0vj3eomLauoMllAXflJnw/95ybXrCBjRqHapPAnEVFNGFgQNTDxrrTxVKiq7qRT/RIXRSuqSJ9aahTfiWsTXp7UWdrmp7J+0WY+Z12sxA7YOmKhD1DEc2WyypoVSYZAw1Npmra2umrd2cWmd7H9PBTSe5vw8U7kGt5jcRXpbX083KQECmn5zFzmaJczi/DkT0cssnptOJki1SqxZyqUGHiWaXQo02ilNN3tgr2kbHk/7r9W5flEROYYWBA1MHGRpPF0fwYWDU+rE7DuWDIA/QXzxmduku7QigqMbuSLgUUnQxYvABbHGzOuqp2UXYJ0w+L8CBsCC3E0S+SvUkhBweFrufrn8zV9PvFC0tqIRWVtA6VF+8+nFeLrXVcA6LNeWSMIAsIMr5fKaTAON/eno9h4Kg13Ld8jbUvJK8VTqyrTAQfbMRXKS+EKcfZpQalGKvwY6KnAnb31BUy3m02XIyKqDgMLogZmXg0c0NdAoIa14WQKPtpyAQBQWKZB1wgfHHt1LBZO7IxOofp1CsUafUG4nOJyFBhSu4r7AMClmoxkv86OlYKLL3fqL9g7h3nbNDUlyNv0WH+Vm0n6ZQCINJtaJQYeey9nWzxfZcpb/XO4u7lgcNsAaf8X8ZcRfyGzyvSzQd5KoxEL08Bi1YEk3PPFXqRyJKPRXM4sAqAfURCZj1TZs8ZCLpdJmcOOX8+T1liE+ihxS89wAFxnQUS2YWBB1MA8raTs5BqLhrX3UhbmrT4uPRZHiGQyGWaPaIe/590EV7kMAmRYsukC+r4VJx0bYjS1xPwi31h0oAr39I802fbulJ421UdRKVxN1nEEeioRa5bpKdzPNHuVGMxkFKqx84Lp3eRsKylIf35sMC6/MwntQ7ygE4BPt140Wcz9wKBoLJ/WF0+ObIeRHYOl5zdfY/HyulM4dDUXsUu24UxKfq3fI9WdSmH5u8N4GlsrPw+L9N228jGcn2BUv2Jyn1aIMHzuUhhIEpENGFgQNTBri4LFC11BEPDj/ms4ci2nsZvVbGUWqvHQioMm2yb3jjB5LJfLpAun/+5PkrYPjAmAXC7Dsgf6Yt7oDhgYE4DqiNlzAKBNoEoqYmcL46lT/p5u8PdU4Myi8dI284vL/m0q23Qtu9hkn5iu1njURCaTwUUuw2u36isvZxeppRELDzcXLLq9Gyb2CMeLEzpDJpNJi7eTckpwOjkfx5JyLdp8+KrlNqp/nlZGO0sMVdqDvJRYN3eI3Zn0xM+XGEAMaOMPb3c3aUpcYVmFNHJCRFQTBhZEDax9iBeeH9vRZJuYbnZHQib+/cdpTFleczVoqp3fj96AVqdf0NImUIU/nxqGt+/sYXHcc6PbQ4bKhS+t/Dzw2f19AAC39AzHc2M71jj64KWsDBpDfNyrObJqxoFFgKd+pMFT6YpPpvZGj1a+eDi2tcnxClc57jWMlIjTtwDgaFIuzqaKgYXl9BhxilNuiUY67+5+kXAzW9jeNtgT/io3FJZV4Nb/2407P9+LbefTYfytqK7+BdUf49HOMyn50Gh1Um2TDiFeCPGu22fOmDhVMzlPP0IlJiwwLo74/K8n7H4dImoZGFgQNTCZTIanR3cw2SZOhTrMkYp6Jy5gfmJEO+x4YRR6RPpanY52a89wLBmgxZ4XR+Drh/tj90ujbA4OjEcsQusYWIipPQFgTJfKopB39G6FP58ehkh/lcU5Pu6G+hRGi7D3X6lcc2FtpEW8YMwv1eAPaVG75fdF6eqCR29qa7Jt1srDJskHcosZWFRnz6UsvLLuFErKK3A2pQAPfnMAW8+l2/w8HkYjFrd8uhtzfjxqVI3dcjSjLsTAItWQmcx4al6vSH2di6oW+xMRmWPlbaJG8tOjg/DkT0eRX6pBbokG8RcykVPMP9j1rUyjD9rc3Wq+b+Lhql9T0SqgbhWmTQKLOqb9nD28LYrVFfBUuODmzrWrNu8tBhallSMWlzP006KeH9tR2m/MOENUUo4+lW1VGYWeHNkO729OqPL1c6uo5N3SpeaXYsaKQ0hI169X6B3lh3Ophdh9KQu7L2Xh9KLxJiMBNTFP8rDlXLr0uba2/qIuxBopyVJgUTmN7sP7emP0f+KRUWCZgcxWgiDYtP6IiJwTRyyIGsnQ9kH457nh0uPpKw6a3PnV6gTkl2ogGN8attGljCIMe28bfjrQcnPPi+tXlK71c0e3OsYX8HUdsfBUuuLVW7ti/rhOtb7wEkcaCo3uJIvz4NuFWA+S3Fzk8Da7qL2rX6TVY2tqB6dCWffibyeloAIArueUIL2wchG8rbVBSgzTnn6fMwRD2ukX9Z9O1i+cr+8RC/G1jGu3iNPnitQVJp81WxWWaTDqgx2Y8d3Bmg8mIqfGwIKoESldTX/kcowu0O78fA96LfoHd36+V1ojYIsyjRZjPozHjdxSvLLutN1tdVa2jFjYq1OYN9oGe6JPtJ+UnrMxVE6F0o9YFKsrcM6wvqJjaNWjL36elYHQlvkjpOepyaf398EtPcIxY0gbAAwsqnI8Kc/k8eXMYpObB7aMUB64ki2NLHkpXREdoJ8SJ1Zor7cRC7PnMR7ZUilcpaxT9qSdXX88BVezS7AjIZMZ8YiaOU6FImpE5nfR842mlJy8ob8Tefx6HlLzS63Ora/O70dv2N/AZqAxRyy8lK7Y9vzIBn8dc+IULDFtrP6CTYc2gSq0C646sHCVVwZb5vUxqnN7rwjc3isCx6/nYeXeq8gpYmBhrkyjRaG6wmTbX6dSTR7n2LA2Za5RETyVwsWioru1jFF1YV5nx3zhf7ivO/JLNUjJL0MHoxovVdHqBCTkyzCyvAK+bvqg5NSNyvTEGQVqRAXY9ruNiJwHRyyIGpHCbMTCvNiVKLcOay+Omd0tbakac8TCUcTaA4lZxShSV+BShn4a1KCYwGqnMRmnw3V3q/7C9A5Dit5xXUOlbVGGYCQlvwwZBSycZizd8P1wd5Pj9zlDrB5jy0iP8e8GT6UrQn1Np9qpbFirUR3zwCLEbK2QmH74w7gLGPTOFmlkzFhBmQa3/t8u9F8ch86vx+Hzsy74cudVab/xdLCUPNbFIGrOmu9fXqImyMUs53y22R3MDob58Tn1MNVEvMBuaRpzxMJRxMJ9+aUadH99Mw5d1WcXC/WpfgH5K7d0Qa9IXzw7pkO1xwHA4snd8cE9vfD+Pb2kbYFeSvQ0ZArafNb2LEfNmThVKMzHHX2i/PDAoGiLY2wJLKKN7up7KV0tFvabT6usKw/zwMLHfMRCH0yeuJ6H9AI13v7rnMVzbDiRitPJBVLldwBYdzxF+rrQKC2yeeFFImpeGFgQNbJnRld9USf+Uc+rQ2BhngK0qtGQ5k7dAkYs2gd7oXuryjS1uy9lAai5lkaQlxLrnxqGZ8d0rPY4QL8w/e5+kRaVnW/vpR/J+DL+sl2JBpob8YI51McdcrkM79zZAydeH4dORtOHlm5KQEZh7S6sBUONlQUTO0PhKkeQlxJX370FCyd2RttgT4zsVLsMYjXxNFpjIZNZnwplzNqAmHkFeKAy2xQAFBkFFil5DCyImjOn/su7ZMkSDBgwAN7e3ggJCcHkyZORkGCaIrGsrAxz585FYGAgvLy8MGXKFKSn804bOc78sR2tLrB1lcukjCw5xeUoUlfg7uV78UX85Vo9r/nohz2LLZ1ZSxixEKuDi4upRebTWBrC1IH6O/E3ckst1hS0ZOLPm/GFuK+HGzbOuwmvTOoibft068Uan0tdocX1HP2UoQndwkz2zR7RDtueH4mYIM/6aLbJiEWASmFRMDHMLLDw8bBc8C9W7e4b7Yd1TwwGACRmF0NnSEJhnFFKrDNDRM2TUwcW8fHxmDt3Lvbv34+4uDhoNBqMGzcOxcXF0jHPPfcc/vzzT6xZswbx8fFISUnBXXfd5cBWEwHLH+yHAW38TbZ5Kl0RYAgscovL8f3eqzh8LRfv/n1eOia7SF3lXWJxYah4h/n49bwGaHnT1xLWWABA60BPvHF7NwxsU1kMr67Vv23hpayfTEHNjTRiYXYh7iKXYXKfVlK2pTMplmsUzC3ZWPkzH+ClqOZI+xmPWHRr5Wuxv6tRAUcA8LKSjUoMFl6/rRs6hXlBDgGCADz47QH8fuQGsoxuerTUkVSilsKps0Jt2rTJ5PHKlSsREhKCI0eOYPjw4cjPz8e3336LVatW4eabbwYAfPfdd+jSpQv279+PwYMHWzynWq2GWl35i6+gQP9HQKPRQKNp/KJQ4ms64rXJNrb0VbSfEj/M7I/fjibj3+vPAgC8lC7wddffPcwsLIPcaM6BRqPBX6fS8OyvJ/HyxE6YOaS1yfPpdAKyDX+wx3UNwZojydh1MRPTB0fVy3tzJmJg4QKh2r5oLj9bPVp546BhjUWYl2ujvJ8wHyXySzX4eudlLJxgvSBffXKGvkrJ1aeBDfFSWLTTz12Obx/qiylfHkBKXmmN72Pl3qvS1+7y6j/H9oqN8cPYLiGI8vfA3JFtLV6rU4gKM4e0xnd79bVxitWmfwt1OkEKFvw9XACdFje3ErA9RY69l7Ox93K2yfNlFpY16X5sSZzh54r0HN1XtryuUwcW5vLz9SntAgL0d/COHDkCjUaDMWPGSMd07twZ0dHR2Ldvn9XAYsmSJVi0aJHF9n/++QcqleNS5MXFxTnstck2tvSVfva1/sewlVsJcm9cBOCC+DPX4acQIA4qbty4ES8ccAEgwzt/JyA074zJ8xzNkqG43AVKuYDW5dcAuCL+Qha+WbMREfUzY8JpFJbov08H9u1GUi1+ZJ39Z6utBhgZLkdHXwEHdm5tlNeUlckByLHmSDLSk69jSoyuxnPqQ1Ptq4xSYPNZ/c9x8sUz2JhjWUcmvxwAXJFRUIY//9oIl2pqELZSuSC5RIaBwTr8/fffDdNoI7f6ARCA3dutT7vsKQDtvF1wuVCGK9dTsHFjZWrrYg2g0erf+6Fd2+AqB26LBvoG6rDxuhync01HDhNTs7Fx48aGeisN7ni2DEeyZLivrQ5eDRtPN5qm+nNFlhzVVyUlJbU+ttkEFjqdDs8++yyGDh2K7t27AwDS0tKgUCjg5+dncmxoaCjS0tKsPs/ChQsxf/586XFBQQGioqIwbtw4+Pj4WD2nIWk0GsTFxWHs2LFwc2smv8Waqbr2VZs+BVh/PBVzR7ZFkboCP/1nF1JKZEgpqbzymDBhIt44sQO5hroXkyZNkvbpdAL+88luAKV4YmR7zB7VDsc1x7DlfCYuukTj0Und6+09OoOFR7YC0GLszSNNMuuYa04/W/c28uvtKD2F88f1NRoO57jh27mjG/T1mnpfLVx3BkAyAGDiyFiTtL4inU7Am8e2oEIHtOk9FK/97xw6hHphSh/9YvhBMZVT2paciQegxoIpsehhZXqSI7i0TsMzv5yEyjcAkyYNBADklWiw61IWcPgU/DzccPut46S+mjF5LG4t0WL4BztNnietVIbInkOl7GLORBAEzHtNf2EXGByCrx7s6+AW2aep/1xRJUf3lTh7pzaaTWAxd+5cnD59Grt377breZRKJZRKywWQbm5uDv3Bc/TrU+3Z2le9ogPRKzoQABAIfeGr4nLTVLHFFZCCCgDIK9Mh2LBQNyWvFEk5pXCVy/D4iPZwc3PFuO7h2HI+E2uPpWDqwNYYaHTR0tyJi7e9PJS16gf+bNlu6sDWOJCYi5T8MpSUa/HSujN4/+5eFumU61tT7atSTeWITcdwvyrbGObrjhu5pbjriwMAgNMpBVh3LAVuLjIcemUM/FQKCIKAHMPPerCPqsm8Xz9P/dqR4nIdXFxcUaiuwM0f7ZJSyQZ5m/68ubm5IcLf+pqf346lol9MUMM3up5dySySvt6ekIU1R1OtphUGgIvphXB3c3GKYoBN9eeKLDmqr2x5zWaxuvGpp57Chg0bsH37dkRGRkrbw8LCUF5ejry8PJPj09PTERYWBqKm6L27e1ps+2zbJZPHn+/QPy6v0GH8R/o7gqE+7vA0FM2a0L3y8302JR8tRUZBGbSGTDTuzTgrlKMNahuIvQtHS9nN1h5Nxo6EDFzJLMJn2y4iv1SD/FIN/m/rRZy8kefYxjaCIkN2rBfGd7JIz2vMvBaFSKMVcMJQnbq4XItyQ3Ac2MALt23hZVTt/e4v9qLXon9M6lMEe1nekHM1yzAlVozPKXaeBdyrDiRh6lf7cORaLuLN0uou2XjOar2gnOJyjP1oJ25aup0pmanFcerAQhAEPPXUU1i3bh22bduGmJgYk/39+vWDm5sbtm6tnHeckJCApKQkxMbGNnZziWrl1p4R+Prh/ibbvt931eTx4au5AICDiTlSyk/xjzYA+Li74X5DWtC80paxMC+zUI1h722XHiubeVaopuBlozSqz685gXf/Po8P/rmAXov+Qa9F/+A/cRfwwNcHLGqsNDcZhqxIXSOqny47Z2Q76esld/XA/oWjMbS9frTypCGLW46hyJy7mxwqKxmYHMXbcNMiOa8UR5PyLPYH15DqeGSnYCyerJ+WWeQkaYpLyivw8rpT2H8lBx/GJUj1Ol6a0Bmt/DxQqK7A1nMZFuddyqgc2ShUV+DAlewWW7CUWh6n/ss7d+5c/Pjjj1i1ahW8vb2RlpaGtLQ0lJbqc2r7+vrikUcewfz587F9+3YcOXIEM2fORGxsrNWF20RNxdiuoUhYPEGqpOxvSFXpaphqkpBeiAqtDkk5lQuqjO8eApVpZ/NbQGBxObMIA97egnKt/k7vTR2C6q0yMVVtZKcQfDtdHwTnlWjwj5Vq3EXqClwymkLSHF3KKARQcx2RcF8P/D5nCJ4d0wFT+kYizNcdQ9rppwQlZuvTpGcaMiwFejZ8TRJbRAWo0MrPo8r9VQUWnz3QB8M7BuP9u3vBX6rT4xy/k7KNKonvv5KD7Qn6wGJgjD+Gd9T324X0QovzjH/n/rQ/Cfd9tR+zVh5q4NYSNQ1O/Zd3+fLlyM/Px8iRIxEeHi79++WXX6RjPvroI9x6662YMmUKhg8fjrCwMKxdu9aBrSaqHaWrC7wMdwmzDH/gPp/WFyqFC8ordLiaXWzyR808gGgpgcXiDWcx+j/x0uMXJ3TCf2cNhMxaiWCqd6O7hKJzmHe1x+Q04xGLXw9dh0arn+5S0117AOjX2h/PjukIhSHwDTJMd8ozrKtIL7AstNcUuLu5YPXjg6usD1PVe7+1ZwT+O2sggr2VUmDhLCNYeUbr2sQplgAQ6a+CjyHFckl55Q0ddYUWL6w5gcf+e1ja9tVOfaatvZez8dOBa5waRc2eUwcWgiBY/TdjxgzpGHd3dyxbtgw5OTkoLi7G2rVrub6CnIa4ZkIU6KVE60B9/tjrOaVIzKosBvn2nabZn8TAoqAZBxZnUwrwze5EAEDbYE8smNgZs4e3Y1DRyAa3DTR5vH7uUFx6e6K0pqA5Bxb7r+jrNAR4KqyuM6iJn3ixXaL/HqXmWy+01xREBaiw4emb0MtK1qugWrx3f0/976ScknKnuMDOMfSJeVKCYC+lNE2tSF05xenw1VysOXLD5Fjjc19Zd9rqNDKi5sSpAwui5s7TbI51gKdCmo5wI69Uqnz87l09cEfvVibHtoQRixOGhcHD2gdh2/Mj8cSIdg2emYgsRfiZXgSH+Cjh6iJHgKc49aX5BhZixe3Xbu1ap4BW/B6Jd/HT8vVTecMboYp6XbQP8TJJjSvqEl79qBVQ+V7LK3ROsc4izxBYmL9fuVwGT6U+OUSx0fsQK5Abyyoy/ewb3wwiao4YWBA1YZH+pnOaA1QKtDJcxKXklUoXNf1a+1uc2xICC/GCtalNG2lpIszm3otTXlpEYCGOMNQxEBDXT+WWaHAtuxhf79KPwIU14c90p9DKIKJXpC/eubMHukXUXJfCw80FYYbv06bT1mtJNZbrOSUmQYE1YrDnr1Lgrj76Gzfi9FRxNFn8HQwA2Ybjjb8/5oynThE1RwwsiJow4xzpngoX+Hi4opUh2LiQVigFDdamTbSEwEJcXBnQhNJytkS9Iv1MHru76e/mmt+Nry1BEHA+rQAabeNU9K6tI9dysP54sjSNRxAEaepSXYNbMQjLL9Xgrs/3StubSmE8ayb3aYVVjw1C3HPDsf6pYVXWcjAnk8lw74AoAPo1B45y8kYeRv8nHt1e34xHvz+MrCLr6W/FeiL+nm5YfGd3PHZTDH54RF8cUAwsDibm4Ls9+mAw2/A8se0C8en9fUyeSww2mnOQTQQwsCBq0vpE++Prh/vj8eFtsWxaX8hkMrQN0tcO2Hpen+ZQLqtMBWlMzIGfXVQOna7pz2euCzEfflATy6DT0kQFqDB7RFsAMFnILc6733g6Fd/sulLrlJubz6Rjwse7sOD3U7VuQ0ZBGb7ZdQWFZQ0XSD/6/WHMW30cPx1IAgAUlFWg1PCe6jpiYVz3Iru4HK38PPDdzAEYZLZupSlxkcswpF0QOlRzZ74qYqa7mkYLGtKXO69IGeS2nEvHD/uuWT0u3zAVys9DAZXCFa/c0hV9ovWjw17Kyjo5i/48i8SsYulGR5CXAhO7h6F1YGVxvM6GqWLGC8KJmqOmkySbiKwa2zUUY7uGSo+HdQiCl9JVmqM8KCbQ6txuMUtLhU5ATkl5rRZXOhtx6oF4Z5wcZ8GEzhjaLsik0vDgtvq56WUaHRb/dQ6XM4ux5K4eNT7Xsu36ApC/H72B/9zbq8bjv997Fa//7wwAYP3xFPz59LC6vIUa5RouCt/7+zymDYqW7nR7K13hoahbQUZXFznaBnsiMasYjw9vi/ljO0LZjIs7iuvGSsobvq6DIAjILFIjxNs06DuYmGPy+Pj1PLz6x2lMGxyNzmGVtUiKDW00T6IBwKLGyO2f7ZZSfgd4KuHmIkf8C6Ow9ugNaZRi/fEUjlhQs8cRCyIn4+7mgtdv64r+rf3x8qTO+H7WQKvHubnIpVSWGQXOU+nWFuIfaU6FcjyZTIbhHYMRE+QpbYv0V6FvtJ/0eMOJlFo9l/Gd3q3nLGtjmBODCgA4lZzfIMXI1BWVz1morsDey9nSHWp7K2T/OjsW258fiYUTuzTroAKAFICZrzVIyy+r935bujkBA9/eijij+iqnk/ORWaiGXAZMNUzLir+QiR/2X8OEj3fhvi/3odQQUIj/q6wEjV5mwYZxHSFxVAYA7uobiUdvaitl//rfiRRUNLEpfkT1iYEFkRO6p38UfpszBI8Pbyflw7dGvFOXXlhW5THOTJy7H6BiYNFUrZgxAEvv7glAf0GeX4upIKVGd7NX7r1a7bHW1mE0xHSTYrXpRe+slYfwYVwCAPtHzIK8lGhjFJA1Z9ZGLL7eeQWDl2zF7Z/trtfXWr5DX0Pi9fWnAehHMJ5adRQA0CHEG0PaB1mccyAxB8eu5wIAig3Bj7XRKONgY9Ht3RDgqUDnMG9Mj22NoVae1zjY2FKLYJnIWXEqFFEzFuqjxNlU4LX1p7HrxZsd3Zx6J04H83Lnr7Kmyk+lwL39o/DB5gRkFKqRmF2M3iq/as8xXkxrbbStTKPFl/FX8NGWC1bPLyjT1HtWJeM1AUFeSmQVqbH/in5KTQDX+NRa5YhFZWDx5c4rAIAL6UUor9BVe7OkLjSGNWa3fbYbV7NLAAD/Gt8JnlVMXxPTxoptNE/7DQBKt8pzJ/duhelD2lTbBrHCOgBcM7SBqDniiAVRM9Y+RL/Q+3pOKcormt/we3V/+KlpEe/IX8uuOY9/hlE9gEwrGXv+/cfpKoMKoGEyoYlBbJCXAj8+OhBtgytHGII4Fa/WxPoP4lSo6zklJoFkQyzqFqcenU4uAKDP4DW2ayiCzKqFK1z0l0TpBWU4cT0PlzKKAFifChXu446BbQJwc+cQ+HjU/PvHRS7D7OH6BAfGKWqJmhsGFkTN2EsTOktfZxc3r3UW5RU6VBjuRNZ14Sw1HjEda03rfY5cy5VSuAL6dTTG050EQcCfZms1zOe7N0S1efGC11Ppis5hPvjqof7SvuYYtDcUlZvpVKjLmUUm++urcJ5xn1RoBZNK3/9nSAUb6uMO43qaXSL0C7e/2ZWIO5btkdZwWfv9IpfL8OsTsVgxY0CtCyOKo2hnUgqgbaaZ+ohqdZuvT58+tf7BOXr0qF0NIqL64+oiR5iPO9IKypBZqEa4r0fNJzkJ48Wf1u4oUtMirkPIriErjhg03Nw5BPEXMqHVCSbnFKoroDZcNJ5/awLOpBQgKsADmYVqzPnxKJJyShp0xEIcHWsf4oWBMQE4mJiDEZ2C6/31miuVsnIqlE4n4HqO6bSg+soW9eJvJ6Svyyq00mcGADoZUiL7erjh4dg20jqebhE+OHE9z2TEDKi/EVGxOODBxBw88PV+/DI7tl6el6gpqdVPy+TJk6Wvy8rK8Pnnn6Nr166IjdX/UOzfvx9nzpzBk08+2SCNJKK6C/ZWSoFFcyKmglS4yuHmwsHXpi5QqsJd/edQvNAc3SUEp5PzkVGoRlZhZWCRZfgceyld4e7mIlWdD/F2R49WvkjKKWmgEQut9LqiHx8ZhGNJuejfJqDeX6+5Mr4JUFahxY6ETJP99TVicS61UPpaoxWQklcqPTYOFOaN7oCVe6/Cx90VvaP8sMpQo8RYfY2IGqdiPpCYA51OgFxeu5u2RM6iVoHF66+/Ln396KOP4plnnsFbb71lccz169frt3VEZDd/wwXdI98fxh9zh6J3lJ9jG1RPSg0jFhytcA7iAuec4uov+q8ZAovoABXaBHkio1CNQ9dyIVZyyTIqQmbOR6o2X//z9CunQlV+3hSu8iZdyK4pcjdKp/v8ryekQp+i+lpjoXQzvdkgLphWKVxMLub9PRXY9eIo6AQB6VVM06uv3zHdInzwyqQueHvjOQD6FLW+KrcaziJyLjbf5luzZg0efvhhi+0PPvggfv/993ppFBHVnzKjqQVVVZh1RuIdZC7cdg7iVKjErCKrKWIBmEyNaR3giVt7hgMA/jGqQ7D3chYAWC34KFaxboj0ykVGayyo7uRyGVwMF/Z/n06TtkcF6Kdp1ldgYT6lKjFLnzTAWv9FBajQOtDTJCWssfr6HSOTyfDY8LZSNqqcEhbLo+bH5sDCw8MDe/bssdi+Z88euLvXb3o/IrLfzKFtpK9T80urPtDJlFRTvIqaHrGI3OXMYtz2f7tNFtOK8ko10lz4cD93dIvwBQBpMfeF9EJ8vOUiAOvTU8RpUeuOJuPE9TysOpBUb5/5gjL9SIu3O+8w26t9sJf0tcJFjvVzh6JtkH5bfU2FEmuhuBtGLq4aspFVlWIWgEmF7mCjjFH1nRxCLJaXy8CCmiGbw/Bnn30Wc+bMwdGjRzFwoL7i74EDB7BixQq8+uqr9d5AIrLPxB7hWPXYIDzw9QGk5Tt/msP8Ug0OXMmGmFRFxTvITiHSvzJxwPm0QmQWqhHiY3ozSszC4+PuCjcXOUIMF3cZhWoIArD2WGU2qJs6WBYhG9MlBN1b+eB0cgHuWLZHOu6HRwbZ3X6x6J4/p67Y7YdHBmLnxSwMaRcIlcIFfiqFtHblhd9OQunmgtt7RVR5flaRGtNXHMRdfSPxyLAYq8eUGqp4RweocCG9CP81jNZWN+JkHEBE+XtI69Lqu66Gv6cbkvNKkcfAgpohm/8iL1iwAG3btsUnn3yCH3/8EQDQpUsXfPfdd7j33nvrvYFEZL9WfvqLutT8MgiCUOssb03NlcwiTP/uIK7nVN6F9nDjwm1nEO7rgVWPDsID3xwAoB+5MA8sxDu4gYZpTuJdY41WwI5UGeCn3z97eFs8dlNbi9eQyWSYNTQG83+tzAhUX8XIxItAf1Z5t1uIjzvu7hdpss145PGl305WG1gs33EZZ1IKcCblbJWBhZg1rnsrX1xIr0xpW9upbIPaBmJctzCrU+7sJX6GcmtYb0TkjGwKLCoqKvDOO+9g1qxZdQ4iEhMTsWvXLly7dg0lJSUIDg5Gnz59EBsby6lURA0k1HABV6rR4tOtl/DM6PYWwUWZRovScq202Lspevfv8yZBBQAcuprroNaQrYa0D8LITsHYkZCJxKxixLYzXficXSRevOtHBdyNqhv/cc0FIxT6C7F2IV5VBse39AzHe5vOSwtxbakXUKbRpyUV12oYyzNkmuJi24YRYPR7p6qpR4VlGmh1gkk6WGs3SnQ6AWUa/ZS6BRM6Y+3RZGmfec0Tc2/c1hXrjqfg8ZvaNtjvQnEqlLjug6g5selWn6urK5YuXYqKCtvnQP70008YOHAg2rVrh5deegl//PEHdu3ahW+++QYTJkxAaGgonnzySVy71nwWlxI1Fe5uLugVqZ+v/tGWCxYX47nF5Zj4yS4MfGcLfjtywxFNrJG6Qovdl/QLdwfGVKb3fGhwa0c1ieqgTaC+YnWSWf0CrU7Av/84DcD0ItOYWIU7oJpRA6WrC1Y/HotFt3cDABSX1+7vlSAImLJ8L27+YIdJJWhRrjQVqukG3s7s4SFtEGvIsGUtZEzJK8XYD3dixPs7TGpf5Fipi1JWUblw28vdFa0DK9O81jRiMWNoDNbPHdqgN1j6GDLzLdtxCTsSMqo/mMjJ2DyHYPTo0YiPj7fpnD59+uDTTz/FjBkzcO3aNaSmpuLIkSPYvXs3zp49i4KCAqxfvx46nQ79+/fHmjVrbG0WEdXgS6NKwW9uOGNSYG7tsWQkZhVDoxXw6h+nrV5YOdqFtCKUlGsR4KnAL48PRuKSSfjhkYGYP66jo5tGNhBTwhapTaeBrNx7VfrcGQ8yGE+ZuZihn9JS00VfTJAnRncJAVD7gmsZhWqcSSlAdnE5fjaqZXDgSjbu/2o/TlzPAwD4ccSiQbTy88AXD/UDoC+iuCMhw6TQ4Qf/JCCtoAz5pRocN/QFAPRbvAWnk/NNnsu4z91dXUyCwVDv+p/aZKsZQ9pgUo8wCAIQZ5TxjKg5sHmNxcSJE7FgwQKcOnUK/fr1g6enp8n+22+/3eKcd999F+PHj6/yOZVKJUaOHImRI0fi7bffxtWrV21tFhHVIMzXHR/f1xvP/nIcp5ML8MjKw1j12CDIZDLcyK28A1iq0eLUjXzkl2rwn7gELJ/WD91b+Tqw5XppBfqF51H+HtLUh5s6sOKxs/Ey1IEQ0wWL9hnSyAIwKXC3eHJ3aRRNo9VHHFWNaBhTGVKEllfooNUJUorTqpxNKZC+PnGj8kL1vq/2mxzHxdsNx9fDDd5KVxSqKzDju0O4pUc4lk3ri8IyDTacSK3yvF0Xs0x+RxlnhJLLZSaflzBfx0+5lstl6N86ABtPpUlT7IiaC5sDC7G69ocffmixTyaTQau1vDtUXVBhLjAwEIGBLDhE1BBu6RmO9IIyvLvpPPZdycaOC5kY1SkEGWaFofYnZuPL+CsAgO/3XsX79/Sy+7XVFVokZhUjp7gcsW0DbV5Anm4ILMwX/JJzEaeimKcVNZ7SsnBSZ+lrcRqf8cV+dVOhRMaLgUvKK2pME3surTKwqG7ELtiLn7+GFOilQKHhs/HXqVQsA3Doag7Kq6h9AlQWy5Qea8RU1PrPmvGIRbivB5oCceQrv4SBBTUvNk+F0ul0Vf6zFlSYO3r0KE6dOiU9Xr9+PSZPnoyXX34Z5eVMvUbUkNxc5Jg9oh0eNWRS+SjuAgRBkEYDfNz1f4jFoAIAar/0tXrzfz2BCR/vwgNfH8DOi1k1n2BGDCyqKmJFzkFcPGteCE0MLH6dHYt+rQNM9nUK9TJ57O1e8z0xpascYuxaWovpUMYXeFUFFjd1COLi7QbmY7ZwXhAE7LucDQDoFOptsu+xm/S/x3LNLs7FqVAehsX/xqNM4X5NIzAUg528Ul73UPPS6HkaZ8+ejQsXLgAArly5gqlTp0KlUmHNmjV48cUXG7s5RC3SEyPaQaVwwckb+Rj0zlYcuaZfzN0z0s/i2Ppab2E8l/hCWqHN54uBRRhHLJyaWMW4qsDC2jSn+WPaw8dNH+I+NLg15DVMawL0I+jia9VmnYXxIu8buaUo02gtivjd0z+qxuch++jMvue/HbmBr3clAgBu6xUubVcpXBBmGH0wLzQnTqUTs0vdbFhv4yKXITpAhaZADFDF+ijlFTqUaWq3HoioKatTZani4mLEx8cjKSnJYpThmWeeqfbcCxcuoHfv3gCANWvWYPjw4Vi1ahX27NmDqVOn4uOPP65Lk4jIBoFeSkwf0gbLd1w2Sd3YNthTyrwkSi+on8Ai2EuJ5Dx9qthsK5lcaiK207g6Ljkfa1OhNFodCsr0j60FFoFeSszrrkWOb0c8PqJ9rV/LQ+GCInVFrTJDlZit+Xjp95O4pUe4yTYGtQ3PfMbTC7+dlL6+pWcEPvhHf2NSpXCRRiLEi3OtTkB2kRoPrzgIAOgS7gMAGNIuCL/PGYJidUWD1KWoCz+PyqlQ13NKcPtnu1FcrsXmZ4cjJsjT4vjCMg3ySjTw91TUmDKXyJFs/nQeO3YMkyZNQklJCYqLixEQEICsrCyoVCqEhITUGFgIggCdTv+bY8uWLbj11lsBAFFRUcjKsn16BBHVzeM3tcXmM2mAoE/JOL5bGGKCPKUKtaK0/NIqnsE2xhd3mYVVBytVFfDLreaONjmPyqlQlRfy4h1nuQxWa0gAQJA78PDo9nBzq/1UJHGdxfQVh/Dn00OrnV9vPqqx/niKRSpTBhYNT1dF3ZG7+rZCG6O0sW4u8spCcyXl2HspC498f1haXwEA47uFSl/3a+3fQC2uG7GWRaG6AvuuZEvTuQ4l5kiBRWm5Fr8cSkL7EG88+K2+sGSQlxIbnh7WJBahE1ljc2Dx3HPP4bbbbsMXX3wBX19f7N+/H25ubnjwwQcxb968Gs/v378/Fi9ejDFjxiA+Ph7Lly8HoC+cFxoaWsPZRFRf/D0ViHtuhEm2HEEQ8OYd3fDa+jPSttwSDf44low7ekfYVbHb+I5wVdOryjRa3PLpLnQK88bn0/qZ7MsRKx8zsHBqnoasUMYjFmJdAj+VosbsTbaoMGSRyipS4+W1p7BixoAqP8PWRjWSc02D6hCu72lw0YEqJKTrp0oGeyuRXaTGT48OloopDmsfhN2XsjCkXZC0ADqvRIN1x5JNgoqpA6IwqXu45Qs0ET7urpDL9KmVjTOSpeSX4mxKAT7ZegEarYBt503rXGQVqbH5TBqmD2nTyC0mqh2bA4vjx4/jyy+/hFwuh4uLC9RqNdq2bYulS5di+vTpuOuuu6o9/+OPP8a0adPwxx9/4JVXXkH79vph7d9++w1Dhgyp27sgojoxv4iTyWR4OLYNftqfJP1xB4BnfzmOjMIyPD68ncnxN3JL8PmOyxjbNRSjOoVU+TrlFTqTrC5VBRa7LmbhcmYxLmcWW4xc5BWLBcq4eNaZSSMW5RUQBAFnUwswZfk+APqLxvpUoav8zG1PyMTiv87htl4RKFFXYIjZa4kjFmJKZgC4YqiMvHBiZ7QN9jKpBE4NY9Ht3aDVCZg5tA0Gtw1EQakGgUbTl1bOHIBj1/PQLcJHymaXnFeKNYaUxO9N6YExXUJNzmmKXF3k6BPtjyPXcvHLoevS9tS8Mny58zI2n6m6voV5RjWipsTmwMLNzQ1yuX7Nd0hICJKSktClSxf4+vri+vXrNZwN9OzZ0yQrlOj999+Hiwt/aRM1BT89NgiHr+YgJa8Mb244CwB4Z+N5i8Di54NJWHVA/+/Ea+OqzJhTYnY3uKqpUMbHFaor4GNIEVpeoZNSULLysXMT11gIgv5ifsHvlX8PnhzVrqrT6uTZMR3x88EktA3yxB/HU/Dt7kR8uzsRMhmw56WbEeFXOTVKXEwe6KVAKz8PaT2Qi1yGR4bFwNWl0XOdtEgRfh5YMWOA9Ng8QHB1kWNAG33WMH+V5bSp8d3CpGlGTd3E7mE4ci3XZKQlJb8U2iqmg43qFIztCZkWv0+JmhKbA4s+ffrg0KFD6NChA0aMGIHXXnsNWVlZ+OGHH9C9e/daP8/hw4dx7tw5AECXLl3Qv3//Gs4gosYS5KXEhO7h2HQ6zWS7+ShCal6Z9HVWsbrKwKLYbP56dnE5dDrBIrtPdlHlnPbc4nIpsMgzmoNvno6SnItK4QJ3NznKNDpkFalxylA1eUAbf3QO86nX17p/YDTuHxgNQRDQOdwHX8RfRl6JBoIAXM4sMgksxBELlcIVPSN9pcAiJsiTQUUTZZ52+PHhbZ0mqAD0QdDiv86ZbEvOLbX6O25Up2B0CffB9oRMi+KSRE2Jzb8t33nnHYSH6+ctvv322/D398ecOXOQmZmJr776qsbzb9y4gZtuugkDBw7EvHnzMG/ePAwcOBDDhg3DjRs3bH8HRNRgzBcIxp1Nx+nkfCzfcRnlFTqTjFLmWXWMlRjuBnsb7lZrdYLVirOpRgvFjRfOigsbfT3c6nUOPjU+mUwmXdBfzymV+vOzB/o26Gs+MaIdDr8yBn2i/QBYrp8Q7wJ7Kl0wtmvler+FEzuDmibzGxMLJjhXX0UFqNDDqGI4ACTllCAlzzJhxrtTekqjfbWpy0LkKDaPWBiPLISEhGDTpk02nf/oo49Co9Hg3Llz6NSpEwAgISEBM2fOxKOPPmrz8xFRwzEvRvf4D0ekryP9PZBRWDliUd28X3Gfj4cbXFxkyCvR4N2/z8HDzQVpBWV4ZnQHdIvwRUp+5fMZBxbi15wG1Ty08vPAlcxivLXhLLQ6ATJZ42T7cnWRo2u4D44l5WHFnkRMHRgt7RPvAnsqXHFbrwhcyihC/zb+uLkzk4o4A3c3ea3qmzQ1E3uESaN2AFChE0xu2ADA73OGINTHXcpyVpv0yUSOYnNgsWLFCowaNQoxMTF1esH4+Hjs3btXCioAoFOnTvi///s/3HTTTXV6TiJqGMHVLIC8klls8gfQvOCZMXGaiZfSFQpXOfJKNPj1cOUI5Ynr+dj/8mik5lkfsRAXewd5N+0FmVQ7kf76EQsxQUCASgG3RppuJI6WXEgvwtmUAnSN8IFOJ0jz3FUKF7i5yPGik939bulc7MhY50izhsZAUyFgQBt/LPn7vEmQIYowVAsXA4vSci1S80uxdFMCukX44N4BUdK0USJHszmwWLJkCR577DG0atUKI0aMwIgRIzBy5Egpu1NNoqKioNFYToHQarWIiIiwtTlE1IBcXeSIe244NFoBXcK9kZpfhodXHMSljCLcyC2RClMB1d9FE4MOldIFCUmWVbfTDFW1U41GLC5mFKFCq4NMJpMWewczsGgW2gZ5mTyuqnZFQxDTlgLAlawidI3wwZEkfeV5pasc3rxAc0pyJw0s3N1cMG9MBwDA1IFRSI0rRVZROUK8lcgoVOPufpFS/RWVojKj2vrjKVh3LBnrjiXjTEoBPrqvt6PeApEJm28RXbx4EUlJSViyZAlUKhU++OADdOrUCZGRkXjwwQdrPP/999/H008/jcOHD0vbDh8+jHnz5uGDDz6wtTlE1MA6hHqja4SPNDd+emxrAJDSO4qqW1AoVtr283DDXX1aAQCm9I00KTimrtCajID8sO8a7vlyH2KXbMWVrCIA1Y+gkPO4f1A03prcXVpz07cRi5f1jfaX1lBczynFhfRCTPtaX3zs5s4hULhyobYz6RXlBwC4s28rxzakHkwb1BqH/z0WV9+9BQdeHo34F0bi/bt7SvuNRyzSCypvwlzOLGr0thJVpU514Vu1aoVp06bhzjvvxK5du/Dzzz/jp59+wurVq/Hjjz9We+6MGTNQUlKCQYMGwdVV//IVFRVwdXXFrFmzMGvWLOnYnJycujSPiBpQSBXVh6ubCiX+EQzzdcf8sZ1wU8cg3NZTP0LZ9bXNKNfqMP/XE1KaRW93VxSWVeBYUh4A4Mf9SQA4YtFceCld8dDg1rirTytczChCt4j6zQZVk7aGysbvbTqP9zadl7Y/Prxto7aD7Pft9P7Ydj5D+n3SXMhkMrQO9DTZVjlioTVJ2W0cZBA5ms2BxT///IMdO3Zgx44dOHbsGLp06YIRI0bgt99+w/Dhw2s8/+OPP65LO4moiWgdqJK+HtkpGFqdgF0Xs6qdCpVuKGQV6uOOYG8l7uwTKe0bGBOA3Zey8NfJVAD6hb1tgz2x62KWxfNwxKJ58VS6orfhjnNjCvSyXCgeE+SJPtGNN3JC9SPIS4l7+0c5uhmNQqxafymjCAqjNUmZhWpodYJJxryMAn0NogcHt8bgtoEWz0XUUGwOLCZMmIDg4GA8//zz2LhxI/z8/Gw6f/r06ba+JBE1IZ3DfLDkrh5oE+iJ2HaBeGfjOX1gUYsRi1Arox2PD2+L3Zcqg4gP7umFfVeyrQYWxkENUV11Dfe12NYYWamI7CFOhQKAs6kF0tc6AcguViPEu/L361t/ncOGk6nYcDIVV9+9pVHbSS2bzZNJP/zwQwwdOhRLly5Ft27d8MADD+Crr77ChQsXav0cly9fxr///W/cf//9yMjIAAD8/fffOHPmjK3NISIHuH9gtLQI1tMwPL89IbPK46WpUFYCi+Edg7HzhVHY9vwIJC6ZhNh2gRjfzTLF5119WmFgTEB9NJ9auKHtAxFoFkgwsKCmzrigo7n0fNMUtZcyuO6CHMPmwOLZZ5/F2rVrkZWVhU2bNmHIkCHYtGkTunfvjsjIyBrPj4+PR48ePXDgwAGsXbsWRUX6D/+JEyfw+uuv2/4OiMihAgzTSi5lFOF6TonFfkEQkGTYHu5nfX1GdKAKbYO9pKreXcN9MKVvJCb1CMPvc2Lx3pQe+M+9vUyqfhPVlUwmw939TP9eBVmZHkXUlKgUrvj3LV1MtnUO8wYAJGYXm2x3wpIe1EzUKf2FIAg4evQo4uLisHnzZmzfvh06nQ7BwcE1nrtgwQIsXrwYcXFxUCgqf5HffPPN2L9/f12aQ0QONLl35aJJa9lJMgvVKCyrgFymn8deGzKZDP+5txc+n9YP/VoH4L4B0QwqqF6ZF1PjiAU5g/Yhlama/VRuUiX5C2mmabxdjT7fN3Itb/hQ06bR6vD93qtIyna+vrM5sLjtttsQGBiIgQMH4qeffkLHjh3x/fffIysrC8eOHavx/FOnTuHOO++02B4SEoKsLMs51UTUtHm7u+HmziEATOtQiMQh+egAFZSuLhb7iRwhwtd09My8tgZRUyTWtACACF8PdAzVj1gcTMyBIOiz6pVptDhxo7LQ3uRle/DHsWSmpXUib/91Dq//7wxm/3jE0U2xmc2Ltzt37ozZs2fjpptugq+v5QK4mvj5+SE1NdWicvexY8fQqpXz56EmaonCDBdpxpWzRVey9EP07YJ54UZNx70DonD4Wi7CfT3Qr7U/RnWqecSdyNEijKaT5pdqMLhtIGQy4ODVHGw+k44J3cOw+mCSyTlZReV49pfjaB/ihS3zRzR2k6kOVu69CgA4Z7RI31nYPGLx/vvv49Zbb4Wvry/KymzPnTx16lS89NJLSEtLg0wmg06nw549e/Cvf/0LDz/8sM3PR0SOJ979PXg1B+uO3UCZprJYnlj0LszX+voKIkdQurrgk6l9sGBiZ4ztGgpXFxbGo6bP290NPSP1N3Vv7RWOLuE+mD28HQBg6abzKCzT4HSK9YvRSxlFUFdUXci0IeUUl+OTLReRbOXmEzUvNv8m1el0eOutt9CqVSt4eXnhypUrAIBXX30V3377bY3nv/POO+jcuTOioqJQVFSErl27Yvjw4RgyZAj+/e9/2/4OiMjhxOH5/Vdy8NwvJ9DltU1IMMz5zSzU34AwToVIRER18/Njg7F4cnfMGaEPKGYPb4sgLwWuZBVj/fEU5JVoAABzR7XDgZdH48uH+knnJuc65sL+hTUn8NGWC3jiB+eb2tPYzFO370jIkKa5OQObA4vFixdj5cqVWLp0qcni6+7du+Obb76p8XyFQoGvv/4aV65cwYYNG/Djjz/i/Pnz+OGHH+DiwvnXRM7IPNuTIAC7LurTz4oVYlk1m4jIfp5KVzw4uDX8VPprMH9PBaYOiAYAnLqRL6X37h3lj1Afd4zvFoZOhrUYSzcloKiamkMNZet5fWmBU8n5NRxJ2UXlJo9nfHcIx687z/fN5sDiv//9L7766itMmzbNJBDo1asXzp8/X+P5b775JkpKShAVFYVJkybh3nvvRYcOHVBaWoo333zT1uYQURMQ4WuZX12cAiX+H8LAgoioQXRvpZ8edSo5H2lW6gaJxUU3nUnDT/uvNVq7tp5Lxx2f7W6012sOMovUFtsuOdHCe5sDi+TkZLRv395iu06ng0ajqfH8RYsWSbUrjJWUlGDRokW2NoeImgBr6yc2nU7DvNXHcNKQnYQjFkREDUNMO3s2tUAaJTb+vfzEyHbS142ZHWr1oesmGaqoZlmGwKJXlJ+UcTGjsLy6U5oUmwOLrl27YteuXRbbf/vtN/Tp06fG8wVBsJqP/sSJEwgIYFVdImfk7mY5jTEppwTrj6cAAMJ93dEhlFmhiIgaQqiPO3pH+UmPb+4cYnIzp2+0P964rSsAIKe45pvA9kjMKsaAt7fghTUnrBZNbbPgL6zYndigbXBWOxIy8Mq60wCAYC8Fuob7AKicUuwMbE43+9prr2H69OlITk6GTqfD2rVrkZCQgP/+97/YsGFDlef5+/tDJpNBJpOhY8eOJsGFVqtFUVERnnjiibq9CyJyuCAvBbLM5oZ6u+srxY7sFAKVwuZfN0REVEvPje2IpZvOo3OYD167tavF/gg//ZRVMaFGQ/nvvqvILFRjzZEb0rbHborB17sqg4k3N5zFrGEx1k5vseIvZGLGd4ekxzFBngj10QeHGYVqwM9BDbORzX/p77jjDvz5559488034enpiddeew19+/bFn3/+ibFjx1Z53scffwxBEDBr1iwsWrTIpAaGQqFAmzZtEBsbW7d3QUQO9+vsWDz98zEMignEij36PyAzh7TBfYZFhURE1HBGdAzGiI5V12MJMay5OHEjH1lFavgq65ZiObe4HK/8cQr39I/CqE4h0vasIjUe++9hHEvKszhnzsj2UClc8cnWiwAAF7nlzJWWSqsT8FHcBXy2/ZK07ZFhMXh+XCfsSNAnQWnWgQUA3HTTTYiLi7PYfvjwYfTv39/qOdOnTwcAxMTEYOjQoXB1tf/u5c6dO/H+++/jyJEjSE1Nxbp16zB58mRp/4wZM/D999+bnDN+/Hhs2rTJ7tcmIlNtg73w1zM3AQCGtg/E2ZQCzOQdKSKiJsF4atS3uxMxf3S7ao6u2vv/JGDjqTRsPJWGq+/eYvKcYlAxpksIkvPKpAJv/io3PDe2I0Z2Csadn++Fn4ebxfOK9Y+sTa1tzk7cyDMJKn5+bDBi2wUCAAK99Jm/xBTCzsDmcLWoqAilpaZ5kI8fP47bbrsNgwYNqvF8b29vnDt3Tnq8fv16TJ48GS+//DLKy21bnFJcXIxevXph2bJlVR4zYcIEpKamSv9+/vlnm16DiGw3uksonh7dAV5KTn8iImoKQo0Ci0sZdV/AnZRduW5i18VMzP7hMK7nlGDv5WwA+voZXz7UH1891A+R/h6Y3DtCmv4uTsfKLSmHTldZm0EQBDz4zQHELtmK/FLnuYiuD6l5plPTjNfKeBiCrFKNYwob1kWt/+pfv34d9957Lw4ePAgXFxc89dRTWLx4MZ544gn88ssvuPPOO7F3794an2f27NlYsGABevTogStXruC+++7DXXfdhTVr1qCkpAQff/xxrRs/ceJETJw4sdpjlEolwsLCav2carUaanXlIpmCAn20rdFoapX1qr6Jr+mI1ybbsK+cC/vLebCvnAf7qmn719gO+CDuIhQusjr3lXGxttf+OI3E7BJsPpMubbuvXyvotBUI83bD1meHQS6vfC1PN32AoROAnKJS+Hq4oVhdgU+2Xcbha7kAgBNJ2YhtG2jX+3QmybnF0tcvT+wEV5kOGo0OAKCQ67/XpeX6wMJRP1e2vG6tA4sXXngBZWVl+OSTT7B27Vp88skn2LVrFwYNGoTLly8jMjKyVs9z4cIF9O7dGwCwZs0ajBgxAqtWrcKePXswdepUmwKL2tixYwdCQkLg7++Pm2++GYsXL0ZgYNUf2CVLllhNe/vPP/9ApVLVa9tsYW3qGTVN7Cvnwv5yHuwr58G+apqup8sAuODajRTExekXV9vaV1lZcogTXhKzTbM+9Q3U4diebThWzfkKuQvKdTKs3RiHUA8gLlmGDUmV05827zqI3PONU2n6RjGg1gLtfBrl5azaf03//RwZrkNo3hls3HhG2penBgBXFJfrL+wd9XNVUmKZ3asqtQ4sdu7cibVr12Lw4MG49957ERYWhmnTpuHZZ5+1qXGCIECn00diW7Zswa233goAiIqKQlZWlk3PVZMJEybgrrvuQkxMDC5fvoyXX34ZEydOxL59+6qs8r1w4ULMnz9felxQUICoqCiMGzcOPj6N/8nTaDSIi4vD2LFj4eZmOSeRmg72lXNhfzkP9pXzYF81bRUnUvHLlVPw8Q/C2LG96tRXq9MPA/k5Vvc9e/uAGkcb3ju7Eyn5ZXjnuCt+mz0IFQVXAVSOeARGd8SkUXVb/2GLr3Yl4v19+sXkBxeOhL+hknlj27rmFJCSikE9O2PSsDYm+wpKNXj96HboBBkqdMDE8Y75uRJn79RGrQOL9PR0xMToF2KGhIRApVLVOA3Jmv79+2Px4sUYM2YM4uPjsXz5cgBAYmIiQkNDbX6+6kydOlX6ukePHujZsyfatWuHHTt2YPTo0VbPUSqVUCotC3m5ubk59Jeko1+fao995VzYX86DfeU82FdNk5eH/uK5tEIn9Y8tfaXTCdh3xTSoCPRUQCcI8HBzweB2IXBzrX75bvdWvkjJ168r+HDLJYtU5Kn56gb97JRX6DDsvW36TEsGKQUahPh6NthrGhMEAekFaoT6KCGTyZBpSNMe4a+yeN8+8sqb4OU6x/1c2fKaNq2slMvlJl8rFLZHdx9//DGmTZuGP/74A6+88opUxfu3337DkCFDbH4+W7Rt2xZBQUG4dOlSlYEFERERUXOkUhgWA5fXbTHwH8eTTR73ifbD25N7IMhbAVe5HIoaggoA+GRqH+y8mInZPxzBwcQcdA73BgCM6xqKf86mIyW/tIZnsM/FjEKToAIA0vJLAaNF0/WpQquDi1xfx61YXYHnfz2BTWfS8NDg1riRW4J9V/SL3kO83S3OdXORw1UuQ4VOQB27rNHVOrAQBMGksF1RURH69OljEmwAQE6O9eExUc+ePXHq1CmL7e+//36V05Pqy40bN5CdnY3w8PAGfR0iIiKipsaeLEPLtl/C+5sTpMef3t8Ht/YIh9zGmhQeCheM6xqKQE8FsovLcTpZP82mS7gP/jmbjoyChq0ynZxbGbhM6BaGTWfSkJpfu6KBR5Ny8cO+a1g4sbNUF6Q6V7OKMeGTnYgOUGHuqPb4Yd81aZH6D/uvmRwrFsMz56FwQWFZBdS6WjXR4WodWHz33XcN2Q64u9fcQeaKiopw6VJl7t/ExEQcP34cAQEBCAgIwKJFizBlyhSEhYXh8uXLePHFF9G+fXuMHz++PptORERE1OR5GEYsSgy3vyt0wLXsErQP863ynLT8MugEwSSoeGF8J9zeK6LO7ZDJZOgV5Ydt5zOkbV3C9etY0wvsrwxeUl5hMsWqsEwDlcIVLnIZbhgCi0k9whDuq09/eyG9EIIgSDfPq3LX5/rsp0XqCnz9sPW6bca2ns9AmUaHC+lFmLf6OABAJgNk0GfGMlZVoKIyBBbNbsRCLHDXlBw+fBijRo2SHouLrqdPn47ly5fj5MmT+P7775GXl4eIiAiMGzcOb731ltU1FERERETNmXixXWa4Sv0mQY5zB3bj19mxGBgTYHH80aRc3PflPni7m86xD63F3fqavDShMzIL1TiVnA8A6GKYElVQVmERGNSWTifgiR+PYNv5DKx5IhZ9ov3x7z9O4cf9SQCAZ8d0kOpkRPqrEOWvDyx+Pngdvh4KLJjYuVavc/x6Xq2OK68wHWYY2j4QDw5qjTFdQzF86XaTkZKq6j7pvw9qlDe3EYumaOTIkSb5lM1t3ry5EVtDRERE1HSJU6FKNFoIgoBzefrp7N/vu2oRWCz5+xy+jL8CAMgpNi1gXNW0HVt0CvPG97MGYtxH8fB2d0OUf2VK/+6vb8b+haNrNd3I2JGkXPxzVp9h6si1XPSM9MPqg9el/R9vuQhvd/2lb4SvO+7sG4lX1+vTux5IzLZ4vnc2nsPfp1Px/NhOmNynlbRdo636Kn/zmTR4KlzRM8oXWUX6aV2PDovBv8Z3Mqkqvn7uUAx8ZysA/ShGVcQ+yyi1bcqZozh1YEFEREREtSNOhdLqBGi0lTdmdWbzctQVWvy4z3QNgDFrC43rIsBTga3Pj4TCRQ65XIYOIV64mFEEnQDsT8yxebpVSl7l+omsonJkF6lRYfbeCssqAACBXkp4KV3xwyMD8dC3B5FptqC7vEKHr3bqA6u3NpzFhO6VxZbzSjRIySuVKokD+u/ht7sT8fbGcxbtCvFRmgQV+m3u2PzscMxbfcwkaDGXlKOvIbH6igueLlSjVUDTzrZW8/J9IiIiInJ6HkYXt7kllaMQOrPZH0eu5aK4XAtPhQvc3SwvFVsH1l/BYF8PNyngee/untL2jDqstTBen5FZqEaa4XGojxKvTOpicmygpz6zaZtAfZrZjEK1ySyYpJzKitjZxeWY/+txk/NXH0ySvq7Q6nD/1/utBhX617I+wtMpzBubnh2OJ0ZUXbdDHGHxcRMQ4t30p/I3amBx9uxZPPnkk+jTpw/Cw8MRHh6OPn364Mknn8TZs2cbsylERERELYrCVQ6lISXsubRCaXux2nRl8LlU/b7hHYPhpTS9Q/7r7FiLu+/1pW+0Px4Zpq+ZZj6CUBvGaxYyi9RIMzwO83FHt1amRY4DvPSBRbDhYr28QietvwCASxnFJsdvPJVm8vh0SgHe/PMszqcV4NDVXBxIrDorqhgc1MWSu3pgfNcQPN/DOVZvN9pUqL///huTJ09G3759cccdd0jF8NLT0xEXF4e+ffti/fr1zNhERERE1ED6RPth/5UcPPbDMWlbYlYxdDoBcrkM2UVqXM4sAgC0C/bC36crL6iXPdDX6iLv+iTelTevNVGTlLxSfLfnqvT45I08pHcJAaBfbN4t3DTzVYCh0ra7mwt8PdyQX6rBd3uu4rmxHQEApw2Lysd1DUWwtxKnk/PRIdQb7UO88O7f56WMVpcyi3BnH/2UrR6tfLHmiVgUllVg+Y7LWLEnETIZ0Cfa38bvQqWRnUIwtK0/Nm5MqfNzNCabAwutVouVK1di69atyMjIgE5nuoBl27ZtVs9bsGABXnrpJbz55psW+9544w288cYbeOGFFxhYEBERETWQ4R2Dsd+senZyXinavrzR4ti2wabVqG/p2fB1wEIMC8ONpzVlF6nxn7gLmDogCj0j/ayeJ9aHEOWVaKSF2a0DVfBVucFb6YpCtX6Nhb9nZZFnNxf9wug/T6TgubEdkVWkxk8H9GtMJvUIN1kDsd0oRS4AHEvKRWzbQABAh1AvuLu5wN3NBa/d1hULJnaGThAabISnKbJ5KtS8efMwb948aLVadO/eHb169TL5V5ULFy5g2rRpVe6///77cfHiRVubQ0RERES1NHVANAIMF9UTInW4p1/VC4djgjzxn3t6IdRHiT+fGtYo7RMXhhtPhXprw1msOpCEycv2VHme2qjo34RuYSb7RnXSj1x0DPOWtrm5VF4CL7lLv7ajwLCwe+Weq8gt0cBb6Ypx3UJNnivYbJ1DYVkF3tt0HoB+ypUxhau8RQUVQB1GLFavXo1ff/0VkyZNsum8Nm3a4K+//kKnTp2s7v/rr7/QunVrW5tDRERERLUU4KnA73OGICW3CFln96P74Bj8eTIVZRrLFKoRfh7oE+2PKf0iG619/oYpSrkllesdxPUL5kXljIlF/yb1CMPn0/phzo9H8PfpNAR4KjDAMH3rP/f0wl3L96Jfa9OpSb0i9dOkcorV0OoEpOTrs0vNHNrGop5GdQuozYOOlsjmwEKhUKB9+/Y2v9Cbb76JBx54ADt27MCYMWNM1lhs3boVmzZtwqpVq2x+XiIiIiKqvZggT0T6KrDxLBAdoMLW50dC5eaCixlFuPfLfQAAF7kMQV6Nf6Hs76lfLJ5XUi5VwzYvNGeNGFiIgcArt3RBpL8HHo5tI41OtAnyxN4FN0sL2EUBngrIZPrAJae4HHmGoKaVvwfMBXkpMcgQqJgv2Barh7dkNgcWzz//PD755BN89tlnNZY+N3bPPfegVatW+PTTT/Gf//wHaWn6xUBhYWGIjY3Fjh07EBsba2tziIiIiMgOrQz1GIzvuAd7KeEib/yibH4e+hGLCp2A4nItTl7PQ7ZRgb4idYXVKtWl5fppTCpD6tpIfxVeuaWrxXHWpia5usjhrXRFQVkFPtt2UVqY7euhsDhWLpfhl9n669XZPxzG8et5+PmxwUgrKJMCjpbM5sBi9+7d2L59O/7++29069YNbm6macjWrl1b5blDhgzBkCFDbG8lERERETWoIK/KC2lfD8cUYvNQuEDpKoe6Qofc4nJ8v++qyf4dCRm4tadl4bxiw4iFWBPDVuKi7u+NCgP6q6r/Hnz5UH9odQJc5DK0Dfaq0+s2NzYHFn5+frjzzjvtetH8/HyTEQtfX98aziAiIiKihuSldIWLXAatTkC4X/1U164LP5Ub0gvUSMkrxc4LWQCAjqFeuJBehFUHkqwGFuJUKE9F3SopPDosBl/vSjTZZpw5qiqOGNVpymz+7n/33Xd1frFvvvkGH374IRISEgBAmjvXqVMnPP/883jkkUfq/NxEREREVHcymQxv3tEN51IL8ORI29fT1hd/lQLpBWr837ZLKNVo0SZQhbfu6I77vtqP5LxSq+eYT4Wy1Su3dMUL4zsj7mw65q46CkAf4JBtGq1A3vvvv4833ngDzzzzDMaPH2+yePuff/7BvHnzkJubi3/961+N1SQiIiIiMjJtkOMzdPoYpmHtvqQfrXhgUDQiDOtAUvPLpBvTopLyCrunQgH69LCtA1XSYz8rayyoenUKLH777Tf8+uuvSEpKQnl5ucm+o0ePWj3ns88+w3fffYd7773XZHuXLl0wcuRI9OrVCy+88AIDCyIiIqIWbHiHIBw0ZFxqG+SJhwa3gYtcBpkMKK/Q4Z4v9mH5g/0Q7K3EnktZmPbNAencuk6FEnWL8MG0QdHw9XCDwtXmcm8tns3fsU8//RQzZ85EaGgojh07hoEDByIwMBBXrlzBxIkTqzwvIyMDPXr0qHJ/jx49kJWVZWtziIiIiKgZeermDjizaDx+enQQ/vf0MHgoXKBwlcPFMEpx+Fou5v50FAVlGny184rJufaMWAD66WBv39kDL07obNfztFQ2Bxaff/45vvrqK/zf//0fFAoFXnzxRcTFxeGZZ55Bfn5+lecNGDAA7777LioqKiz2abVavPfeexgwYICtzSEiIiKiZsZT6Yqh7YNMUssObR8kfX3wag56LfoHV7OLTc+zc8SC7GPzdz8pKUlKGevh4YHCwkIAwEMPPYTBgwfjs88+s3reZ599hvHjxyMsLAzDhw83WWOxc+dOKBQK/PPPP3V9H0RERETUjH32QB/cyC1FbnE5HvjmAAQBuJZdIu1XusrRPoRpXx3J5hGLsLAw5OTo571FR0dj//79AIDExEQIQtW11nv27IkLFy7grbfegre3N65cuYIrV67A29sbixcvxvnz59G9e/c6vg0iIiIias683d3QJdwHQ9oHYevzI0z2LZ7cHQdfHoMwX8elyaU6jFjcfPPN+N///oc+ffpg5syZeO655/Dbb7/h8OHDuOuuu6o919vbG3PmzMGcOXPq3GAiIiIiatnaBXvhxQmdsHSTvoRB91a+8GV6WIezObD46quvoNPpAABz585FYGAg9u7di9tvvx2zZ8+uc0M0Gg1SU1MRHR1d5+cgIiIiopbh7r6RUmAR7K10cGsIqENgIZfLIZdXzqCaOnUqpk6dandDzp49i759+0Kr1dr9XERERETUvIX4uOON27qisKwCrQx1Lsix6pSgd9euXXjwwQcRGxuL5ORkAMAPP/yA3bt312vjiIiIiIiqMmNoDJ4e3cHRzSADm0csfv/9dzz00EOYNm0ajh07BrVaDQDIz8/HO++8g40bN1o9r2/fvtU+b2mp9RLtRERERETU9NkcWCxevBhffPEFHn74YaxevVraPnToUCxevLjK886ePYupU6ciJibG6v7U1FRcuHDB1uYQEREREVETYHNgkZCQgOHDh1ts9/X1RV5eXpXnde/eHYMGDaoyI9Tx48fx9ddf29ocIiIiIiJqAupUx+LSpUsW23fv3o22bdtWed7QoUORkJBQ5X5vb2+rAQsRERERETV9No9YPPbYY5g3bx5WrFgBmUyGlJQU7Nu3D//617/w6quvVnneJ598Uu3ztmvXDtu3b7e1OURERERE1ATYHFgsWLAAOp0Oo0ePRklJCYYPHw6lUol//etfePrppxuijURERERE1MTZPBVKJpPhlVdeQU5ODk6fPo39+/cjMzMTb731VpXnJCUl2fQaYgpbIiIiIiJyDnWqYwEACoUCXbt2xcCBA+Hl5VXtsQMGDMDs2bNx6NChKo/Jz8/H119/je7du+P333+va7OIiIiIiMgBaj0VatasWbU6bsWKFRbbzp49i7fffhtjx46Fu7s7+vXrh4iICLi7uyM3Nxdnz57FmTNn0LdvXyxduhSTJk2q/TsgIiIiIiKHq3VgsXLlSrRu3Rp9+vSBIAg2vUhgYCA+/PBDvP322/jrr7+we/duXLt2DaWlpQgKCsK0adMwfvx4dO/e3eY3QEREREREjlfrwGLOnDn4+eefkZiYiJkzZ+LBBx9EQECATS/m4eGBu+++G3fffbfNDSUiIiIioqar1mssli1bhtTUVLz44ov4888/ERUVhXvvvRebN2+2eQSDiIiIiIiaF5sWbyuVStx///2Ii4vD2bNn0a1bNzz55JNo06YNioqKGqqNRERERETUxNU5K5RcLodMJoMgCNBqtfXZJiIiIiIicjI2BRZqtRo///wzxo4di44dO+LUqVP47LPPkJSUVGPKWSIiIiIiar5qvXj7ySefxOrVqxEVFYVZs2bh559/RlBQUEO2jYiIiIiInEStA4svvvgC0dHRaNu2LeLj4xEfH2/1uLVr19Zb44iIiIiIyDnUOrB4+OGHIZPJGrItRERERETkpGwqkEdERERERGRNnbNCERERERERiRhYEBERERGR3RhYEBERERGR3RhYEBERERGR3RhYEBERERGR3RhYEBERERGR3RhYEBERERGR3RhYEBERERGR3RhYEBERERGR3Zw6sNi5cyduu+02REREQCaT4Y8//jDZLwgCXnvtNYSHh8PDwwNjxozBxYsXHdNYIiIiIqJmzKkDi+LiYvTq1QvLli2zun/p0qX49NNP8cUXX+DAgQPw9PTE+PHjUVZW1sgtJSIiIiJq3lwd3QB7TJw4ERMnTrS6TxAEfPzxx/j3v/+NO+64AwDw3//+F6Ghofjjjz8wderUxmwqEREREVGz5tSBRXUSExORlpaGMWPGSNt8fX0xaNAg7Nu3r8rAQq1WQ61WS48LCgoAABqNBhqNpmEbbYX4mo54bbIN+8q5sL+cB/vKebCvnAf7ynk4uq9sed1mG1ikpaUBAEJDQ022h4aGSvusWbJkCRYtWmSx/Z9//oFKparfRtogLi7OYa9NtmFfORf2l/NgXzkP9pXzYF85D0f1VUlJSa2PbbaBRV0tXLgQ8+fPlx4XFBQgKioK48aNg4+PT6O3R6PRIC4uDmPHjoWbm1ujvz7VHvvKubC/nAf7ynmwr5wH+8p5OLqvxNk7tdFsA4uwsDAAQHp6OsLDw6Xt6enp6N27d5XnKZVKKJVKi+1ubm4O/cFz9OtT7bGvnAv7y3mwr5wH+8p5sK+ch6P6ypbXdOqsUNWJiYlBWFgYtm7dKm0rKCjAgQMHEBsb68CWERERERE1P049YlFUVIRLly5JjxMTE3H8+HEEBAQgOjoazz77LBYvXowOHTogJiYGr776KiIiIjB58mTHNZqIiIiIqBly6sDi8OHDGDVqlPRYXBsxffp0rFy5Ei+++CKKi4vx+OOPIy8vD8OGDcOmTZvg7u7uqCYTERERETVLTh1YjBw5EoIgVLlfJpPhzTffxJtvvtmIrSIiIiIianma7RoLIiIiIiJqPAwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbgwsiIiIiIjIbs0+sHjjjTcgk8lM/nXu3NnRzSIiIiIialZcHd2AxtCtWzds2bJFeuzq2iLeNhERERFRo2kRV9iurq4ICwtzdDOIiIiIiJqtFhFYXLx4EREREXB3d0dsbCyWLFmC6Ohoq8eq1Wqo1WrpcUFBAQBAo9FAo9E0SnuNia/piNcm27CvnAv7y3mwr5wH+8p5sK+ch6P7ypbXlQmCIDRgWxzu77//RlFRETp16oTU1FQsWrQIycnJOH36NLy9vS2Of+ONN7Bo0SKL7atWrYJKpWqMJhMRERERNQklJSV44IEHkJ+fDx8fn2qPbfaBhbm8vDy0bt0aH374IR555BGL/dZGLKKiopCVlVXjN7MhaDQaxMXFYezYsXBzc2v016faY185F/aX82BfOQ/2lfNgXzkPR/dVQUEBgoKCahVYtIipUMb8/PzQsWNHXLp0yep+pVIJpVJpsd3Nzc2hP3iOfn2qPfaVc2F/OQ/2lfNgXzkP9pXzcFRf2fKazT7drLmioiJcvnwZ4eHhjm4KEREREVGz0ewDi3/961+Ij4/H1atXsXfvXtx5551wcXHB/fff7+imERERERE1G81+KtSNGzdw//33Izs7G8HBwRg2bBj279+P4OBgRzeNiIiIiKjZaPaBxerVqx3dBCIiIiKiZq/ZT4UiIiIiIqKGx8CCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjsxsCCiIiIiIjs1mICi2XLlqFNmzZwd3fHoEGDcPDgQUc3iYiIiIio2WgRgcUvv/yC+fPn4/XXX8fRo0fRq1cvjB8/HhkZGY5uGhERERFRs9AiAosPP/wQjz32GGbOnImuXbviiy++gEqlwooVKxzdNCIiIiKiZsHV0Q1oaOXl5Thy5AgWLlwobZPL5RgzZgz27dtncbxarYZarZYe5+fnAwBycnKg0WgavsFmNBoNSkpKkJ2dDTc3t0Z/fao99pVzYX85D/aV82BfOQ/2lfNwdF8VFhYCAARBqPHYZh9YZGVlQavVIjQ01GR7aGgozp8/b3H8kiVLsGjRIovtMTExDdZGIiIiIqKmrLCwEL6+vtUe0+wDC1stXLgQ8+fPlx7rdDrk5OQgMDAQMpms0dtTUFCAqKgoXL9+HT4+Po3++lR77Cvnwv5yHuwr58G+ch7sK+fh6L4SBAGFhYWIiIio8dhmH1gEBQXBxcUF6enpJtvT09MRFhZmcbxSqYRSqTTZ5ufn15BNrBUfHx/+4DsJ9pVzYX85D/aV82BfOQ/2lfNwZF/VNFIhavaLtxUKBfr164etW7dK23Q6HbZu3YrY2FgHtoyIiIiIqPlo9iMWADB//nxMnz4d/fv3x8CBA/Hxxx+juLgYM2fOdHTTiIiIiIiahRYRWNx3333IzMzEa6+9hrS0NPTu3RubNm2yWNDdFCmVSrz++usW07Oo6WFfORf2l/NgXzkP9pXzYF85D2fqK5lQm9xRRERERERE1Wj2ayyIiIiIiKjhMbAgIiIiIiK7MbAgIiIiIiK7MbAgIiIiIiK7MbBo4pYtW4Y2bdrA3d0dgwYNwsGDBx3dpBZlyZIlGDBgALy9vRESEoLJkycjISHB5JiysjLMnTsXgYGB8PLywpQpUywKMiYlJeGWW26BSqVCSEgIXnjhBVRUVDTmW2lx3n33XchkMjz77LPSNvZV05KcnIwHH3wQgYGB8PDwQI8ePXD48GFpvyAIeO211xAeHg4PDw+MGTMGFy9eNHmOnJwcTJs2DT4+PvDz88MjjzyCoqKixn4rzZpWq8Wrr76KmJgYeHh4oF27dnjrrbdgnPuFfeUYO3fuxG233YaIiAjIZDL88ccfJvvrq19OnjyJm266Ce7u7oiKisLSpUsb+q01O9X1lUajwUsvvYQePXrA09MTERERePjhh5GSkmLyHE7RVwI1WatXrxYUCoWwYsUK4cyZM8Jjjz0m+Pn5Cenp6Y5uWosxfvx44bvvvhNOnz4tHD9+XJg0aZIQHR0tFBUVScc88cQTQlRUlLB161bh8OHDwuDBg4UhQ4ZI+ysqKoTu3bsLY8aMEY4dOyZs3LhRCAoKEhYuXOiIt9QiHDx4UGjTpo3Qs2dPYd68edJ29lXTkZOTI7Ru3VqYMWOGcODAAeHKlSvC5s2bhUuXLknHvPvuu4Kvr6/wxx9/CCdOnBBuv/12ISYmRigtLZWOmTBhgtCrVy9h//79wq5du4T27dsL999/vyPeUrP19ttvC4GBgcKGDRuExMREYc2aNYKXl5fwySefSMewrxxj48aNwiuvvCKsXbtWACCsW7fOZH999Et+fr4QGhoqTJs2TTh9+rTw888/Cx4eHsKXX37ZWG+zWaiur/Ly8oQxY8YIv/zyi3D+/Hlh3759wsCBA4V+/fqZPIcz9BUDiyZs4MCBwty5c6XHWq1WiIiIEJYsWeLAVrVsGRkZAgAhPj5eEAT9LwM3NzdhzZo10jHnzp0TAAj79u0TBEH/y0QulwtpaWnSMcuXLxd8fHwEtVrduG+gBSgsLBQ6dOggxMXFCSNGjJACC/ZV0/LSSy8Jw4YNq3K/TqcTwsLChPfff1/alpeXJyiVSuHnn38WBEEQzp49KwAQDh06JB3z999/CzKZTEhOTm64xrcwt9xyizBr1iyTbXfddZcwbdo0QRDYV02F+cVqffXL559/Lvj7+5v8DnzppZeETp06NfA7ar6sBYHmDh48KAAQrl27JgiC8/QVp0I1UeXl5Thy5AjGjBkjbZPL5RgzZgz27dvnwJa1bPn5+QCAgIAAAMCRI0eg0WhM+qlz586Ijo6W+mnfvn3o0aOHSUHG8ePHo6CgAGfOnGnE1rcMc+fOxS233GLSJwD7qqn53//+h/79++Oee+5BSEgI+vTpg6+//lran5iYiLS0NJP+8vX1xaBBg0z6y8/PD/3795eOGTNmDORyOQ4cONB4b6aZGzJkCLZu3YoLFy4AAE6cOIHdu3dj4sSJANhXTVV99cu+ffswfPhwKBQK6Zjx48cjISEBubm5jfRuWp78/HzIZDL4+fkBcJ6+ahGVt51RVlYWtFqtRXXw0NBQnD9/3kGtatl0Oh2effZZDB06FN27dwcApKWlQaFQSD/4otDQUKSlpUnHWOtHcR/Vn9WrV+Po0aM4dOiQxT72VdNy5coVLF++HPPnz8fLL7+MQ4cO4ZlnnoFCocD06dOl77e1/jDur5CQEJP9rq6uCAgIYH/VowULFqCgoACdO3eGi4sLtFot3n77bUybNg0A2FdNVH31S1paGmJiYiyeQ9zn7+/fIO1vycrKyvDSSy/h/vvvh4+PDwDn6SsGFkS1NHfuXJw+fRq7d+92dFPIiuvXr2PevHmIi4uDu7u7o5tDNdDpdOjfvz/eeecdAECfPn1w+vRpfPHFF5g+fbqDW0fGfv31V/z0009YtWoVunXrhuPHj+PZZ59FREQE+4qonmk0Gtx7770QBAHLly93dHNsxqlQTVRQUBBcXFwsMtakp6cjLCzMQa1quZ566ils2LAB27dvR2RkpLQ9LCwM5eXlyMvLMzneuJ/CwsKs9qO4j+rHkSNHkJGRgb59+8LV1RWurq6Ij4/Hp59+CldXV4SGhrKvmpDw8HB07drVZFuXLl2QlJQEoPL7Xd3vwLCwMGRkZJjsr6ioQE5ODvurHr3wwgtYsGABpk6dih49euChhx7Cc889hyVLlgBgXzVV9dUv/L3YeMSg4tq1a4iLi5NGKwDn6SsGFk2UQqFAv379sHXrVmmbTqfD1q1bERsb68CWtSyCIOCpp57CunXrsG3bNoshxn79+sHNzc2knxISEpCUlCT1U2xsLE6dOmXyC0H8hWF+YUV1N3r0aJw6dQrHjx+X/vXv3x/Tpk2TvmZfNR1Dhw61SN184cIFtG7dGgAQExODsLAwk/4qKCjAgQMHTPorLy8PR44ckY7Ztm0bdDodBg0a1AjvomUoKSmBXG56ueDi4gKdTgeAfdVU1Ve/xMbGYufOndBoNNIxcXFx6NSpE6dB1SMxqLh48SK2bNmCwMBAk/1O01eNtkycbLZ69WpBqVQKK1euFM6ePSs8/vjjgp+fn0nGGmpYc+bMEXx9fYUdO3YIqamp0r+SkhLpmCeeeEKIjo4Wtm3bJhw+fFiIjY0VYmNjpf1iCtNx48YJx48fFzZt2iQEBwczhWkjMM4KJQjsq6bk4MGDgqurq/D2228LFy9eFH766SdBpVIJP/74o3TMu+++K/j5+Qnr168XTp48Kdxxxx1WU2X26dNHOHDggLB7926hQ4cOTGFaz6ZPny60atVKSje7du1aISgoSHjxxRelY9hXjlFYWCgcO3ZMOHbsmABA+PDDD4Vjx45JmYTqo1/y8vKE0NBQ4aGHHhJOnz4trF69WlCpVEw3a6Pq+qq8vFy4/fbbhcjISOH48eMm1xvGGZ6coa8YWDRx//d//ydER0cLCoVCGDhwoLB//35HN6lFAWD133fffScdU1paKjz55JOCv7+/oFKphDvvvFNITU01eZ6rV68KEydOFDw8PISgoCDh+eefFzQaTSO/m5bHPLBgXzUtf/75p9C9e3dBqVQKnTt3Fr766iuT/TqdTnj11VeF0NBQQalUCqNHjxYSEhJMjsnOzhbuv/9+wcvLS/Dx8RFmzpwpFBYWNubbaPYKCgqEefPmCdHR0YK7u7vQtm1b4ZVXXjG54GFfOcb27dut/o2aPn26IAj11y8nTpwQhg0bJiiVSqFVq1bCu+++21hvsdmorq8SExOrvN7Yvn279BzO0FcyQTAqnUlERERERFQHXGNBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBREQNZsaMGZDJZHjiiScs9s2dOxcymQwzZsxo/IYREVG9Y2BBREQNKioqCqtXr0Zpaam0raysDKtWrUJ0dLQDW0ZERPWJgQURETWovn37IioqCmvXrpW2rV27FtHR0ejTp4+0Ta1W45lnnkFISAjc3d0xbNgwHDp0CACg0+kQGRmJ5cuXmzz3sWPHIJfLce3aNQBAXl4eHn30UQQHB8PHxwc333wzTpw40QjvkoiIGFgQEVGDmzVrFr777jvp8YoVKzBz5kyTY1588UX8/vvv+P7773H06FG0b98e48ePR05ODuRyOe6//36sWrXK5JyffvoJQ4cORevWrQEA99xzDzIyMvD333/jyJEj6Nu3L0aPHo2cnJyGf5NERC0cAwsiImpwDz74IHbv3o1r167h2rVr2LNnDx588EFpf3FxMZYvX473338fEydORNeuXfH111/Dw8MD3377LQBg2rRp2LNnD5KSkgDoRzFWr16NadOmAQB2796NgwcPYs2aNejfvz86dOiADz74AH5+fvjtt98a/00TEbUwro5uABERNX/BwcG45ZZbsHLlSgiCgFtuuQVBQUHS/suXL0Oj0WDo0KHSNjc3NwwcOBDnzp0DAPTu3RtdunTBqlWrsGDBAsTHxyMjIwP33HMPAODEiRMoKipCYGCgyWuXlpbi8uXLjfAuiYhaNgYWRETUKGbNmoWnnnoKALBs2bI6Pce0adOkwGLVqlWYMGGCFEgUFRUhPDwcO3bssDjPz8+vrs0mIqJa4lQoIiJqFBMmTEB5eTk0Gg3Gjx9vsq9du3ZQKBTYs2ePtE2j0eDQoUPo2rWrtO2BBx7A6dOnceTIEfz222/SNChAv0g8LS0Nrq6uaN++vck/49ERIiJqGByxICKiRuHi4iJNa3JxcTHZ5+npiTlz5uCFF15AQEAAoqOjsXTpUpSUlOCRRx6RjmvTpg2GDBmCRx55BFqtFrfffru0b8yYMYiNjcXkyZOxdOlSdOzYESkpKfjrr79w5513on///o3zRomIWigGFkRE1Gh8fHyq3Pfuu+9Cp9PhoYceQmFhIfr374/NmzfD39/f5Lhp06bhySefxMMPPwwPDw9pu0wmw8aNG/HKK69g5syZyMzMRFhYGIYPH47Q0NAGe09ERKQnEwRBcHQjiIiIiIjIuXGNBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2Y2BBRERERER2e3/AejO142RGcbmAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x450 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Make plots\n",
    "model_name = \"a2c_ps_random_conv_v1_1\"\n",
    "kernel_size = 100\n",
    "fig, ax = plot_step_rewards(rewards, kernel_size, save_path=os.path.join(figs_path, f\"{model_name}_rewards.png\"))\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "DL",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}