MultiClass Classification-BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BBC Text MultiClass Classification - BERT: Bidirectional Encoder Representations from Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BERT stands for Bidirectional Encoder Representations from Transformers. It is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of NLP tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![BERT](Screenshot-from-2019-06-18-08-46-42.png)](https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![BERT](Screenshot-from-2019-06-18-08-45-09.png)](https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get BERT server and client\n",
    "\n",
    "#### server\n",
    "\n",
    "```bash\n",
    "$ pip install bert-serving-server  \n",
    "```\n",
    "#### client, independent of bert-serving-server`\n",
    "\n",
    "```bash\n",
    "$ pip install bert-serving-client \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### download BERT\n",
    "```bash\n",
    "$ wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip && unzip uncased_L-12_H-768_A-12.zip\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run BERT Server\n",
    "```bash\n",
    "$ bert-serving-start -model_dir uncased_L-12_H-768_A-12/ -num_worker=2 -max_seq_len 50\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/syava/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/syava/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/syava/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/syava/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/syava/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/syava/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "from sklearn import model_selection, preprocessing\n",
    "import re\n",
    "import spacy\n",
    "from nltk.stem import WordNetLemmatizer  \n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential,Model\n",
    "from tensorflow.keras.layers import  Reshape,Input,Embedding, Lambda,Dropout,Conv1D,Conv2D,Activation, Dense, Bidirectional,GlobalMaxPooling1D, LSTM, SpatialDropout1D, TimeDistributed,Masking,Layer\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint,EarlyStopping\n",
    "from tensorflow.keras.layers import MaxPool1D\n",
    "import tensorflow.keras.backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset\n",
    "trainDF = pd.read_csv('bbc-text.csv') # encoding = \"latin\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>tech</td>\n",
       "      <td>tv future in the hands of viewers with home th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>business</td>\n",
       "      <td>worldcom boss  left books alone  former worldc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sport</td>\n",
       "      <td>tigers wary of farrell  gamble  leicester say ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sport</td>\n",
       "      <td>yeading face newcastle in fa cup premiership s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>entertainment</td>\n",
       "      <td>ocean s twelve raids box office ocean s twelve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>politics</td>\n",
       "      <td>howard hits back at mongrel jibe michael howar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>politics</td>\n",
       "      <td>blair prepares to name poll date tony blair is...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>sport</td>\n",
       "      <td>henman hopes ended in dubai third seed tim hen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sport</td>\n",
       "      <td>wilkinson fit to face edinburgh england captai...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>entertainment</td>\n",
       "      <td>last star wars  not for children  the sixth an...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        category                                               text\n",
       "0           tech  tv future in the hands of viewers with home th...\n",
       "1       business  worldcom boss  left books alone  former worldc...\n",
       "2          sport  tigers wary of farrell  gamble  leicester say ...\n",
       "3          sport  yeading face newcastle in fa cup premiership s...\n",
       "4  entertainment  ocean s twelve raids box office ocean s twelve...\n",
       "5       politics  howard hits back at mongrel jibe michael howar...\n",
       "6       politics  blair prepares to name poll date tony blair is...\n",
       "7          sport  henman hopes ended in dubai third seed tim hen...\n",
       "8          sport  wilkinson fit to face edinburgh england captai...\n",
       "9  entertainment  last star wars  not for children  the sixth an..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDF.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2225, 2)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDF.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['tech', 'business', 'sport', 'entertainment', 'politics'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDF['category'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sport            511\n",
       "business         510\n",
       "politics         417\n",
       "tech             401\n",
       "entertainment    386\n",
       "Name: category, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainDF['category'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fa0caf4d860>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAV50lEQVR4nO3de5gldX3n8fcHRjBR5OJMZpEBx0fJKomri/MoiolGIlEShRggapQRMaO7eIuXjckmBvLoE7ziLcGwogzGG6gIIjGSQVBRLjNyGYSoE5XALMjIzduqAb/7R/265kzTPfQMXX2amffrec7TVb+qOudb1af601V16ndSVUiSBLDDuAuQJM0fhoIkqWcoSJJ6hoIkqWcoSJJ6C8ZdwL2xcOHCWrp06bjLkKT7lDVr1vygqhZNNe0+HQpLly5l9erV4y5Dku5Tklw33TRPH0mSeoaCJKlnKEiSeoaCJKlnKEiSeoaCJKlnKEiSeoaCJKlnKEiSevfpO5o1M//xt48edwmzbp83rt2q5Q5874GzXMn4XfSKi8ZdgrYhgx4pJPlekrVJrkiyurXtkeS8JN9uP3dv7UnyniTrklyVZP8ha5Mk3d1cnD76nap6bFUta+NvAFZV1b7AqjYO8Exg3/ZYAZw0B7VJkkaM45rCocDKNrwSOGyk/bTqXAzslmTPMdQnSdutoUOhgC8kWZNkRWtbXFU3tuGbgMVteC/g+pFlb2htm0iyIsnqJKs3bNgwVN2StF0a+kLzk6tqfZJfA85L8m+jE6uqktSWPGFVnQycDLBs2bItWlaStHmDHilU1fr282bgTODxwPcnTgu1nze32dcDe48svqS1SZLmyGChkOQBSXaZGAYOBq4GzgaWt9mWA2e14bOBo9qnkA4A7hg5zSRJmgNDnj5aDJyZZOJ1PlpVn09yGXB6kmOA64Aj2/znAocA64CfAkcPWJskaQqDhUJVfQd4zBTttwAHTdFewLFD1SNJumd2cyFJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqSeoSBJ6hkKkqTegnEXIEnj9r7XfnbcJcy6l7/jWVu1nEcKkqSeoSBJ6hkKkqTe4KGQZMcklyc5p40/LMklSdYl+USSnVr7zm18XZu+dOjaJEmbmosjhVcB146MvwU4saoeAdwGHNPajwFua+0ntvkkSXNo0E8fJVkC/D7wZuA1SQI8DXh+m2UlcBxwEnBoGwb4JPC+JKmq2prXftzrT9v6wuepNW87atwlSNrGDX2k8C7gfwG/bOMPBm6vqjvb+A3AXm14L+B6gDb9jjb/JpKsSLI6yeoNGzYMWbskbXcGC4UkfwDcXFVrZvN5q+rkqlpWVcsWLVo0m08tSdu9IU8fHQg8O8khwP2BBwHvBnZLsqAdDSwB1rf51wN7AzckWQDsCtwyYH2SpEkGO1Koqr+oqiVVtRR4LnB+Vf0J8EXg8DbbcuCsNnx2G6dNP39rrydIkrbOOO5T+HO6i87r6K4ZnNLaTwEe3NpfA7xhDLVJ0nZtTvo+qqoLgAva8HeAx08xz8+AI+aiHklw4W8/ZdwlzLqnfOnCcZdwn+cdzZKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKknqEgSeoZCpKk3mChkOT+SS5NcmWSbyQ5vrU/LMklSdYl+USSnVr7zm18XZu+dKjaJElTG/JI4efA06rqMcBjgWckOQB4C3BiVT0CuA04ps1/DHBbaz+xzSdJmkODhUJ1ftxG79ceBTwN+GRrXwkc1oYPbeO06QclyVD1SZLubtBrCkl2THIFcDNwHvDvwO1VdWeb5QZgrza8F3A9QJt+B/DgKZ5zRZLVSVZv2LBhyPIlabszaChU1V1V9VhgCfB44JGz8JwnV9Wyqlq2aNGie12jJGmjGYVCklUzaZtOVd0OfBF4IrBbkgVt0hJgfRteD+zdnnsBsCtwy0xfQ5J07202FNoniPYAFibZPcke7bGUjad9plt2UZLd2vCvAE8HrqULh8PbbMuBs9rw2W2cNv38qqotXyVJ0tZacA/TXwq8GngIsAaYuPD7Q+B997DsnsDKJDvShc/pVXVOkmuAjyd5E3A5cEqb/xTgw0nWAbcCz93SlZEk3TubDYWqejfw7iSvqKr3bskTV9VVwH+fov07dNcXJrf/DDhiS15DkjS77ulIAYCqem+SJwFLR5epqtMGqkuSNAYzCoUkHwYeDlwB3NWaCzAUJGkbMqNQAJYB+3nhV5K2bTO9T+Fq4L8MWYgkafxmeqSwELgmyaV0fRoBUFXPHqQqSdJYzDQUjhuyCEnS/DDTTx9dOHQhkqTxm+mnj35E92kjgJ3oejz9SVU9aKjCJElzb6ZHCrtMDLfurA8FDhiqKEnSeGxxL6ntexI+A/zeAPVIksZopqePnjMyugPdfQs/G6QiSdLYzPTTR88aGb4T+B7dKSRJ0jZkptcUjh66EEnS+M30S3aWJDkzyc3t8akkS4YuTpI0t2Z6oflDdF+C85D2+GxrkyRtQ2YaCouq6kNVdWd7nAr4BcmStI2ZaSjckuQFSXZsjxfg9ydL0jZnpqHwYuBI4CbgRrrvUH7RQDVJksZkph9J/VtgeVXdBpBkD+DtdGEhSdpGzPRI4b9NBAJAVd3KFN+/LEm6b5tpKOyQZPeJkXakMNOjDEnSfcRM/7C/A/hakjPa+BHAm4cpSZI0LjO9o/m0JKuBp7Wm51TVNcOVJUkahxmfAmohYBBI0jZsi7vOliRtuwwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVJvsFBIsneSLya5Jsk3kryqte+R5Lwk324/d2/tSfKeJOuSXJVk/6FqkyRNbcgjhTuB11bVfsABwLFJ9gPeAKyqqn2BVW0c4JnAvu2xAjhpwNokSVMYLBSq6saq+nob/hFwLbAXcCiwss22EjisDR8KnFadi4Hdkuw5VH2SpLubk2sKSZbSff/CJcDiqrqxTboJWNyG9wKuH1nshtY2+blWJFmdZPWGDRsGq1mStkeDh0KSBwKfAl5dVT8cnVZVBdSWPF9VnVxVy6pq2aJFi2axUknSoKGQ5H50gfCRqvp0a/7+xGmh9vPm1r4e2Htk8SWtTZI0R4b89FGAU4Brq+qdI5POBpa34eXAWSPtR7VPIR0A3DFymkmSNAeG/ErNA4EXAmuTXNHa/hI4ATg9yTHAdcCRbdq5wCHAOuCnwNED1iZJmsJgoVBVXwEyzeSDppi/gGOHqkeSdM+8o1mS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1BssFJJ8MMnNSa4eadsjyXlJvt1+7t7ak+Q9SdYluSrJ/kPVJUma3pBHCqcCz5jU9gZgVVXtC6xq4wDPBPZtjxXASQPWJUmaxmChUFVfAm6d1HwosLINrwQOG2k/rToXA7sl2XOo2iRJU5vrawqLq+rGNnwTsLgN7wVcPzLfDa3tbpKsSLI6yeoNGzYMV6kkbYfGdqG5qgqorVju5KpaVlXLFi1aNEBlkrT9mutQ+P7EaaH28+bWvh7Ye2S+Ja1NkjSH5joUzgaWt+HlwFkj7Ue1TyEdANwxcppJkjRHFgz1xEk+BjwVWJjkBuBvgBOA05McA1wHHNlmPxc4BFgH/BQ4eqi6JEnTGywUqup500w6aIp5Czh2qFokSTPjHc2SpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpJ6hIEnqzatQSPKMJN9Msi7JG8ZdjyRtb+ZNKCTZEfh74JnAfsDzkuw33qokafsyb0IBeDywrqq+U1W/AD4OHDrmmiRpu5KqGncNACQ5HHhGVb2kjb8QeEJVvXzSfCuAFW30vwLfnNNCp7YQ+MG4i5gn3BYdt8NGbouN5su2eGhVLZpqwoK5ruTeqqqTgZPHXceoJKuratm465gP3BYdt8NGbouN7gvbYj6dPloP7D0yvqS1SZLmyHwKhcuAfZM8LMlOwHOBs8dckyRtV+bN6aOqujPJy4F/AXYEPlhV3xhzWTM1r05njZnbouN22MhtsdG83xbz5kKzJGn85tPpI0nSmBkKkqSeoTCNJLsl+Z9bueyp7b6LeS3J0iRX38vneEiST85WTduTJE9N8qRx1wGQ5LCt6UFgpuuQ5Nnj6rrm3uzLs/DaFyRZ1obPbbVsUs9824cMhentBozljXRfUlX/t6rmfQDON0kWAE8F5kUoAIfRdS8zY1uyDlV1dlWdsHWl3WvzYl+uqkOq6vbJ9cy7faiqfEzxoOtm4/8BVwBvA15P97HZq4DjR+Y7qrVdCXy4tZ0KvAf4KvAd4PBxr88067gU+DfgI8C1wCeBXwW+Byxs8ywDLmjDT2nb4wrgcmCX9hxXt+kvAj4NfB74NvDWkdc6GPga8HXgDOCBrf0E4Jq2Dd/e2o4Arm7b9Evj3k6tpgcAn2s1XQ38cdtObwXWApcCjxjZrue3dVoF7DPyvng/cEnbTjfR3YtzBfBbA9T8glbXFcA/0n2q78fAm9t6XAwspvujfivw3Tbvw9vj88Aa4MvAI2eyDsCz2rTLgX8FFo+8N963uf2DLmAuBM5q7ScAf9LWYS3w8DbfIuBTdPvjZcCBrf044IPABW35V061Lw+0zxzU1nltq2HnNv8FwLI2/D26O5on/21ZysZ9aEfg7XTvsauAV0y3nwz2Xh/3zjZfH5N+UQfTfZQsdEdX5wC/DfwG8C02/gHdY+RNf0abdz+6Pp3Gvk7TrGON7FQfBF7H9KHw2ZF5H0j3kebR7fSitjPuCtwfuI7uhsSFwJeAB7T5/hx4I/Bgum5KJj4Ft1v7uRbYa7Rt3A/gj4D/MzK+a9tO/7uNHwWcM7KdlrfhFwOfGXlfnAPs2MaPA143UL2PanXcr43/Q6uxgGe1trcCfzVS2+Ejy68C9m3DTwDOn8k6ALuP/D5fArxj5L0xGgp32z/oQuF2YE9gZ7qwOb5NexXwrjb8UeDJbXgf4NqRWr7all0I3ALcb/Q9OtA+81fA9cCvt7bTgFe34Qu4eyhsUg+b7kP/gy5oFrTxPZhmPxnqMW/uU5jnDm6Py9v4A4F9gccAZ1TVDwCq6taRZT5TVb8ErkmyeC6L3ULXV9VFbfifgFduZt6LgHcm+Qjw6aq6IcnkeVZV1R0ASa4BHkp3uLwfcFGbfye6o4Y7gJ8BpyQ5h+6PzcTrnJrkdLr/RueDtcA7kryF7o//l9u6fKxN/xhwYht+IvCcNvxhuj++E86oqrvmoN6DgMcBl7U6fwW4GfgFG7fzGuDpkxdM8kC6o4czRn6/O4/Msrl1WAJ8IsmedL/n704z33T7x2VVdWOr49+BL7T2tcDvtOHfBfYbqe1BrWaAz1XVz4GfJ7mZ7khotk3eZ/4a+G5Vfau1rQSOBd61Fc/9u8D7q+pO6P6mtNN0U+0ngzAUZibA31XVP27SmLxiM8v8fNLy89XkG1UKuJON15vu30+oOiHJ54BD6P7A/x7dm3XU6HrfRfceC3BeVT1v8osneTzdH7DDgZcDT6uqlyV5AvD7wJokj6uqW7Z2BWdDVX0ryf506/6mJKsmJo3ONoOn+smsFze1ACur6i82aUxeV+3fTTb+fibbAbi9qh47zXNvbh3eC7yzqs5O8lS6/96nMt3+Mdr+y5HxX47UugNwQFVt8t5rITHV+2+2Tf4930733/wgqrux9277yVCv54Xm6f2I7pw5dHdZv3jiv5EkeyX5NbrzxkckeXBr32Msld47+yR5Yht+PvAVusPcx7W2P5qYMcnDq2ptVb2F7lzuI2f4GhcDByZ5RHueByT59bY9d62qc4E/ozvymnidS6rqjcAGNu0TayySPAT4aVX9E9154P3bpD8e+fm1NvxVum5aoDsn/uVpnnb0PTbbVgGHt/cpSfZI8tDNzN/XUlU/BL6b5Ii2bJI85p6Wa3ZlY59ly+9F/ZvzBaD/hyzJdOE1Yba38+R9ZjWwdOL9DbyQ7trI1tRzHvDSdnQw8Xubcj8ZiqEwjfaf6UXtI5tPpzuP+bUka+nO+e1SXTccbwYuTHIl8M6xFbz1vgkcm+RauvPBJwHHA+9Ospruv60Jr05ydZKrgP8E/nkmL1BVG+jOKX+sLfs1ukDZBTintX0FeE1b5G1J1rZt/1W6i6Lj9mjg0iRXAH8DvKm1797qfxXdDgvdH6yjW/sL27SpfBb4wyRXJPmt2Sy2qq6hO9f9hVbHeXTn6qfzceD1SS5P8nC6MDumva+/wfTfbTJ5HY6jO+20huG6iH4lsCzJVe0U5cs2N/PovpzkbbPw+pP3mROBo+nWey3dUc37t7KeDwD/AVzVtv3zmX4/GYTdXEhbKcn36C4izof+8TUHkiylu6b0m2MuZTAeKUiSeh4pSJJ6HilIknqGgiSpZyhIknqGgrQF5lPPptIQDAVpyzyVgXs2bTeLuW9qLHzjSUCSo9rNUFcm+XCSZyW5pN3M9a9JFrfPqL8M+LOJm7WSLEryqSSXtceB7fkWJTkvyTeSfCDJdUkWtmmvaTcuXZ3k1a1taZJvJjmNrofMv07yrpH6/jTJiZPrlmabH0nVdi/JbwBnAk+qqh+07kqKrv+fSvIS4FFV9dokxwE/rqq3t2U/CvxDVX0lyT7Av1TVo5K8D1hfVX+X5Bl0d38vousg8FTgALo+fy6h6+L6NroeZp9UVRe3rg2upOuy+j+TfBV4aVWtnaPNou2UHeJJXedim/R2m+TRzKy3z+l67Hwy8Ift+T6f5LY2/cnAmVX1E4Akn6b7HoKzgeuq6uK2zI+TnA/8QetO4X4GguaCoSBNbaa9fW6ux84tNbn30Q8Af0n3pS4f2ponlLaU1xSkqXu7na63z8k9XE7XY+dFwJGt7WC6jtOg6zH1sCS/muQBdEcTU/aiWlWX0PUQ+3w2fm+DNChDQdu9aXq7PY6pe/uc3CvodD12Hg8c3Hp6PYLuayt/VFVfp7umcCnd9YQPVNXlTO904KKqum0z80izxgvN0gCS7Azc1b4g5YnASZv50prNPc85wIlVteoeZ5ZmgdcUpGHsA5ze7jf4BfCnW7Jwkt3ojiauNBA0lzxSkCT1vKYgSeoZCpKknqEgSeoZCpKknqEgSer9fzWaen2U/uX8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(trainDF['category'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean text from noise\n",
    "def clean_text(text):\n",
    "    \n",
    "    # remove next lines\n",
    "    text = text.strip().replace(\"\\n\", \" \").replace(\"\\r\", \" \")\n",
    "    \n",
    "    # filter to allow only alphabets\n",
    "    text = re.sub(r'[^a-zA-Z\\']', ' ', text)\n",
    "    \n",
    "    # remove Unicode characters\n",
    "    text = re.sub(r'[^\\x00-\\x7F]+', '', text)\n",
    "    \n",
    "    # convert to lowercase to maintain consistency\n",
    "    text = text.lower()\n",
    "    \n",
    "    # tokenize words\n",
    "    # tokens = word_tokenize(text)\n",
    "    \n",
    "    # lemmatization of words\n",
    "    # lemmatizer = WordNetLemmatizer()\n",
    "    \n",
    "    # lemmas = []\n",
    "    # for tok in tokens:\n",
    "    #    lemmas.append(lemmatizer.lemmatize(tok.lower()))\n",
    "    \n",
    "    # get unique words\n",
    "    # tokens = list(set(lemmas))\n",
    "    \n",
    "    # STOPLIST = set(stopwords.words('english'))\n",
    "    # SYMBOLS = \" \".join(string.punctuation).split(\" \")\n",
    "    # reomve stop words and special charaters\n",
    "    # tokens = [tok for tok in tokens if tok not in STOPLIST]\n",
    "    # tokens = [tok for tok in tokens if tok not in SYMBOLS]\n",
    "    \n",
    "    # tokens = [tok for tok in tokens if len(tok) >= 3]\n",
    "    \n",
    "    # remove remaining tokens that are not alphabetic\n",
    "    # tokens = [tok for tok in tokens if tok.isalpha()]\n",
    "    \n",
    "    # stemming of words\n",
    "    # porter = PorterStemmer()\n",
    "    # tokens = [porter.stem(word) for word in tokens]\n",
    "        \n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data cleaning\n",
    "trainDF['text'] = trainDF['text'].apply(lambda x:clean_text(x))\n",
    "# Label Encoding\n",
    "le = preprocessing.LabelEncoder()\n",
    "le.fit(trainDF['category'])\n",
    "\n",
    "def encode(le, labels):\n",
    "    enc = le.transform(labels)\n",
    "    return tf.keras.utils.to_categorical(enc)\n",
    "\n",
    "def decode(le, one_hot):\n",
    "    dec = np.argmax(one_hot, axis=1)\n",
    "    return le.inverse_transform(dec)\n",
    "\n",
    "\n",
    "X = trainDF['text']\n",
    "y = encode(le, trainDF['category'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the dataset into training and validation datasets\n",
    "x_train, x_val, y_train, y_val = model_selection.train_test_split(np.asarray(X), np.asarray(y), test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1780,), (1780, 5), (445,), (445, 5))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape,y_train.shape, x_val.shape,y_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Building"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/syava/.local/lib/python3.6/site-packages/bert_serving/client/__init__.py:299: UserWarning: some of your sentences have more tokens than \"max_seq_len=50\" set on the server, as consequence you may get less-accurate or truncated embeddings.\n",
      "here is what you can do:\n",
      "- disable the length-check by create a new \"BertClient(check_length=False)\" when you do not want to display this warning\n",
      "- or, start a new server with a larger \"max_seq_len\"\n",
      "  '- or, start a new server with a larger \"max_seq_len\"' % self.length_limit)\n"
     ]
    }
   ],
   "source": [
    "from bert_serving.client import BertClient\n",
    "\n",
    "# make a connection with the BERT server using it's ip address\n",
    "bc = BertClient(ip=\"YOUR_SERVER_IP\")\n",
    "# get the embedding for train and val sets\n",
    "X_tr_bert = bc.encode(x_train.tolist())\n",
    "X_val_bert = bc.encode(x_val.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Build model...\n",
      "WARNING:tensorflow:From /home/syava/.local/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /home/syava/.local/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "reshape (Reshape)            (None, 768, 1)            0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 768, 64)           128       \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 768, 64)           0         \n",
      "_________________________________________________________________\n",
      "conv1d (Conv1D)              (None, 766, 64)           12352     \n",
      "_________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D) (None, 383, 64)           0         \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 100)               66000     \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 5)                 505       \n",
      "=================================================================\n",
      "Total params: 78,985\n",
      "Trainable params: 78,985\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(Reshape((768,1),input_shape=(768,)))\n",
    "model.add(Dense(64, input_shape=(768,1),activation=\"relu\"))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Conv1D(64,3,padding='valid',activation='relu',strides=1))\n",
    "model.add(MaxPool1D(pool_size = 2))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(5, activation = 'softmax'))             \n",
    "model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train...\n",
      "Train on 1780 samples, validate on 445 samples\n",
      "WARNING:tensorflow:From /home/syava/.local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 1.6002 - acc: 0.2373\n",
      "Epoch 00001: val_acc improved from -inf to 0.27191, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 1.6000 - acc: 0.2365 - val_loss: 1.5890 - val_acc: 0.2719\n",
      "Epoch 2/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 1.5604 - acc: 0.3113\n",
      "Epoch 00002: val_acc improved from 0.27191 to 0.36404, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 1.5601 - acc: 0.3096 - val_loss: 1.4986 - val_acc: 0.3640\n",
      "Epoch 3/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 1.3774 - acc: 0.4323\n",
      "Epoch 00003: val_acc improved from 0.36404 to 0.47191, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 1.3756 - acc: 0.4326 - val_loss: 1.2983 - val_acc: 0.4719\n",
      "Epoch 4/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 1.2819 - acc: 0.4838\n",
      "Epoch 00004: val_acc improved from 0.47191 to 0.49213, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 1.2785 - acc: 0.4876 - val_loss: 1.2499 - val_acc: 0.4921\n",
      "Epoch 5/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 1.1457 - acc: 0.5295\n",
      "Epoch 00005: val_acc did not improve from 0.49213\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 1.1514 - acc: 0.5253 - val_loss: 1.2732 - val_acc: 0.4719\n",
      "Epoch 6/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 1.0374 - acc: 0.5909\n",
      "Epoch 00006: val_acc improved from 0.49213 to 0.62472, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 21s 12ms/sample - loss: 1.0371 - acc: 0.5904 - val_loss: 0.9566 - val_acc: 0.6247\n",
      "Epoch 7/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.9142 - acc: 0.6568\n",
      "Epoch 00007: val_acc did not improve from 0.62472\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.9125 - acc: 0.6573 - val_loss: 0.9107 - val_acc: 0.6247\n",
      "Epoch 8/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.8543 - acc: 0.6730\n",
      "Epoch 00008: val_acc improved from 0.62472 to 0.71685, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 19s 10ms/sample - loss: 0.8554 - acc: 0.6708 - val_loss: 0.7761 - val_acc: 0.7169\n",
      "Epoch 9/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.7743 - acc: 0.7043\n",
      "Epoch 00009: val_acc did not improve from 0.71685\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.7751 - acc: 0.7045 - val_loss: 0.7353 - val_acc: 0.7056\n",
      "Epoch 10/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.6920 - acc: 0.7373\n",
      "Epoch 00010: val_acc did not improve from 0.71685\n",
      "1780/1780 [==============================] - 21s 12ms/sample - loss: 0.6959 - acc: 0.7354 - val_loss: 0.7397 - val_acc: 0.7034\n",
      "Epoch 11/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.6813 - acc: 0.7488\n",
      "Epoch 00011: val_acc improved from 0.71685 to 0.74157, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 24s 13ms/sample - loss: 0.6793 - acc: 0.7494 - val_loss: 0.6607 - val_acc: 0.7416\n",
      "Epoch 12/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.6468 - acc: 0.7639\n",
      "Epoch 00012: val_acc did not improve from 0.74157\n",
      "1780/1780 [==============================] - 23s 13ms/sample - loss: 0.6427 - acc: 0.7646 - val_loss: 0.7015 - val_acc: 0.7303\n",
      "Epoch 13/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.6517 - acc: 0.7569\n",
      "Epoch 00013: val_acc improved from 0.74157 to 0.77753, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 0.6447 - acc: 0.7601 - val_loss: 0.5963 - val_acc: 0.7775\n",
      "Epoch 14/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.6109 - acc: 0.7818\n",
      "Epoch 00014: val_acc improved from 0.77753 to 0.79326, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.6091 - acc: 0.7837 - val_loss: 0.5768 - val_acc: 0.7933\n",
      "Epoch 15/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.5808 - acc: 0.7865\n",
      "Epoch 00015: val_acc did not improve from 0.79326\n",
      "1780/1780 [==============================] - 19s 10ms/sample - loss: 0.5812 - acc: 0.7860 - val_loss: 0.5876 - val_acc: 0.7910\n",
      "Epoch 16/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.5689 - acc: 0.7934\n",
      "Epoch 00016: val_acc improved from 0.79326 to 0.79775, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 23s 13ms/sample - loss: 0.5642 - acc: 0.7949 - val_loss: 0.5407 - val_acc: 0.7978\n",
      "Epoch 17/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.5478 - acc: 0.7934\n",
      "Epoch 00017: val_acc did not improve from 0.79775\n",
      "1780/1780 [==============================] - 24s 14ms/sample - loss: 0.5487 - acc: 0.7921 - val_loss: 0.5900 - val_acc: 0.7685\n",
      "Epoch 18/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4839 - acc: 0.8258\n",
      "Epoch 00018: val_acc improved from 0.79775 to 0.80674, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 22s 12ms/sample - loss: 0.4823 - acc: 0.8258 - val_loss: 0.5452 - val_acc: 0.8067\n",
      "Epoch 19/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4616 - acc: 0.8328\n",
      "Epoch 00019: val_acc did not improve from 0.80674\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.4584 - acc: 0.8348 - val_loss: 0.5761 - val_acc: 0.7955\n",
      "Epoch 20/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4601 - acc: 0.8385\n",
      "Epoch 00020: val_acc improved from 0.80674 to 0.81348, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 22s 12ms/sample - loss: 0.4614 - acc: 0.8376 - val_loss: 0.5024 - val_acc: 0.8135\n",
      "Epoch 21/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4410 - acc: 0.8362\n",
      "Epoch 00021: val_acc improved from 0.81348 to 0.81798, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.4433 - acc: 0.8360 - val_loss: 0.5158 - val_acc: 0.8180\n",
      "Epoch 22/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4152 - acc: 0.8484\n",
      "Epoch 00022: val_acc did not improve from 0.81798\n",
      "1780/1780 [==============================] - 22s 12ms/sample - loss: 0.4214 - acc: 0.8478 - val_loss: 0.4999 - val_acc: 0.8157\n",
      "Epoch 23/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4188 - acc: 0.8490\n",
      "Epoch 00023: val_acc improved from 0.81798 to 0.84045, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 25s 14ms/sample - loss: 0.4213 - acc: 0.8489 - val_loss: 0.4560 - val_acc: 0.8404\n",
      "Epoch 24/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.4089 - acc: 0.8547\n",
      "Epoch 00024: val_acc improved from 0.84045 to 0.84270, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 21s 12ms/sample - loss: 0.4120 - acc: 0.8522 - val_loss: 0.4570 - val_acc: 0.8427\n",
      "Epoch 25/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.3855 - acc: 0.8628\n",
      "Epoch 00025: val_acc did not improve from 0.84270\n",
      "1780/1780 [==============================] - 22s 12ms/sample - loss: 0.3872 - acc: 0.8629 - val_loss: 0.4556 - val_acc: 0.8247\n",
      "Epoch 26/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.3497 - acc: 0.8773\n",
      "Epoch 00026: val_acc did not improve from 0.84270\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.3552 - acc: 0.8758 - val_loss: 0.4302 - val_acc: 0.8404\n",
      "Epoch 27/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.3421 - acc: 0.8819\n",
      "Epoch 00027: val_acc did not improve from 0.84270\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.3417 - acc: 0.8820 - val_loss: 0.4198 - val_acc: 0.8427\n",
      "Epoch 28/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.3410 - acc: 0.8721\n",
      "Epoch 00028: val_acc improved from 0.84270 to 0.84944, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 0.3477 - acc: 0.8697 - val_loss: 0.4245 - val_acc: 0.8494\n",
      "Epoch 29/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2784 - acc: 0.9022\n",
      "Epoch 00029: val_acc did not improve from 0.84944\n",
      "1780/1780 [==============================] - 25s 14ms/sample - loss: 0.2829 - acc: 0.9000 - val_loss: 0.4703 - val_acc: 0.8337\n",
      "Epoch 30/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.3388 - acc: 0.8721\n",
      "Epoch 00030: val_acc did not improve from 0.84944\n",
      "1780/1780 [==============================] - 23s 13ms/sample - loss: 0.3359 - acc: 0.8730 - val_loss: 0.4306 - val_acc: 0.8404\n",
      "Epoch 31/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.3037 - acc: 0.8877\n",
      "Epoch 00031: val_acc improved from 0.84944 to 0.85843, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.3035 - acc: 0.8882 - val_loss: 0.3933 - val_acc: 0.8584\n",
      "Epoch 32/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2844 - acc: 0.9057\n",
      "Epoch 00032: val_acc improved from 0.85843 to 0.86292, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.2831 - acc: 0.9062 - val_loss: 0.3979 - val_acc: 0.8629\n",
      "Epoch 33/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2747 - acc: 0.8976\n",
      "Epoch 00033: val_acc did not improve from 0.86292\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.2722 - acc: 0.8989 - val_loss: 0.4153 - val_acc: 0.8562\n",
      "Epoch 34/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2629 - acc: 0.9086\n",
      "Epoch 00034: val_acc did not improve from 0.86292\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.2653 - acc: 0.9067 - val_loss: 0.3765 - val_acc: 0.8539\n",
      "Epoch 35/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2459 - acc: 0.9115\n",
      "Epoch 00035: val_acc did not improve from 0.86292\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.2462 - acc: 0.9118 - val_loss: 0.4753 - val_acc: 0.8315\n",
      "Epoch 36/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2417 - acc: 0.9201\n",
      "Epoch 00036: val_acc improved from 0.86292 to 0.86517, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.2423 - acc: 0.9191 - val_loss: 0.3580 - val_acc: 0.8652\n",
      "Epoch 37/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2245 - acc: 0.9282\n",
      "Epoch 00037: val_acc did not improve from 0.86517\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.2227 - acc: 0.9281 - val_loss: 0.5510 - val_acc: 0.8112\n",
      "Epoch 38/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2460 - acc: 0.9120\n",
      "Epoch 00038: val_acc did not improve from 0.86517\n",
      "1780/1780 [==============================] - 22s 12ms/sample - loss: 0.2464 - acc: 0.9135 - val_loss: 0.4356 - val_acc: 0.8472\n",
      "Epoch 39/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2176 - acc: 0.9259\n",
      "Epoch 00039: val_acc did not improve from 0.86517\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 0.2154 - acc: 0.9270 - val_loss: 0.3905 - val_acc: 0.8629\n",
      "Epoch 40/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.2150 - acc: 0.9230\n",
      "Epoch 00040: val_acc did not improve from 0.86517\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.2133 - acc: 0.9242 - val_loss: 0.4313 - val_acc: 0.8427\n",
      "Epoch 41/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1790 - acc: 0.9427\n",
      "Epoch 00041: val_acc improved from 0.86517 to 0.87191, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.1787 - acc: 0.9427 - val_loss: 0.3818 - val_acc: 0.8719\n",
      "Epoch 42/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1832 - acc: 0.9346\n",
      "Epoch 00042: val_acc did not improve from 0.87191\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.1822 - acc: 0.9354 - val_loss: 0.4445 - val_acc: 0.8494\n",
      "Epoch 43/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1676 - acc: 0.9375\n",
      "Epoch 00043: val_acc did not improve from 0.87191\n",
      "1780/1780 [==============================] - 21s 12ms/sample - loss: 0.1672 - acc: 0.9376 - val_loss: 0.4170 - val_acc: 0.8562\n",
      "Epoch 44/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1647 - acc: 0.9421\n",
      "Epoch 00044: val_acc improved from 0.87191 to 0.87640, saving model to ./model.h5\n",
      "1780/1780 [==============================] - 23s 13ms/sample - loss: 0.1623 - acc: 0.9427 - val_loss: 0.3732 - val_acc: 0.8764\n",
      "Epoch 45/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1207 - acc: 0.9583\n",
      "Epoch 00045: val_acc did not improve from 0.87640\n",
      "1780/1780 [==============================] - 18s 10ms/sample - loss: 0.1209 - acc: 0.9579 - val_loss: 0.3447 - val_acc: 0.8719\n",
      "Epoch 46/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1421 - acc: 0.9525\n",
      "Epoch 00046: val_acc did not improve from 0.87640\n",
      "1780/1780 [==============================] - 19s 11ms/sample - loss: 0.1410 - acc: 0.9528 - val_loss: 0.4041 - val_acc: 0.8629\n",
      "Epoch 47/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1537 - acc: 0.9485\n",
      "Epoch 00047: val_acc did not improve from 0.87640\n",
      "1780/1780 [==============================] - 22s 12ms/sample - loss: 0.1546 - acc: 0.9472 - val_loss: 0.3777 - val_acc: 0.8674\n",
      "Epoch 48/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1544 - acc: 0.9468\n",
      "Epoch 00048: val_acc did not improve from 0.87640\n",
      "1780/1780 [==============================] - 23s 13ms/sample - loss: 0.1597 - acc: 0.9433 - val_loss: 0.4207 - val_acc: 0.8652\n",
      "Epoch 49/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1943 - acc: 0.9300\n",
      "Epoch 00049: val_acc did not improve from 0.87640\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.1960 - acc: 0.9281 - val_loss: 0.4786 - val_acc: 0.8629\n",
      "Epoch 50/50\n",
      "1728/1780 [============================>.] - ETA: 0s - loss: 0.1675 - acc: 0.9398\n",
      "Epoch 00050: val_acc did not improve from 0.87640\n",
      "1780/1780 [==============================] - 20s 11ms/sample - loss: 0.1642 - acc: 0.9410 - val_loss: 0.4665 - val_acc: 0.8382\n"
     ]
    }
   ],
   "source": [
    "print('Train...')\n",
    "# checkpoint\n",
    "checkpoint = ModelCheckpoint(\"./model.h5\", monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n",
    "stoperly = EarlyStopping(monitor=\"val_loss\",mode=\"auto\",verbose=1)\n",
    "callbacks_list = [checkpoint]\n",
    "history = model.fit(X_tr_bert, y_train,batch_size=64,epochs=50,validation_data=(X_val_bert, y_val),callbacks=callbacks_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fa0c864ee48>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAHSCAYAAADfUaMwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeVyVZd7H8c/FLrLJroIKCuZSmaJSaWlmavtmti9TmdM2LVNTM8002/M8U1PTTNM+ZftmWU1Z2e5aLqiVu7iLCwIqAsp+P39cgCAoIAfOgfN9v16+iPvc5z4/a4Yv124cx0FERETcx8fdBYiIiHg7hbGIiIibKYxFRETcTGEsIiLiZgpjERERN1MYi4iIuJmfuz44Ojra6dWrl7s+XkREpM0tWbIk13GcmMOvuy2Me/XqRUZGhrs+XkREpM0ZY7Y0dF3d1CIiIm6mMBYREXEzhbGIiIibuW3MWEREpLnKysrIysqiuLjY3aUcVVBQEAkJCfj7+zfpfoWxiIi0G1lZWYSGhtKrVy+MMe4up0GO45CXl0dWVhZJSUlNeo+6qUVEpN0oLi4mKirKY4MYwBhDVFRUs1rvCmMREWlXPDmIqzW3RoWxiIiImymMRURE3ExhLCIi0gwPPPAATz/9dM33f/zjH3nsscda9EzNphYRkXbpT5+sZNWO/S59Zv9uYTx83oCj3jNp0iTuuusubrvtNgCmTZvGF1980aLPVRiLiIg0w0knncTu3bvZsWMHOTk5dOnShcTExBY9U2EsIiLtUmMt2NY0ceJE3n//fXbt2sWkSZNa/DyFsYiISDNNmjSJm2++mdzcXGbPnt3i5zU6gcsYM9UYs9sYs+Io94wyxvxojFlpjGl5VSIiIh5swIABFBQU0L17d7p27dri5zWlZfwK8BTwWkMvGmMigGeA8Y7jbDXGxLa4KhEREQ+3fPlylz2r0Zax4zhzgD1HueVK4APHcbZW3b/bRbU1y5a8Ind8rIiISIu5Yp1xKtDFGDPLGLPEGHOtC57ZLF+vymb0Y7P4elV2W3+0iIhIi7kijP2AIcA5wDjg98aY1IZuNMZMNsZkGGMycnJyXPDR1oiUaPp3C+PuaT+qhSwiIu2OK8I4C/jCcZwix3FygTnAiQ3d6DjOC47jpDmOkxYTE+OCj7aC/Hx4cVwnfHGY8sZSDpZWuOzZIiIirc0VYfxfYIQxxs8YEwwMB1a74LlNt/Yz4t8aw9Qx5azZtZ+HPlqB4zhtWoKIiMixasrSpreBH4C+xpgsY8yNxpgpxpgpAI7jrAZmAj8Di4AXHcc54jKoVpE8CvyDGZz/Lb8ak8L0pVm8tWhrm5YgIiJyrBpd2uQ4zhVNuOfvwN9dUtGxCOgMqeNh1X+5855H+HHbPv708SoGdAtnUGKE28oSERFpio5zatPAi+FALj5b5vLPSYOIDQvk1jeWsKeo1N2ViYiIHFXHCeM+YyEgFFZMJyI4gOeuHkJuUSl3vr2MikqNH4uIiOtceOGFDBkyhAEDBvDCCy+0+HkdZ29q/yDody6s/gTOeYKB3cP5ywUD+M305Tzx1Tp+Pa6vuysUERFX+vwB2OW6XbAAiD8eJvyt0dumTp1KZGQkBw8eZOjQoVxyySVERUUd88d2nJYxwICLoTgfNnwLwKShPZiUlshT363XhiAiIuIyTz75JCeeeCLp6els27aNzMzMFj2v47SMwc6q7tQFVkyHvuMB+NMFA1i5M5+7p/3IjDtG0DOqs1tLFBERF2lCC7Y1zJo1i6+//poffviB4OBgRo0aRXFxcYue2bFaxn4B0O98WPsZlB0EIMjfl2evGkJpeSWv/bDFzQWKiEh7l5+fT5cuXQgODmbNmjUsWLCgxc/sWGEMdlZ1aSFkfllzKTEymL7xoazLLnBjYSIi0hGMHz+e8vJy+vXrxwMPPEB6enqLn9mxuqkBeo2EzrG2q7r/BTWXU2JDmbfedfthi4iIdwoMDOTzzz936TM7XsvYxxcGXAjrvoCSQy3h1LgQsveXkH+wzI3FiYiI1Nfxwhhg4CVQXgxrD/3mkhoXCkCmuqpFRMTDdMwwThgGYd1hxQc1l/rEhgCwLrvQXVWJiIg0qGOGsY8PDLgI1n8NB/cC0D2iE8EBvprEJSLSzrWHU/maW2PHDGOwXdWVZbB6BgA+PoaU2BAydyuMRUTaq6CgIPLy8jw6kB3HIS8vj6CgoCa/p+PNpq7W7STo0gtWfgCDrwEgJS6U2es0o1pEpL1KSEggKyuLnBzP/lkeFBREQkJCk+/vuGFsjG0dz/snFOZASAypcSG8vySLfQdKiQgOcHeFIiLSTP7+/iQlJbm7DJfruN3UYMPYqYDV/wVsyxg0iUtERDxLxw7j2P4Q3bdmVnVqTRhr3FhERDxHxw7j6q7qLd/D/h10Cw8iJNBPa41FRMSjdOwwBrtXNQ6s/AhjDH1iQ9RNLSIiHqXjh3F0CsSfYPeqxm6LqeVNIiLiSTp+GINtHW/PgL2bSY0LJbewlD1Fpe6uSkREBPCWMB5wsf266uNaM6rVOhYREc/gHWHcpSeEJ8Ku5aTG2T2qNYlLREQ8hXeEMdix49x1xIcFERrop0lcIiLiMbwnjKNSIG89BkiJC1E3tYiIeAzvCePoFCgthIKdpMaFkrlbLWMREfEMXhTGqfZr7jpS4kLZU1RKbmGJe2sSERHBq8I4xX7NzayZxKWuahER8QTeE8ahXSEgpCqM7fKmTE3iEhERD+A9YWwMRPWBvExiQwMJC/JTy1hERDyC94Qx2HHj3EyMMXYSl1rGIiLiAbwsjFMgfxuUFpESF8q63QU4juPuqkRExMt5XxgD5G0gNS6EfQfKyNGMahERcTMvC+NDy5s0iUtERDyFd4VxZDJgIG89KVreJCIiHsK7wti/E0T0gNx1xIQEEhHsrz2qRUTE7bwrjKHmwAhjDKmxoTq9SURE3M4LwzgV8jZAZWXNgRGaUS0iIu7khWGcAmUHYP92UuNC2V9czu4CzagWERH38b4wjqpe3pSpSVwiIuIRvC+Ma5Y3HdqjWpO4RETEnbwvjENiITAMctcRHRJIZOcA1u9Wy1hERNzH+8LYmKoZ1ZkApMSGqGUsIiJu5X1hDDUHRgCkxoVqRrWIiLhVo2FsjJlqjNltjFnRyH1DjTHlxphLXVdeK4nqAwU7oKSA1LgQCorLyd6vGdUiIuIeTWkZvwKMP9oNxhhf4BHgSxfU1PqqJ3HlrSelZhKXxo1FRMQ9Gg1jx3HmAHsaue0OYDqw2xVFtbrq05tyM0mJ1fImERFxrxaPGRtjugMXAc+2vJw2EpkMxgdyM4kKCSSqc4BObxIREbdxxQSufwK/cRynsrEbjTGTjTEZxpiMnJwcF3z0MfILhIiekLsOwG6LqeVNIiLiJq4I4zTgHWPMZuBS4BljzIUN3eg4zguO46Q5jpMWExPjgo9ugehUyFsP2BnV67MLNaNaRETcosVh7DhOkuM4vRzH6QW8D9zqOM5HLa6stUWn2DCurCAlLpSCknJ25he7uyoREfFCfo3dYIx5GxgFRBtjsoCHAX8Ax3Gea9XqWlN0CpQXQ/42UmMPzajuFtHJzYWJiIi3aTSMHce5oqkPcxzn+hZV05Zq9qheT2q30wDIzC5kVN9YNxYlIiLeyDt34IJDpzflrqNL5wCiQwK1vElERNzCe8O4czQERUCe3Razb3wIq3bud3NRIiLijbw3jI2ps0f1qX2iWbljPzvzD7q5MBER8TbeG8ZQdXqTXWs8bkA8AF+uzHZnRSIi4oUUxoXZUJxP75gQUmJDmLlil7urEhERL+PdYVwzictu/jF+YDwLN+Wxp6jUjUWJiIi38e4wrjm9yY4bjxsQT6UDX69SV7WIiLQd7w7jLr3A+NaMGw/oFkb3iE58sVJd1SIi0na8O4z9AiAyqSaMjTGMHxjP3MxcCkvK3VyciIh4C+8OY6ha3rS+5tvxA+MprajkuzXt42hmERFp/xTGUX1gzwaorABgcI8uRIcEMFNd1SIi0kYUxtGpUFEK+7YA4OtjGNs/nllrdlNcVuHm4kRExBsojKOrlzdl1lwaPzCeotIK5q/PdVNRIiLiTRTGNac3rau5dHJyFKFBftoARERE2oTCODgSgqPqtIwD/HwYc1wsX6/Opryi0o3FiYiIN1AYg92Jq1YYg+2q3nugjEWb97ipKBER8RYKY7Djxnl1w/i01BiC/H34Ql3VIiLSyhTGYMO4KAcOHGoFBwf4cXpqDF+szKay0nFjcSIi0tEpjKHWHtXr61weNyCeXfuL+SlrnxuKEhERb6Ewhlozqut2VY85Lg4/H8MXOuNYRERakcIYIKIn+PjXWd4EEB7sz8m9o5i5YieOo65qERFpHQpjAF8/iEyu100Ntqt6c94B1mUXuqEwERHxBgrjal1PhI2zIH97nctn9Y/DGLQBiIiItBqFcbXRD9rDIj69F2p1SceGBTGkRxedcSwiIq1GYVwtMhlG/xbWfQ6rPqrz0viB8azauZ+teQfcVJyIiHRkCuPa0m+13dWf3VdnzfG4AfEAah2LiEirUBjX5usH5//bBvFXv6+5nBgZTP+uYTrjWEREWoXC+HBdT4RT7oBlb9gJXVXGDYhn6da95BaWuK82ERHpkBTGDRn1gB1D/uQuKLXjxCNSonEcWLzpCAdH7N0MM+6B/Tvark4REekQFMYN8e8E5/0L9m6C2X8D4Pju4XTy92VhQ2G8egY8fxpkvARrPm3jYkVEpL1TGB9J0mkw+Fr4/inY8SMBfj4M7hlRN4zLS2Hmg/DuVbYlbXyhQOPKIiLSPArjoxn7Z+gcDR/fARXlDOsVxZpd+8k/UAZ7t8DUcbDgGRg+BX7xBYTEQqHCWEREmkdhfDSdusDZf4ddP8OCpxmeHInjwOb578LzIyFvA1z2Okx4BPwCISQOCnSohIiINI/CuDH9zofjzoXv/peTOmXzsP8bnDj/NuiSBLfMhv7nH7o3NF4tYxERaTaFcWOMsa1j3wACXzydG3w/Y0an8+HGLyEyqe69IXEaMxYRkWZTGDdFWDeY8Ch0iuTj1P/jV/lXUFjhW/++0HgoyoWK8ravUURE2i2FcVMNugJ+vZaItIlUVDos3bK3/j0hcYADRbvbvDwREWm/FMbNNKRnF3x9DAs35dV/MdTuYa2uahERaQ6FcTN1DvRjYPdwFjW0+UdIVRgXaka1iIg0ncL4GKQnRfLTtnyKyyrqvhAaZ7+qZSwiIs2gMD4Gw5IiKa2oZNnWfXVf6Bxrv6plLCIizaAwPgZpvSIxhvpd1X4BEByllrGIiDSLwvgYhHfyp198GIs2NzCJKyReLWMREWkWhfExGpYUyZIteyktr6z7Qqg2/hARkeZRGB+j9ORIissqWb49v+4LahmLiEgzNRrGxpipxpjdxpgVR3j9KmPMz8aY5caY740xJ7q+TM8ztFckQP31xqFxNowrKxt4l4iISH1NaRm/Aow/yuubgNMdxzke+Avwggvq8nhRIYH0iQ2pP4krJB4qy+FgA+uQRUREGtBoGDuOMwc4YrI4jvO94zjVe0MuABJcVJvHG54UScbmvVRUOocuaq2xiIg0k6vHjG8EPnfxMz3WsKRICkvKWbVj/6GLoV3tVx2lKCIiTeSyMDbGjMaG8W+Ocs9kY0yGMSYjJyfHVR/tNsOTooDDxo1DqlvGmsQlIiJN45IwNsacALwIXOA4TgOLby3HcV5wHCfNcZy0mJgYV3y0W8WHB9EzKpiFtceNqw+LUMtYRESaqMVhbIzpAXwAXOM4zrqWl9S+DOsVyeLNe6isHjf27wSB4WoZi4hIkzVladPbwA9AX2NMljHmRmPMFGPMlKpb/gBEAc8YY340xmS0Yr0eZ3hyFPsOlJG5u/DQxdA4tYxFRKTJ/Bq7wXGcKxp5/SbgJpdV1M4MTzq03rhvfKi9GKJduEREpOm0A1cLJXTpRNfwoPrjxgpjERFpIoVxCxljGJ4UycKNe3CcqnHjkKpduBzn6G8WERFBYewSw5KiyC0sYVNukb0QGg/lxVCcf/Q3ioiIoDB2ieHJdty4ZmvMkOrlTZpRLSIijVMYu0BydGeiQwIOjRtrS0wREWkGhbELGGMYlhSplrGIiBwThbGLDE+KYvu+g2zbc0AtYxERaRaFsYsMq1pvvGBjHgSGgV8ntYxFRKRJFMYu0jculOiQAOavzwVjbOtYLWMREWkChbGL+PgYTu0Tzbz1uXaf6pB4tYxFRKRJFMYuNDIlhtzCUtbsKlDLWEREmkxh7EIj+kQDMG99DoR2VctYRESaRGHsQvHhQaTEhjA3M9duiVmyH0oPuLssERHxcApjFxuZEsOiTXsoDY61F3SUooiINEJh7GIjU6IpKa9kbVGwvVCgrmoRETk6hbGLDU+OxN/XsHC3v72glrGIiDRCYexiwQF+DOnZha+3GXtBLWMREWmEwrgVjEyJYUE2OD7+ahmLiEijFMatYGRKNGA4GBiltcYiItIohXErGNAtnIhgf3KcLgpjERFplMK4FfhWbY25uSQERxt/iIhIIxTGrWRkn2i2lYVRuX+nu0sREREPpzBuJSNSotntdMG3eC+Ul7q7HBER8WAK41aS0CUYJyTOfqOuahEROQqFcSuK694TgNL8HW6uREREPJnCuBX1Se4DwMaNG91ciYiIeDKFcSsa2DcFgG1bFcYiInJkCuNWFBLZjUoMebu2ursUERHxYArj1uTrx0H/SCjMZm+RZlSLiEjDFMatzITFE8M+5m/IdXcpIiLioRTGrSyoSze6+u5jXqbCWEREGqYwbmU+ofF0893P3MxcHMdxdzkiIuKBFMatLTSesIq97NxXxKbcIndXIyIiHkhh3NpC4vChkij2M2+9uqpFRKQ+hXFrC40H4ISIYuasUxiLiEh9CuPWFmLDeGTXChZszKOsotLNBYmIiKdRGLe2UHtYxJDIEgpLyvlp2z43FyQiIp5GYdzaqk5uSgkuwsfAHC1xEhGRwyiMW5tfIHTqQtDB3RyfEMEP2vxDREQOozBuCyHxUJhNenIkP27bx8HSCndXJCIiHkRh3BZC46BgF+nJUZRVOCzbutfdFYmIiAdRGLeFqpZxWs8u+BhYsDHP3RWJiIgHURi3hdA4KMwmNNCP47uHs2DTHndXJCIiHkRh3BZC4qGiFA7uZXhyFD9u3UdxmcaNRUTEajSMjTFTjTG7jTErjvC6McY8aYxZb4z52Rgz2PVltnNVa43tuHEkpRWVLNW4sYiIVGlKy/gVYPxRXp8ApFT9mQw82/KyOpiqXbgo3EVar0h8DCzcqK5qERGxGg1jx3HmAEdLjguA1xxrARBhjOnqqgI7hKr9qSnIJizInwHdwjWJS0REarhizLg7sK3W91lV16Ra6KGWMUB6ciTLtmncWERErDadwGWMmWyMyTDGZOTk5LTlR7tXQGcICIWCbACGJ0VRWl7Jj9qnWkREcE0YbwcSa32fUHWtHsdxXnAcJ81xnLSYmBgXfHQ7EhpX0zIemhSJ0XpjERGp4oow/hi4tmpWdTqQ7zjOThc8t2MJia9pGYd38mdAtzBN4hIREQD8GrvBGPM2MAqINsZkAQ8D/gCO4zwHfAacDawHDgA3tFax7VpoHOxYVvPt8KQo3liwheKyCoL8fd1YmIiIuFujYew4zhWNvO4At7msoo6qVssYID05ipfmbeKnbfsYnhzlxsJERMTdtANXWwmNg7IiKCkAYFgvO268UFtjioh4PYVxWwk5tNYYIDzYn37xYZrEJSIiCuM2U70lZtWMarBd1Uu27KWkXOuNRUS8mcK4rdS0jA+F8fDkSErKK/k5K99NRYmIiCdQGLeVWodFVBtevd54g7qqRUS8mcK4rQRFgG9gnW7qiOAA+saFahKXiIiXUxi3FWMgrBvs+BEcp+ZyenIUGVv2UFpe6cbiRETEnRTGbWn4LbB5Lvz4Vs2l9OQoissq+TlL+1SLiHgrhXFbGnYL9DwVZj4A+Vn2UlIkoPXGIiLeTGHclnx84IKnobICPr4DHIfIzgEcFx+q9cYiIl5MYdzWIpPgrD/Dhm9hycuAnVWdsXkvZRUaNxYR8UYKY3dIuxGSR8EXD8HezaQnR3GwrELrjUVEvJTC2B2MgfOfAh9f+Og2hvWKAHS+sYiIt1IYu0tEIoz7X9gyj6hVr5EaF6JJXCIiXkph7E4nXQ0pZ8FXD3N2twNkbN6jcWMRES+kMHYnY+C8J8EvgGuzH6G4tIzl2zVuLCLibRTG7hbWFSb8ncg9y7jR9zMWblRXtYiIt1EYe4ITLoPjzuU+//fYunapu6sREZE2pjD2BMbAuU9Q5hvMxB2Pkr2/2N0ViYhIG1IYe4qQWEqH38Zgs44XP1/g7mpERKQNKYw9SJfjRgGw7edZrN9d4N5iRESkzSiMPUm3QTi+gaT7ZfLIzLXurkZERNqIwtiT+AViup3E+LAtfLUqm4zNmlktIuINFMaepsdw4orWkBgC//vZahzHcXdFIiLSyhTGniYxHVNZxsNppSzduo8vVma7uyIREWllCmNPkzgcgNHBm+gd05lHv1hDubbIFBHp0BTGnqZzFESl4Ju1iN+MP46NOUW8m7HN3VWJiEgrUhh7osThsG0hY/vFktazC//8OpMDpeXurkpERFqJwtgT9RgOB/dg8tbz4NnHkVNQwotzN7m7KhERaSUKY0+UmG6/blvAkJ6RjBsQx/OzN5BbWOLeukREpFUojD1RdAp0ioStCwG4f/xxFJdX8u9vMt1cmIiItAaFsScypmrc2O5R3TsmhElDE3lz4VY25xa5uTgREXE1hbGn6jEc8tZDUS4Ad41Jwd/Xh79/oW0yRUQ6GoWxp6oZN7Zd1bFhQdw8MolPl+9k9c79bixMRERcTWHsqbqdBL4BsPXQcYq/GJFEgJ8P7yza6sbCRETE1RTGnso/CLoOgm2Lai5FBAcwbkA8H/24g+KyCjcWJyIirqQw9mSJw2DHMig/tKTpsrQE8g+W8eUq7VktItJRKIw9WY90qCiBHT/WXDq1dzTdIzoxbbG2yBQR6SgUxp6s6tCI6iVOAD4+holpCcxbn8u2PQfcVJiIiLiSwtiThcRCZHLN5h/VJqYlYgy8tyTLTYWJiIgrKYw9XWK6Xd7kODWXukd0YkSfaN7P2EZFpXOUN4uISHugMPZ0PYbDgVzI21Dn8qShiezIL2b++lw3FSYiIq6iMPZ0tQ6NqG1s/zgigv111rGISAegMPZ00akQFFFn8w+AQD9fLhzUna9WZrO3qNRNxYmIiCsojD2dj49db1xr849qk4YmUlpRyYfLtruhMBERcZUmhbExZrwxZq0xZr0x5oEGXu9hjPnOGLPMGPOzMeZs15fqxRKHQ+5aOLCnzuV+XcM4ISGcaRnbcBxN5BIRaa8aDWNjjC/wNDAB6A9cYYzpf9htDwHTHMc5CbgceMbVhXq1HtXjxvVbx5elJbJmVwE/Z+W3cVEiIuIqTWkZDwPWO46z0XGcUuAd4ILD7nGAsKp/Dgd2uK5Eodtg8PGrN4kL4PxB3Qjy92GaJnKJiLRbTQnj7kDtn/RZVddq+yNwtTEmC/gMuMMl1YkVEAxdT6y3+QdAWJA/Zw/sysc/7uBgqQ6PEBFpj1w1gesK4BXHcRKAs4HXjTH1nm2MmWyMyTDGZOTk5Ljoo71EYjrsWArl9WdOXzY0kYKScj5fsdMNhYmISEs1JYy3A4m1vk+oulbbjcA0AMdxfgCCgOjDH+Q4zguO46Q5jpMWExNzbBV7qx7DobwYdv5U76XhSZH0jArmXR0eISLSLjUljBcDKcaYJGNMAHaC1seH3bMVGANgjOmHDWM1fV3pCJt/ABhjuCwtkYWb9rA5t6iNCxMRkZZqNIwdxykHbge+AFZjZ02vNMb82RhzftVt9wI3G2N+At4Grne01sa1QuOgS696m39Uu2RwAj4GTeQSEWmH/Jpyk+M4n2EnZtW+9oda/7wKONW1pUk9icNhw7f20Ahj6rwUHx7EqL6xTF+axT1jU/Hz1X4uIiLthX5ityeJw6EoB7JXNPjyZWmJZO8vYU6mRghERNoThXF7cty50KkLfDgFyg7We3lMv1iiQwJ4Z5G6qkVE2hOFcXsSGgcX/8e2jD+7r97L/r4+XDI4gW/W7GZ3QbEbChQRkWOhMG5vUsbCyHth2evw41v1Xr5saCIVlQ7Tl+jwCBGR9kJh3B6N+i30Ggkz7oHslXVe6h0TwrCkSN5dvFWHR4iItBMK4/bI1w8ueQmCwmDadVBSUOfly4cmsjnvAAs37TnCA0RExJMojNur0DgbyHs2wMd32uVOVSYM7EpokJ925BIRaScUxu1Z0kg44yFY+QEsfrHmcqcAXy4c1J3Plu8k/0CZGwsUEZGmUBi3d6feDSlnwcwHYfuSmsuThiZSUl7Jf3/SRC4REU+nMG7vfHzgouchNB6mXQ8H9wIwsHs4A7uH8faibXUncpUUQoVayyIinqRJ22GKhwuOhImvwNTxdkOQC5+FvZv4TcJKFmZksPetV4gszoI9G6Fot52Jff0Md1ctIiJVFMYdRUIanPVXmPkbeDQJgJHASH/I3xwD3VIh9SwozofVn0D2Kojr796aRUQEUBh3LMNvgYDOULwPuiRBZDIPfFfAjNX7WHTVGIID/KAoD9bOhKWvwYS/ubtiERFBY8YdizEw+Bo45Q7ody7E9eeS9BQKS8r59Oed9p7OUfa1n9+BMm2ZKSLiCRTGHVxazy4kx3Suu+Z48HV2otcajRuLiHgChXEHZ4zh8qGJZGzZy/rdVTt1JZ0OET1h6avuLU5ERACFsVe4eHACfj7mUOvYx8d2Z2+aY2dYi4iIWymMvUB0SCBj+8cxfel2Sssr7cVBV4HxgaWvu7c4ERFRGHuLy4f1YE9RKV+vzrYXwrpByjj48U2oKHdvcSIiXk5h7CVG9Imme0Qn3qkzketaKMyGzC/cV5iIiCiMvYWvj2FiWgJzM3PYtueAvZhyFoTE2zXHTZW7Hp4dAduXtk6hIiJeSGHsRSamJQLw5sKt9oKvH5x0FWR+Cft3NP6A0gMw7VrIXg4/vdOKlYqIeBeFsRfpHtGJcbXleDMAACAASURBVP3jeW72Bq55aSErtufDSVeDUwnL3mz8AZ/fB7tXQZdesG5mnTOURUTk2CmMvcw/Lx/EQ+f0Y8X2fM799zxun7mPgwkjYNlrUFl55DcuexOWvQGn/RpOuRP2bYHcdW1XuIhIB6Yw9jJB/r7cNDKZ2feP5o4z+vDN6t08sPkk2LeVvSu/avhN2avg03vtaU+jHrRjzQDrNPFLRMQVFMZeKizIn3vP6svs+0fRZfDF7HVCWPDeEzz+5VoKimudd1xSaMeJA0PhkpfAxxciEiF2gB1rFhGRFlMYe7nY0CD+ePFgfAZdwVifxbzx7VLOeHw2m3OL7JjwJ7+CPRvg0qkQGnfojanjYMv3cHCf+4oXEekgFMYCQPipN+JHOTNOy6K8opJfvLqYg9//B1a8D6N/B0kj674hdRw4FbDhW/cULCLSgSiMxYrtBwnD6L7xPZ67ajChe1bi99Vvqew9BkbcU//+hKHQqYu6qkVEXEBhLIcMvhZy1zK8Yilvhj9LjhPK/wTejWNM/Xt9fKHPWBvGlRVtX6uISAeiMJZDBlwEAaHw7tWEFO/k24GP8NLS/bw0b1PD96eOgwN52o1LRKSFFMZySGAIHH8JVJTAmX/iyksmMmFgPP/z2Wq+XpVd//7eZ4DxtRuAiIjIMVMYS11jHrZLmE6+DR8fwz8uG8Tx3cO5851lrNqxv+69wZGQOFwHTYiItJDCWOoKjoTjL4WqceJOAb68eG0a4Z38ufHVxezeX1z3/tSzYNfypu1tLSIiDVIYS6Niw4J48bo08g+WcdNrGRwsrTVhK3W8/arduEREjpmfuwuQ9mFAt3D+dflJTH49g7vf/ZHrT+3F/oNl5B/ozPigruz+/gNe3zGM/cVlFBSXc8Ggbpx7Qjd3ly0i0i4ojKXJxvaP47cT+vE/n61m5spdNdcP+g3k0oNz+GTPRoKCOlNR6fDN6mw6B/gx+rhYN1YsItI+KIylWW4amcTgnl0oKasgrJM/4Z38idzhQ/D7X7Hkqk6QcgZFJeVMeuEHbntrKe9OPpnjE8LdXbaIiEfTmLE0izGGIT27cEqfaAZ2DycxMpjOfUeBX6eaWdWdA/2Yet1QugQH8ItXF5O194B7ixYR8XAKY2k5/06QPMquN3YcwE76evmGoRSXVXDDy4vJP1h21EeIiHgzhbG4RupZsG8r5Kw9dCkulOevGcLmvCKmvL6E0vJKNxYoIuK5FMbiGiln2a+HbQBySu9oHrnkBH7YmMcD03/GqWo5i4jIIQpjcY3wBIg7vsH1xhcPTuCesal8sGw7T3yd6YbiREQ8m8JYXCf1LNi6AA7urffSHWf04bK0BJ78JpNpGdvcUJyIiOdqUhgbY8YbY9YaY9YbYx44wj2XGWNWGWNWGmPecm2Z0i6kjAOnAjZ8W+8lYwz/c9HxjEyJ5rcfLGduZo4bChQR8UyNhrExxhd4GpgA9AeuMMb0P+yeFOBB4FTHcQYAd7VCreLpEtKgU+QRt8b09/XhmasG0yc2hMmvLeG7tbvbuEAREc/UlJbxMGC94zgbHccpBd4BLjjsnpuBpx3H2QvgOI5+ynojH19IGQuZX0FlRYO3hAb58/qNw0mO6czNr2YwfUlWGxcpIuJ5mhLG3YHag3xZVddqSwVSjTHzjTELjDHjXVWgtDOp4+DgHsjKOOItMaGBvDM5neHJkdz73k88O2uDZlmLiFdz1QQuPyAFGAVcAfzHGBNx+E3GmMnGmAxjTEZOjsYMO6TeY8D4wsJnoezgEW8LDfLn5euHcf6J3Xhk5hr+PGMVlZUKZBHxTk0J4+1AYq3vE6qu1ZYFfOw4TpnjOJuAddhwrsNxnBccx0lzHCctJibmWGsWT9YpAk65A1Z+CM+kNziZq1qAnw//nDSIX5yaxMvzN3PnO8soKW+4e7vZKiuO2FUuIuJpmhLGi4EUY0ySMSYAuBz4+LB7PsK2ijHGRGO7rTe6sE5pT8b+Ca6bAT5+8PpFMP1mKGy4J8THx/D7c/vx4ITjmPHzTm6YuoiiLUth6WtQcYxbaFZWwKvnwcsTjv0ZIiJtqNFTmxzHKTfG3A58AfgCUx3HWWmM+TOQ4TjOx1WvnWWMWQVUAPc5jpPXmoWLh0saCVPmw9zHYd4TsP4rGPsXOOlqMKbOrcYYbhkII3d8T8CaD+n8clXHS0E2nH5f8z978YuwZb7959mPwhm/a+FfRkSkdRl3TZxJS0tzMjKOPMlHOpDda2DGXbD1B+g1Es59AqJToGAXrPgAlr8HO5YCsC9mKP/KPpHT/FZyOhn43DIH4vo38gG15G+Hp4dB4jDoHGuffeOXdtmViIibGWOWOI5T7weSwljaRmUlLHsNvvqDndjV7STYtghwIP4EOH4iDLwYwhP4ads+fv/2LKYW3cH+gHgOXjeTAQlRTfucd66C9d/ArT9AcCQ8cwr4B8EtcyEguBn1VsCmOfaXB18d+y0irnGkMNZ2mNI2fHxgyPVw22IYcBGUFsHp99vvp8yFU++0+1sDJyZG8P4957P8hN+RXLaOT577LfdO+4md+UeenQ3A6hmwZgaM+g1EJkFQOFz4DOStt78ENFVlBfz3Nnj9Qlj0wrH/nUVEmkgtY/FcjkPZ21djMr/kvLL/YyPduXFEElNG9SYsyL/uvSUF8PRw6NQFJs8C31qvz3wQFjwDV38AfcYc/TOrg/intyEowraub19if5kQEWkhtYyl/TEG//OfwC+oMx8lvMXZA2J4ZtYGRv19Fq9+v7nu+cjf/hX274Dz/lU3iAHG/AGi+9qQPbDnyJ9XWQEf3WqDePTv4OzHYM9G2Hjk5VkiIq6gMBbPFhILEx4lcNcSnuj5A5/cPoK+caE8/PFKznh8Fu8s2krZ1gxY+DwMvanhiVr+neDiF6AoBz77dcOfUx3EP79jg/j0+6H/BXYS2KL/tO7fUUS8nsJYPN/xEyF1Anz7V44PyuGtm4fz8g1DiQoJ5Hcf/Miml2/iQGAMpaMeOvIzug2C0x+AFdNh+ft1X2soiAH8Auw497ovYO/m1vrbiYgojKUdMMYuh/ILhI9vxzgOo/vG8tGtp/BF+gpSnU3cXXAlZzy1lHcWba3bfV3biLuhexp8eq/t0oaqIP5lVRA/dCiIq6XdAMYHFr/Uun9HEfFqCmNpH8K6wrj/s2uVq2Y4m31b6bPy3zh9J3D5tbcSFRLIAx8sr+m+rhfKvn5w0fNQXgL/vR0qyquC+N2qIG5gg5GwbtDvXFj2+lH32hZpVGUFfPl72LvF3ZWIB1IYS/sx6EroMxa++ZOdWPXpvWB8MGc/xujj4vjo1lNquq8f+GA5U95YUv8Z0X3grL/Ahm/g+dOOHsTVht4MB/faLm6RY5W9Er5/EjLUyyL1KYyl/TAGzvunPRXq1fPtFptnPFSzPtkYU9N9/asxKXy7Zjc/bdtX/zlDb7KnS+1e2XgQA/QaATH9bItcRz3KscpdZ79unOXWMsQzKYylfQlPgHH/A/nb7C5ewybXu8UYw00jkwgL8uO52RvqP8MYmPgKXP9p0/a+NgaG3QQ7fzrqOc0iR5Wz1n7d+TMUaet+qUthLO3P4GvthK6Jr4CPb4O3hAb5c83JPZm5chcbcwrr3xAUZlu8TXXC5RAYph255NjlrAEff8CBTbPcXY14GIWxtD/GQNovoEuvo952/SlJ+Pv68MIcF5zmGRgCJ14Bqz6Cwt0tf554n9x10PsMCAyHDd+5uxrxMApj6bBiQgO5LC2BD5ZuJ3t/ccsfOPQmqCiFpa+2/FniXSrKIW8DxPazx4tunKX5B1KHwlg6tMkje1NeWcnUeZta/rCYVEgeDRkv2x+uIk21dxNUlkFMX+g92s552OOCHhvpMBTG0qH1iArmnBO68ebCreQfLGv5A4fdDPu3w9rPWv4s8R7Vk7ei+9pf6AA2aM9zOURhLB3eLaclU1hSzhsLXLDZQup4CE+ExU3Yr7ooVxs8iJVbHcYpEJkM4T20xEnqUBhLhzewezinpcbw8vzNFJdVtOxhPr528timObB7TcP37FgGH9wC/+gH/zoB3rsBcta17HOlfctZB6Hd7Cx+Y6D3KNg0V8MdUkNhLF5hyunJ5BaWMH1pVssfNvha8A2s2zquKIeVH8JL4+CFUbBmBgy5AUbcYw+aeGY4fDhF44TeKnetnXNQLXkUlOTbX9xEUBiLlzg5OYoTE8J5Yc5GKipbOIu1czQMvBh+esd2Q897Av51Irx3PRTusnto37MKzn4UznwY7voZ0m+1Yf3UUPj4Tti3zSV/L2kHHAdyM+14cbWkUYCBjVriJJbCWLyCMYZfjurNlrwDfL5iZ8sfOOxmKC20Ifz1HyEqGS5/G+5YCiffCkHhh+7tHG13DfvVT7aL+6e34d+D4bP7oGBXy2sRz7Z/u/3fSkytMO4cBV1P0Lix1FAYi9cY2z+e5OjOPDtrA05L13h2H2KDdfC18Mvv4bpP4Lizj7gjGACh8XD2321gn3iFPZbxycF2e0TpuKpnUtcOY7CzqrctgpIGdogTr6MwFq/h62O45fRkVu7Yz7z1uS1/4LlPwPlPQtyA5r0vItG+7/bFEBgK02/S8YwdWfUBEdGHh/Eou/Z4y/y2rkg8kMJYvMqFJ3UnLiyw4QMk2lpUb7jwGTu556s/uLsaaS05a6BTFztcUVuPk8EvSFtjCqAwFi8T6OfLjSOSmL8+j5+zGjhesa31GQPDf2kPoMj8yt3VSGvIWWdbxcbUve4fZANZ48aCwli80BXDehAa5MezszygdQxw5h8htj98dKvdKEQ6lsOXNdWWPApyVsN+F0wqlHZNYSxeJzTIn+tP6cXnK3bxp09WUlpe6d6C/IPg4v9A8T677EkHCHQcRXlwIK/+eHG13lVbY6p13Pbyt0P2SndXUUNhLF7pzjEp3DgiiZfnb+byF35gZ76bJ1DFD4QxD8PaT3UqlKfK29D8TTpyjzCTulrc8RAcrfXGba2iHF47H549BZ4bCYv+AwfdO2ylMBav5O/rw+/P7c/TVw5m7a4CznlyHvMy3dxFnH4rJJ0OMx+E3PXurcXVKlxwSIe7/fc2ePuK5vVc1BwQcYRuah8fSD5dRyq2tWWvQ956GDbZfv/Zr+HxvvDBZNg8zy3/LRTG4tXOOaErH98xguiQAK6ZupCnvs2ksqU7dB0rHx+48FnwDYAPbu4YAQaw5BV4pBds+d7dlRy7wt2wdQEU7DwUsE2Ruw78g+3hIkeSPAoKs2H36pZWKU1RegBm/Q0S02HCozBlLkyeDSddDWtnwivn2E155v6jTTflURiL1+sdE8JHt53KBSd247Ev13HTaxnsO1DqnmLCu8N5/4IdS2H2I+6pwZXyNtiWfmmh3S60ve44tvYzoOqXtOaM7+ashag+9hetI6k+UlFd1W1j4XN229oz/3hohnu3QXDO43DvGrjoeXuoxzd/gn/0b7Ota/3a5FNEPFxwgB9PTBrEkF6R/PmTlZz773k8e9UQ+saHklNYwu79xeQUlLC76k9OQTE5BaUM7dWF60/tRaDfUXbeaq4BF0LmVTD3cehzJvRId92z21JlhZ0h7uMPV0+Hd6+xgXzdJ+Dr7+7qmmfNpxDRE4yPDeP0KU17X+46u3zpaCISbWBv+A5Ovq3FpcpRHNgD8/5pj0Lt2cB/l4BgOPFy+ydvA6z/2v73aQMKY5EqxhiuSe/J8d3Due3NpZz/9LwGh46MgajOgYQF+fH16mzeXrSVh87pz5h+sZjD15Ieq/F/s2NXH9wM4x+B8mIoL4Hyg/Zr2cFD3xsfCOgM/p3t19p//DtDeAKExrmmruZY8AxsW2BbGn3OhPP/DdNvtBucjP+/tq/nWJUU2AAeNtm28JdPtxOAfBv58VlSCPnbIOa6xj8jeRT8+BaUl4JfgAuKlgbNewJK9sOYJmyyE9Xb/mkjCmORwwxKjGDGHSN4ef4mfH18iA0LJDY0kNjQIGLDAonqHICfr+12nLMuhz99spKbXsvg9NQY/nBef3rHhLS8iKAwu9zplbPhnSuOfJ9vIDiVdlvFI94TYLfuPOnqltfVVLvXwDd/gb7nwAmT7LXjL4WsDBvS3YfY79uDzK+gohSOO8eO7S55xQ4jJA47+vvyMu3XIy1rqi15NCx+EbIWQa8RLS5ZGpC/3W6uc+Llzd/Ctg0ojEUa0KVzAPec1fgP0dNSY5h512m8+v1m/vV1JuOemMMNp/bizjEphAa1sCu2x3C480c4kGu3Taz+41/11Tfw0FhkeSmUFUFpkZ2gUlpY9c9FsPBZOxM4exWM/XPjLbqWqiiHj35pW+bn/bPuzlNn/cUuD/r4DvsDMbZf69biCms+tcuPEodXLX8xsHF242F8pAMiGpI0Eoyv7apWGLeOWf9nf3Ed9aC7K2mQJnCJtJC/rw83jUzmu/tGccngBF6ct4nRj81mWsa2ls/MjkiEbifZ0IpMgrCudp9j/051JwX5Bdjr4Ql2t6fug+0P+L7j4arpMHwKLHga3poIB/e2rKbGzH/CthzP/QeExNZ9zdcfLnvVHpDx7tVQnN+6tbRUeSlkfgl9J9gTuTpHQfzxTZvElbMWfPwgMrnxe4PCbW+BJ0ziKin0/P8uzZWzDn58E9JuhC493V1NgxTGIi4SHRLII5eewH9vO5UekZ24//2fufS579mcW+Tewnz9YMIjcN6TsGku/GeM/eHUGnYth1mPwICLYcBFDd8TGg8TX4E9m+wEL09eX7t5jh1jPO7cQ9eSR8G2hbbX4Why19kgbupkteRRttegtX9ZOhrHgbcug38Nssc7dhTf/tnOnzjt1+6u5IgUxiIudkJCBO9POYXHJ57I+t2FnP3kXKYt3tbyM5Rbash1diZzyX54cYzrD6YoL4UPp9gW+jmPH/3enqfAWX+FNTNg/j9dW4crrfnU/hBPHnXoWvIoO0a/9Yejvzdn7ZE3+2hI79G2G3XT3ObX6SobZ9kjHStK4dXzYPUM99XiKlkZsPoTOOWO+idneRCFsUgr8PExXDIkgZl3ncYJCeHcP/1nfvnGUvYWuWn9crWeJ8PN39muujcnwvwnXdcynf0IZK+wZzUHRzZ+f/ovYeAl8M2fPXNv5spKWPMZpJxpx+mr9TjZToo7Ws3lpbBnY9PGi6slDIWAEFj7+TGX3CKOY/8bhnaD2xbZMf1p19iJZe2V48DXf4TOMR6/bExhLNKKukV04q2b0nlwwnF8syab8f+a4/5tNyMS4RdfQP8L4Kvfw4e32JmmLZG1xC4bOfFKO77aFMbYrvPovvD+L9psc4Um27HUbg5Ru4sa7FrUxOFHD+M9G8GpaNpM6mq+/jDoSvjpbdg8/5hKJn87fP4AFOY0/72b59nW/oi77OYz130CfcbCp/faX5jc3bNzLNZ/A5vnwmn3Q6ALVjm0IoWxSCvz8THccnpvPrz1VEIC/bj6pYX8dcYqSsor3FdUQGc7bjv6Ifj5XXiiP7w0DhY81/zj/MoOwkdT7Fhwc9cPB4bApDfs1p/vXNn4OGxbWv2JnYCVMrb+a8mn2/HxoryG31tzQEQzuqnBHhbSpZf991lS0Lz3VpTZTVUWPgszH2jee8G2ikPiYXDVuuiAznD5W/b7uY/bGfLtaYvWykrbKu7SC4Zc7+ZiGqcwFmkjA7uHM+OOkVyT3pMX523igqfms2bXfkrLKyktr6SsopLyqj8VlQ4VlU7r7pNtDJx+H9yxFM54yC6Hmvkb+Ec/mDoBFr5Qf/vKsmJ77NzKj2D23+3G+i+MspOVzv83dIpofh3RfeDSqbaL+8Mp9oeoJ1jzKfQaacfAD5c0yn7dNLvh91ZPkGvOmDHYX04ueg7ys+CL3zbvvV//0a5TTjodVrxvd49qqs3zbQvy1F/V7ZL39bPbs47+nW2xvzmx+b8kuMuK6ZC93P7C2Q42UjHumlSSlpbmZGRkuOWzRdztm9XZ3P/+z+Q1YQx5YPcwxhwXx5n94hjYPcx1u3w1JGcdrPoIVn4Iu1cBxk628u8EuZmwbys1ezQDhCXYMO13Pgy9sWWf/cPTNoBO/w2MbmYQuVrOOnh6KJz9GAy7uf7rFeXwaJKdMX7+k/Vfn34TbF0Idy8/ts//6mE7se3KaZA6rvH713xqexaG3gzj/sceDVhRBrcusN3qjXntArsO/Vc/Hfn+pa/DJ7+yY8lXvWd7QjxVaRE8c7LdPGfynKPvDd7GjDFLHMdJq3ddYSziHjkFJXy4LIuyCqdmpnX1/x2r/19ZUl7Bgo17WLp1L44DcWGBnHFcHGf2i+XUPtEE+btwT+zD7V5jg3n1DDDYVl5UCkRX/YnqY7syXcVx4OPbYdkbtqU88BLXPbukEDZ8a/ckbkorae7jdpz0ntUQ1q3he96+EnavtAF2uOdG2jXWV08/tnrLS+CF0VCUYwO1c9SR792zCZ4/3a5Dv/FL8Au0M7JfPRdG3ANnPnz0z9q6EKaeZWe3n3LH0e/N/AqmXWfrufk7z52d/PEd9peH62d43CYqCmORdiyvsITv1ubwzeps5qzLoai0giB/H0b0ieaq9J6M7hvb+EPag/IS20rbsQxu+NxuXtISlRV2s4dv/2q3shxyve12bcx/zrBfb/72yPcsfB4+v9/ukhaZVOszK+F/u0HaL2D8/x577buW20A+7myY+GrdncyqlZfAS2fZQJ4yx46PVvvoVjsf4Ja5ENf/yJ/z+sWw8ye46+em/XKVlQEvT7CTuy5/s+G63GnlR/DedU37RcQNjhTGTWq7G2PGG2PWGmPWG2OOODPAGHOJMcYxxtT7IBE5dlEhgVw6JIFnrx7C0j+M5bVfDGNSWiKrduznhpcX8/iXa913DrMr+QXaCV2dY223a3Mnk9W24Vt4/jTbSoroaffmXvIKZLx89Pft3wHbl9i9qI8meZT9evi4cf42e4BHcydvHS7+eBj9IKz6Lyx/v+F7vvgd7PwRLnq2bhADjP0LBIbBjLuOPA6flQEbvrEt4qb2ciSk2YMW1n4KS19t8l+nTezbBp/cCd3T3D/U0UyNhrExxhd4GpgA9AeuMMbU+zXLGBMK/ApY6OoiReSQQD9fTkuN4U8XDOTbX49i4pAE/v3tem58dTH5B9vRbNcj6RwNV75jJwq9c6Wdrd0cu9fYiUavX2SfMfEV23173pP29KjP7jv67lJrPrVfD1/SdLjoVAjtaveprq16T+rmLGs6klN+BQnD4LN77S8Jta2YDov/Ayff3vAvDp2j7PjxtoVHDs3Zj0CnSBh6U/PqSr/NThSb+SDkrm/ee5uiYFfzNz+prLATCisr4ZIX290xnU1pGQ8D1juOs9FxnFLgHeCCBu77C/AIUOzC+kTkKIL8fXn00hP4ywUDmJuZy4VPzyczu53Mdj2auAH21Kody+whF00ZTivMgRn32MlLWxfaluHti+0kK2Ps3tKXvGj37373mvozxaut+dSOjTe2YYcxNpA2za7b8sxtxgERjfH1s7OrK8rq/nvIXQ8f32mD+sw/Hvn9J15hZ4R//TAUZNd9bftSu+/2ybc1fw2ujw9c+Kzd/OSDm1275Mlx4L0b7Jj3Vw83fXb9nMdg6/d297fawwbtRFPCuDtQezV+VtW1GsaYwUCi4zifHu1BxpjJxpgMY0xGTs4xLEoXkXqMMVxzci/enpxOQXE5Fz49n5krWtC96ymOO9uO+a2YDnMfq/tacb7daOSnd+x48LTr4MmTbDf00BvhzmVw6p2227u2Tl3s2tmSAph2rd0pq7aD++wSn8a6qKslj4IDeXZZVrWctfaUp6bsQtYUUb3taVsbvoWMl2xPwXvX2SCc+PLRW4DG2OMzyw7WXyo1+1EIirDnNB+L8O52/H3HUpj1t2N7RkM2fGNDNf4EO6P8vevsSWRHs3UBzP6bPa7zxEmuq6UNtfgsNWOMD/AP4PrG7nUc5wXgBbATuFr62SJyyNBekcy4YwRT3ljClDeWcvvoPtw9NhVfHw+bYNMcp95lu52//av9un+HPSe4qNYv88bXbu+ZOs4ui2psrDauP1z4tN0gY+ZvbFhVy/wSKssb76Kulny6/bppNnQ9wf5z7jrXtIprG3qTbbF/+XtY/60N/6um21Z+Y6JTYOS99gjBQVfYrvqdP8G6z+364aCwY69rwIWQeRXM+4d9bs+Tj/1ZYFvF3/wFInrATd/Y84e/fMiuu77iHQiNq/+eg/tg+s32PWc/Vv/1dqIpLePtQGKt7xOqrlULBQYCs4wxm4F04GNN4hJpe/HhQbx7SzqT0hJ56ruqceQD7Xgc2Rjb+ko5q2r7SceG7pl/si3c2xbB73bZlvClLzV90tSAi2zQZ0yFJbXGU9fMsLtQdR/StOeEdbNjx9VbYzpO8w+IaApj4IKnbSt47acw8td2z+ymGnG3XYo24x7bypz9KASGH3uruLYJj9gg/GByy49eXP2JnZB2+gN2Cdopt9sZ2zlr7OEm2Svr3u84MONuKNgBl0xt2S8WbtaUMF4MpBhjkowxAcDlwMfVLzqOk+84TrTjOL0cx+kFLADOdxxH65ZE3CDQz5e/XXI8f71wIPPX53LhM/PZld+Op3L4B9lNJu7fAL+YaUNpxF22Kzmm77HvrjTmD9D7DPjs17Btsd1dLPNr2z3enE0ikkfBlu9tl3dRDhTvg5jjjq2mownvbtdfp98Gox5s3nv9AuHcf8K+LfDhZPtLR/ovj23HtMMFhtrx/f3b7eS4Y1VZYXtAolNtd3O1486xy9wqy+2WrbVPG/vxTVj5gZ05ndDEX6A8VKP/i3Mcpxy4HfgCWA1McxxnpTHmz8aY81u7QBFpPmMMV6f35M2b0skpKOHKFxeQU1Di7rI8i48vXPKSnRE97Rq7JresqOnjxdWSR0HZAchafGgmdUuXNR1JnzPt2mXfYxhhTBoJg66yrc/AMEif4rq6EofBaffZf4dHAQbdwgAAFmpJREFUWobVmOXv2clvo39X/+/XbZDtto7sZc9bXvQfO4nts/vtBLVT72rxX8HdtOmHSAe3aNMerpu6iB6Rwbw9OZ3Izp6/T2+b2rUCXhoL5cX2CMP7NjSvtX1wn90ac+Sv7Zjmp/fC3atsS9bTFOXBi2dA2o12gpsrVZTDy+PtVqK/nG9PB2uq8lJ4Kg2CwmHy7CP3TJQU2q1G131ul2SB/awj7ZLmgVq06YeItF/DkiJ58bo0NuUVcc1LCzvGWmRXih8IFzwFTqUdm25ut3enCOg22I4b56yDgFDPDYfOUXDHMtcHMdjW7MUv2KMjP5xiu52batnrtgt9zB+OPkQQGGLHkNNvtePTFzzluf+um0lhLOIFTu0TzfPXDGFddgHXTV1EYUm5u0vyLAMvgas/aP4RkNWSR9ldu7Zn2NnLnrZFZG2teWhCZLKd0LVlnj1UoqIJ/zsrOwhz/g6J6bYbvjE+vva/0wNbmz+k4MEUxiJeYnTfWJ66cjDLt+f/f3v3HV51efdx/P3NySaLvULYKyDLlILgoiioIK5StCp1Vhyt1Tqettbqo7XWp65KrZXSqkVt1SoCjjqQIlOWjACSAAFCIIwkJGQn9/PHOSoqkQRO8stJPq/rynXOb5xzvtwXv/M5v3XfXP23TyguVyB/Ra/v+Qd3OB49TvfvEWavDP5tTaFmyA/9549Xv+C/R7jiGBcPfjIDCnP8e8V1+RFT145KGjmFsUgzMm5ABx7/wRBWZB3kuudXUFpRh0OJUrPk4RAe438e7NuaQo2Zf3zs8Q/7r9qedQmUHjr6uqWHYOGj/qvau41q2DobGYWxSDMzcXAnfn/JYBZlHGDaP1ZSXlnL7galZhHRX3Z40dz3jD834gb/LU87lvi7tiw6Sq+LS5+GkoMw5p6Gr6+RURiLNEOXnJzMby88ifmb93HLS6s4rHPIJ67Hmf7Hdv29raMxGTQZprzkv7Bt5jjIy/pyWfFBWPxH6D/xxIfKbAIUxiLN1GXfTeHeiam8u2EvIx76gAfmprPz4DH6AJaaDb/O30Vlqx5eV9K49DkbrnwDivf7A3lvun/+osehvMh/X7HoPmOR5m71jjxmLtrOW+tycM5xdmoHrhrVjeHdW2HHuKCmqKyS9N2HaNUigl7t4huoYglJezfACxf57+c+/4/+7jNTJ8FFz3hdWYOq6T5jhbGIAJBTUMILS7J4cfkO8osrGNApgatGdWfi4I5EhfvILy5nw+5DrM8uYP3uQ2zILmDbgcM4B4kxESy6ewxxUSc89ow0ZXnb/eNMH9wKYeFw84qQHO7wRCiMRaRWSsqreH11Nn9btI0tuUW0iYskOsLHrrySL9bpnBTDwM4JDOyUSFKLSO55Yz2/OLcf15/W08PKJSQU5cKrV0PKSBjT/A5RK4xFpE6cc3ycsZ+Xlu/AzBjYKZGBnRMY0CnxG11qXj5jGZv3FrLwzjOJjvDV+bMKSipIiA4/5mFxkVBXUxjrmJKIHJWZcWrvtpzau+0x173pzF5c+uxSXlm5iytGdK3T52zeU8jEpz5m4qBOPHLJIMJCefxlkeOkq6lF5ISN6NGKYSlJPLMgk4qq2t+37JzjvjkbcM7x2qpd3DN7PV4drRPxksJYRE6YmXHTmb3YlVfCm2t21/p176zfw+LMA/x6Qio3nN6TWct28Nu3NiqQpdnRYWoRCYox/drRr0M8f/oogwuHdj7m4eaS8ioemLeRfh3iuXR4Cr4wo7SiimcXbiM2MpyfndXMu5WUZkV7xiISFJ/vHWfuO8x/0vccc/1n/ptJdn4Jvzl/AOG+MMyMX09IZXJaMk98sIVnFmQ2QNUijYPCWESC5tyTOtKtdSzT52d+66HmXXnFPP1RJucN6siIHq2/mB8WZjx00SAmDu7EQ29v4oUl2+u/aJFGQGEsIkHjCzOmndGTddkF/HfL/hrXe+itTZjBL879Zj/OvjDj0cmDOSu1PffM3sArK3bWZ8kijYLCWESC6sKhyXRMjGb6/IyjLl+cuZ9563KYdnovOifFHHWdCF8YT102lFN7t+Gu19Yy59PaXxQmEooUxiISVJHhYVx/Wg+WbzvIJ9sPfmVZZVU1989Jp3NSDD8+/dsHVIgK9/GXK9JI69qKn/1zDXPXKpCl6VIYi0jQTflOCq1aRH5j7/jF5TvYtKeQeyb0r1VPXTGRPv76ozROSk7k5hdXc+vLq8k7XF5fZYt4RmEsIkEXE+njmtHd+WjzPtZnFwCQd7icP/znM07p2ZpxAzrU+r3ioyP45/UjuXVsb+auzeGsxxbw9rqc+ipdxBMKYxGpF1eM7Ep8VDhPf+S/RekP722mqKySeycOqHMf1JHhYdw6tg9zbhlNx8QYps1axY2zVrKvsKw+ShdpcApjEakXCdERXHlKV95an8OcT3fz4rIdXDGiK307HP+4x/07JvD6jadw5/i+vJ+ey9mPLWD2mmz12CUhT2EsIvXm6lHdiQoP4ycvryYxJoKfjT3xXrXCfWHceEYv3vrpaLq1acFPX17Ddc+vYO+h0iBULOINhbGI1JvWcVFcOjwF5+COcf1IjI0I2nv3ahfPqzecwq/O68/CLfs57ffzuXHWSt5Zv4fSiqqgfY5IQ9B4xiJSrwpKKnh7XQ7fT+uCr56GR9y+/zB/W7SNuWtzOHC4nPiocMYP7MCkIZ0Z2bN1vX2uSF3VNJ6xwlhEmozKqmoWZx5g9prdvLthD0VllbSJi2LCoI5MHNyJwcmJhPt0QFC8ozAWkWaltKKKDzfl8uaa3Xy4KZfyqmpiI30MSk7k5K4tGZbSkqEpLWnVItLrUqUZURiLSLNVUFLBR5tzWZWVx8odeWzMKaSq2v/d171NC4amJDEspSXnDOxA67goj6uVpkxhLCISUFxeydpdBazakceqrHxW78jjwOFyEqLDufuc/kz5TpdjjscscjwUxiIiNXDOsTGnkPvnbmDp1oMMTUniwQtOIrVTgtelSRNTUxjrSgYRafbMjNROCbx03QgenTyYHQeKmfjUxzwwN53DZZVelyfNgMJYRCTAzLhoWDIf3H46k9OSmfHxNsY+uoB3N+xRL19SrxTGIiJfkxQbyUMXDeK1aSNJjIngxy+s5LrnV7BhdwEl5fXXoUjuoVIOlVbU2/tL46VzxiIi36Kiqpq/LdrGY+9toSTQs1ebuEiSW8aS3DKGLq38j8ktY+nVLo7OSTHH9Tmz12Rz92vraBMfycvXjzzu95HGTRdwiYicgD0FpSzbdoCdB4vZlVcS+CsmO7+Eiir/96gZXDGiK3eM60t8dO26/iyrrOJ/56bzj6U7GJqSREZuEUmxEQrkJkphLCJSD6qqHbmFpezKK2He2hyeW7Kd9vHR3D9pAGcfY9zmnQeLuXHWKtZlF/Dj03twx9l92bD7EJf/dZkCuYlSGIuINIA1O/O5+7W1bNpTyPgBHbhv0gDaJ0R/Y7330/dy27/W4IBHJw/hrNT2Xyz7dGe+ArmJ0q1NIiINYEiXJObcMpq7xvdj/uZcxv5hAS8szaI60ONXZVU1v3t7E9c+v4KU1rHMu+XUrwQxwOAuSfzjmu+SX1zBlL8sITu/xIt/ijQg7RmLiNST7fsP88s31rEo4wAnd23J7Wf34fH3t7B820Eu+24Kv56QSnSEr8bXaw+56dFhahERDzjn+PeqbB6Yl05ecQUxET4evHAgFw1LrtXrFchNi8JYRMRDB4rKeH5JFucN6kif9vF1eq0CuelQGIuIhLDPA7msspo2LSJJiIkgMfCXFPvl846JMZw/pBMRGre5UaopjMNr+eLxwBOAD5jhnPvd15bfBlwLVAL7gKudc1knXLWIiAD+i7peuWEkr67YRX5JBQUlFRQUV5B1oJi1uyrILymntKIagEUZ+/nD5MGYaeSpUHHMMDYzHzAdOAvYBXxiZm8659KPWG01kOacKzazacDvgR/UR8EiIs1Vvw4J/GpCao3Lyyqr+PNHW3ns/c9olxDN3ef0a8Dq5ETUZs94OJDhnNsKYGYvA5OAL8LYOTf/iPWXApcHs0gRETm2qHAfP/leL3ILS/nzgkzaJ0Rx1ajuXpcltVCbMO4M7Dxiehfw3W9Z/xrg7RMpSkREjo+Zcf+kgewrLOP+uem0jY9iwqBOXpclxxDUM/xmdjmQBjxSw/LrzWyFma3Yt29fMD9aREQCfGHGk5cO5eSUltz2z09ZknnA65IanenzM3j4nU1fdMbitdqEcTbQ5Yjp5MC8rzCzscAvgfOdc2VHeyPn3F+cc2nOubS2bdseT70iIlIL0RE+ZkxNI6V1LNe/sIJNew55XVKj8dzi7Tzy7mae/iiTX81e3yjGqq5NGH8C9Daz7mYWCUwB3jxyBTMbCjyDP4hzg1+miIjUVVJsJM9dPZzYSB9TZy5Xt5rA/E253DdnA2P7t2faGT15cdkO7puT7nkgHzOMnXOVwM3Au8BG4F/OuQ1mdr+ZnR9Y7REgDnjFzNaY2Zs1vJ2IiDSgzkkxPHf1cIrLq5g6czn5xeXfun5JeRUVVdUNVF3DSt99iJtfXEVqpwSevHQId47ry7Wju/P3xdv57VsbPQ1kdfohItIMLMk8wNSZyxmUnMjtZ/dlz6ESdueXklNQQk5+KbsL/M/ziyto1SKSa0Z354qRXUmo5bjMjd3eQ6VcMH0RzsHsm0d9MZKWc47fvLmB55ZkceMZPbljXN96vT9bPXCJiDRz89bmcPNLqzjya9/fa1c0nZJi6JgYTcfEaFZm5TF/8z7io8O56pRuXDWqOy1bRHpX+AkqLq/kB88sJXNfEa/cMJIBnRK/stw5xy9eX89Ly3dw69je3Dq2T73VckI9cImISOg7b1BHurYeTX5xBR2T/MEbG3n0GFi3q4Dp8zN48sMMZny8jStGdOWaU7vTLv6bYzM3ZlXVjltfXsOG3QXMmJr2jSAG/+1gD14wkIqqah5/fwsRvjBuOrNXg9apMBYRaUYGdv5mGB3NScmJ/PmKk/lsbyHT52fw7MKt/H3xdqZ8pwvXndaD5Jax9VxpcPzu7Y38J30v905MZUy/9jWuFxZmPHzxICqrqnnk3c1E+sK47rQeDVanwlhERGrUp308T0wZys/G9uHpjzKZtWwHzy3JonWLSHq2jaNnuzh6tm1Bz3Zx9GobR+ekGMLCGkef2LOWZfHswm1MHdm1Vj2R+cKM//v+YCqqHA++tZEIn/GjBurBTOeMRUSk1rLzS3hrbQ6Z+4rI3FdERm4RecUVXyyPCg+jT/t4HrhgIIO7JHlW54LP9nH13z/htN5tePbKNMLrMIpVRVU1N81axQebcvngttPp1qZF0OrSBVwiIlIvDh4u94dzrj+g567NwYB5Pzm13i/8yi8uZ0tuEZ/tLWTLXv+Pg8/2FpJbWEa/DvG8Ou0U4qLqfhC4vLKalVl5jOzZOqj1KoxFRKRBrN2Vz8VPL+a03m2ZMTUtqLcKlVVWMefTHN5Ync2mPYXsL/qyw8fYSB+928XRu308vdvFcfHJybSJiwraZweDrqYWEZEGMSg5iV+e25/fzElnxsJtQbkQ6kBRGf9YuoMXlmaxv6iMHm1bMKZfW3q3i6d3e38Ad0yIbjTnq+tKYSwiIkE39ZRuLN16kIff2cTJ3VoyLKXlcb3P5j2FzPx4G6+vyaa8spoz+rblmtHdGd2rTb12ztHQdJhaRETqRUFJBec9uRDnYN5PRpMUW7vzx9XVjgVb9jHz420s3LKf6IgwLh6WzFWjutGrXXw9V12/dJhaREQaVGJMBNMvG8Ylf17Mz19Zy7NXnnzMvdmM3ELuem0dK7PyaJ8QxR3j+nLZ8JSQ7gGsNhTGIiJSbwZ3SeJ/zunP/XPT+evH27j21KOfPy6vrOaZBZn88cMMYqN8PHzxSVw4NJnI8NrfkhTKFMYiIlKvrhrVjaVbD/DwO5tI69aKIV+7//jTnfnc9dpaNu0pZMKgjvzm/AGN7iro+tY8fnKIiIhnzIxHLhlM+4Robpq1ioJAJyEl5VU8OC+dC/+0iLzicp69Mo2nLhvW7IIYFMYiItIAEmMjeOqyYeQWlnLHq5+yOGM/4x7/L88u3MaU4Sm8d9vpnJVac9/RTZ0OU4uISIMY0iWJu8b344F5/sEburWO5eXrRzCiR3B7uQpFCmMREWkw14zuTk5BKTERPm4e04voCJ/XJTUKCmMREWkwZsY9E1K9LqPR0TljERERjymMRUREPKYwFhER8ZjCWERExGMKYxEREY8pjEVERDymMBYREfGYwlhERMRjCmMRERGPKYxFREQ8pjAWERHxmMJYRETEYwpjERERjymMRUREPKYwFhER8ZjCWERExGMKYxEREY8pjEVERDxmzjlvPthsH5AVxLdsA+wP4vs1Z2rL4FFbBo/aMnjUlsFT17bs6pxr+/WZnoVxsJnZCudcmtd1NAVqy+BRWwaP2jJ41JbBE6y21GFqERERjymMRUREPNaUwvgvXhfQhKgtg0dtGTxqy+BRWwZPUNqyyZwzFhERCVVNac9YREQkJDWJMDaz8Wa22cwyzOxur+sJJWY208xyzWz9EfNamdl7ZrYl8NjSyxpDhZl1MbP5ZpZuZhvM7KeB+WrPOjCzaDNbbmafBtrxvsD87ma2LLCd/9PMIr2uNVSYmc/MVpvZ3MC02vI4mNl2M1tnZmvMbEVgXlC275APYzPzAdOBc4BU4FIzS/W2qpDyd2D81+bdDXzgnOsNfBCYlmOrBG53zqUCI4CbAv8X1Z51UwaMcc4NBoYA481sBPAw8JhzrheQB1zjYY2h5qfAxiOm1ZbH70zn3JAjbmcKyvYd8mEMDAcynHNbnXPlwMvAJI9rChnOuf8CB782exLwXOD5c8AFDVpUiHLO5TjnVgWeF+L/8uuM2rNOnF9RYDIi8OeAMcCrgflqx1oys2TgPGBGYNpQWwZTULbvphDGnYGdR0zvCsyT49feOZcTeL4HaO9lMaHIzLoBQ4FlqD3rLHBYdQ2QC7wHZAL5zrnKwCrazmvvceBOoDow3Rq15fFywH/MbKWZXR+YF5TtOzwY1UnT5ZxzZqZL7uvAzOKA14BbnXOH/DsifmrP2nHOVQFDzCwJeB3o53FJIcnMJgC5zrmVZnaG1/U0AaOdc9lm1g54z8w2HbnwRLbvprBnnA10OWI6OTBPjt9eM+sIEHjM9biekGFmEfiDeJZz7t+B2WrP4+ScywfmAyOBJDP7fAdC23ntjALON7Pt+E/hjQGeQG15XJxz2YHHXPw/EocTpO27KYTxJ0DvwNWBkcAU4E2Pawp1bwJTA8+nArM9rCVkBM7F/RXY6Jx79IhFas86MLO2gT1izCwGOAv/+ff5wCWB1dSOteCc+x/nXLJzrhv+78YPnXM/RG1ZZ2bWwsziP38OnA2sJ0jbd5Po9MPMzsV/XsQHzHTOPehxSSHDzF4CzsA/8she4F7gDeBfQAr+kbUmO+e+fpGXfI2ZjQYWAuv48vzcL/CfN1Z71pKZDcJ/IYwP/w7Dv5xz95tZD/x7d62A1cDlzrky7yoNLYHD1D93zk1QW9ZdoM1eD0yGAy865x40s9YEYftuEmEsIiISyprCYWoREZGQpjAWERHxmMJYRETEYwpjERERjymMRUREPKYwFhER8ZjCWERExGMKYxEREY/9PzGqR6hsuwvAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "ax.plot(history.history[\"loss\"])\n",
    "ax.legend(\"loss\")\n",
    "ax.plot(history.history[\"val_loss\"])\n",
    "ax.legend(\"val_loss\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fa0c85cee10>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAHSCAYAAADfUaMwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hVVdrG4d9O7yEklEAIvQakd1QUFaw4KqLYsGHDMjOOo98UZxwdnbFMUZSxYseuICBKU0RAQpfQQgkJJCGk97q/P1YghYQkcJJzkjz3deVKzjk7+6wg8mTt/a53WbZtIyIiIs7j5uwBiIiItHYKYxERESdTGIuIiDiZwlhERMTJFMYiIiJOpjAWERFxMg9nvXFYWJjdrVs3Z729iIhIk9q4ceMx27bb1fSa08K4W7duREdHO+vtRUREmpRlWXG1vabL1CIiIk6mMBYREXEyhbGIiIiTOe2ecU2Ki4tJSEigoKDA2UOplY+PDxEREXh6ejp7KCIi0kK4VBgnJCQQGBhIt27dsCzL2cM5iW3bpKamkpCQQPfu3Z09HBERaSFc6jJ1QUEBoaGhLhnEAJZlERoa6tIzdxERaX5cKowBlw3i41x9fCIi0vy4XBiLiIi0NgpjERERJ1MYV/Loo48yZ86cE4//8pe/8NxzzzlxRCIi0hq4VDV1ZX9duIOYI1kOPeeATkE8fnlUra9Pnz6dhx56iPvuuw+Ajz/+mKVLlzp0DCIiItW5bBg7w9ChQzl69ChHjhwhJSWFkJAQunTp4uxhiYhIC+eyYXyqGWxjmjZtGp9++ilJSUlMnz7dKWMQEZHWxWXD2FmmT5/OnXfeybFjx/j++++dPRwREWkFVMBVTVRUFNnZ2XTu3Jnw8HBnD0dERFoBzYxrsH37dmcPQUREWhHNjEVEpEkVl5Zh27azh+FSNDMWEZEmE3Mki5veWI+/tweXnhXOpYPCieoU1OpbDSuMRUSkScQcyeKG19fh7eFO11A/Xv1hP6+s2kfXUD8uHRTOJa04mBXGIiLS6HYmVgTx/Flj6BbmT1puEd/uSGLR9kT+98N+Xl61j26hflwyKJzLB3eif3iQs4fdZBTGIiLSqHYlZXHD6+urBDFAW38vrhsVyXWjIknLLWLpjiQWVwrmAeFBXD08gqlDOhEW4O3kn6JxKYxFRKTR7ErKYsZr6/F0t/iwUhBX19bfi+tHRXL9qEhScwr5elsin21K4G9fx/D04p1M7Nuea4Z35rx+7fH2cG/in6LxKYxFRKRR7E7KPhHE82eNpXstQVxdaIA3t4zrxi3jurEnOZvPNibwxebDLNuZTBs/T64Y3IlfDe3MWRFtcHdrGfeXFcYiIuJwJojX4eFm8eGdY+odxNX16RDIY5f053eT+/Jj7DE+3ZjA/A3xvLM2jmBfT8b2CGVcr1DG9QyjZzv/Zlv8pTCuwZVXXkl8fDwFBQU8+OCDzJo1y9lDEhFpNvYkmyB2d7OYP2sMPdoFnPE5PdzdmNi3PRP7ticzv5hVu4+yJvYYa2JT+WZHEgAdgrwZ1zOMcT1DGd8rjE5tfM/4fZuK64bxkkchycGdsDoOgoufqfOwN998k7Zt25Kfn8/IkSO5+uqrCQ0NdexYRESaWGZeMX9e8AvDIkOYPrILPp6Ovfcan5bHou2JvPbDftzdzD1iRwRxdcG+nkwd0pmpQzpj2zbxafms2XeMNbHH+GFPCl9sPgzAby/sw/2Tep/2+xSXluHp3jS9sVw3jJ3ov//9L1988QUA8fHx7N27V2EsIs1aaZnN/fM388OeFL7acoQXV8Ry59nduWFMVwK8Tz8K4tPyWLw9kUXbE9mWkAnA0Mg2PHvNYHo2QhBXZ1kWkaF+RIaa4i/bttmdnM3LK/fx/Hd7ABocyLZt89KKWFbuPsoHd45x+C8tNXHdMK7HDLYxrFq1imXLlrF27Vr8/PyYOHEiBQUFThmLiIijPLNkJz/sSeGZqwbRLcyfl1bE8vSSXbzy/T5uG9+dW8Z1I9jXs17nik/LY8kviSzalsjW8gAeHBHMYxf345JB4XRp69eYP8opWZZFv45B/Gv6EDzcLJ7/bg+WBbPPr18gF5WU8djn2/lsUwK/GtqZproF7bph7CSZmZmEhITg5+fHrl27WLdunbOHJCJyRj7flMBrqw9wy9iuXDcqEoAxPULZfCidOStjeeG7Pbz6w35uHtuV2yd0JzTAm5LSMhLS8zlwLJf9x3I5cCyHA8dyOZCSy5FMM0EZ1DmYRy/ux6VODuCauLtZPDttMADPfbsHy7K477xep/yejLwi7np3I+sPpPHQBb15cFLvJisIUxhXM2XKFObOnUv//v3p27cvY8aMcfaQRERO25b4DB79fDtje4Tyx8sGVHltaGQIr98ykh1HMnl55T5e+X4fb605SHgbHw6l5lFSVrGZQ5CPBz3aBTCmRyj9wgOZEhVOZKhrBXB1xwPZBp5duhug1kA+eCyXW+dt4HB6Pv+ePoQrh3ZuwpEqjE/i7e3NkiVLnD0MEZEzdjSrgLvejaZ9oDdzbhhWazFSVKdg5twwjNijOby+ej/peUVMjupI9zB/erbzp3tYACF+ns1y2ZC7m8Vz0wZj2zbPLt2NZcG9E6sG8oaDacx6JxqA9+8czchubZt8nApjEZEWqKC4lFnvbiS7oITP7x1HW3+vOr+nV/sAnrn6rCYYXdNyd7N4/toh2MA/v9mNhcU9E3sC8OXmwzzy6TYiQnx5c+bIWjuENTaFsYhIC2PbNn/44he2xGcw98Zh9OvYejZcqI27m8Xz0wZj2/CPb3YBUFhSyr+X7WVMj7bMvXE4bfzq/oWlsSiMRUSawI97j/HhhkM8cH5v+nYMPK1zlJXZ5BSVEORz6qrnN9cc5LNNCTw4qTdTBoaf1nu1RB7ubrxwrbmHfDyQrx4WwdNXDcLLo2nWE9c6Nqe+ew1s23bp+xK2bdd9kIhIucMZ+Tz5dQxLfjFdon6KPca7t49mYOfgBp0nOauAW9/aQExiFp2CfRjQKYgB4UEM6BRE//AguoT44eZmsXpvCk8timFyVAcePIOGFy2Vh7sb/7p2MB2DvOkQ5MPtE7q7ROa4VBj7+PiQmppKaGioS/zhVGfbNqmpqfj4+Dh7KCLi4gqKS3l99X5eWhkLmG5Qkwd25Na3NjDjtXW8fdsohkaG1OtcsUdzuOXNn0nPK+L+83txKC2PmCNZrNh1lOMFzwHeHvQPD2R3Uja92wfywrVDcGshmyg4moe7G3+4dEDdBzYhy1kzvREjRtjR0dFVnisuLiYhIcGlm2z4+PgQERGBp2f9FseLSOuzYlcyf10YQ1xqHhcP7MgfLu1PRIhZBpSQnseM19aTllvEW7eOrLNyd2NcGre/HY2Hm8VbM0cxKKJiRl1QXMqe5GxijmQRk5jFzsQsikptXrxuqMsvO2qNLMvaaNv2iBpfc6UwFhFpzuJSc3liYQzLdx2lRzt//npFFGf3bnfScUmZBcx4bR2JmQW8ccsIxvUKq/F8S3ck8cCHm+nUxpe3bx2lgG3mThXGzr1jLSLSAhzNKuDJr2O48F8/sG5/Ko9d3I9vHjynxiAG6Bjsw/y7xtClrS+3ztvAqt1HTzrmvXVx3PPeRvqFB/Hp3WMVxC2cZsYiIqcpIT2P/32/n4+i4ykts5k6pBOPTO5Hx+D61ZWk5RZx0xvr2ZOczZwZw7goqiO2bfP8t3t4aWUsk/q158UZQ/HzcqnyHjlNp5oZ67+wiEgDHTiWy8srY/li82EsC64ZHsHd5/aka2jDGka09ffigzvGcMtbP3Pv+5t4btpgfow9xqcbE7huZBeevHIgHk20hZ84l8JYRKSedidlM2dlLF9vO4Knuxs3junKrHN6nNEm9sF+nrx7+yhum7eBhz7aAtDkmxSI8ymMRUROITO/mO9iklm49Qjf70nB38udO8/pwR0TetAu0Nsh7xHo48nbt43iiYUxjOjWlmuGRzjkvNJ8KIxFRKrJzC9mWUwyi7YnsnpvCsWlNp3b+PLgpN7cOr5bo7RN9PPyaJF9oaV+FMYiIkBWQXkAb0vkh0oBfOv47lwyKJzBEcG6bCyNRmEsIs3eJ9HxbDqUwe0TutOrfUCDvjcjr4i53+/n7Z8Okl9cSqdgH2aO68Ylg8IZ0qWNAliaRL3C2LKsKcB/AHfgddu2n6n2elfgTaAdkAbcaNt2goPHKiJykuU7k3nks23YNszfcIhLBoZz33m9GNDp1DsV5RSW8MbqA7y+ej85RSVMHdyJm8d1Y6gCWJygzjC2LMsdmANcCCQAGyzLWmDbdkylw54D3rFt+23Lss4HngZuaowBi4gctzMxiwc+3ExUpyDm3jic+T/H8/ZPB1m0PZEL+rfnvvN6ndT/uaC4lHfXxvHK9/tIyy1iclQHfnNh39PeSUnEEeps+mFZ1ljgL7ZtTy5//BiAbdtPVzpmBzDFtu14y/xKmWnb9il/LVXTD5HWIbugmNijOfXeFKG+UrILuXLOGkrKyvjqvgknGm1k5hfzzk8HeWPNATLyipnQK4zZ5/diWGQIH0fH8+KKvSRnFXJ27zAevqgvg7u0cei4RGpzpk0/OgPxlR4nAKOrHbMVuApzKftXQKBlWaG2badWG8gsYBZAZGRk/UYvIs3a4wt28MXmw3x9/wSiOjVs28DaFBSXcte70aTmFvLJXeOqdLwK9vXk/km9uW1Cd95fH8erPxzgulfXEejjQXZBCSO7hfDf64YyukeoQ8Yi4giOKuB6GHjJsqyZwA/AYaC0+kG2bb8KvApmZuyg9xYRF3XgWC5fbj6MbcNTi3by/h2jz/h+rG3b/P6zbWw6lMHcG4dV2cWoMn9vD2ad05Obx3bj4+h41u1PZdqILkzs0073hMXl1CeMDwNdKj2OKH/uBNu2j2BmxliWFQBcbdt2hqMGKSLN00srYvF0d+POs3vw0spYVuw6yqT+Hc74nF9tOcLvJvdlysDwOo/38XTn5rHduHlstzN6X5HGVJ+mpxuA3pZldbcsywu4DlhQ+QDLssIsyzp+rscwldUi0oodPJbLl1sOc+OYrjx4QW96hPnz1OKdFJeWnfY5v952hOe/28NVQztz78SeDhytiHPVGca2bZcAs4GlwE7gY9u2d1iW9YRlWVeUHzYR2G1Z1h6gA/BUI41XRJqJl1bG4uFmcde5PfB0d+OxS/qzPyWXD38+dFrn2xqfwW8/3srwriE8ffUgXWqWFqVe94xt214MLK723J8rff0p8KljhyYizVVcai5fbD7MLWO70T7QFFdd0L89Y3uE8q/v9jB1SGeCfT3rfb7EzHzufCeadoHe/O+m4Xh7uDfW0EWcQntziYjDvbTCzIrvPrfHiecsy+IPl/YnI7+Yl1fG1vtcGXlF3DYvmryiUt6cOZKwAMdsziDiShTGIuJQcam5fL75MDNGR9I+yKfKawM7B3P1sAjeWnOQ+LS8Os+VkVfEjW+sZ9/RHF6+YRh9Oqgxh7RMCmMRcag5K2Nxd7O4+9yaC6wevqgv7m4Wz3yz65Tnycwr5sY31rMnKYf/3Tycc/q0a4zhirgEhbGI1OpodgEfb4iv1ywW4FBqHp9tOsyMUZF0qDYrPq5jsA+zzunBom2JbIxLq/GYKkF803DO69v+tH8GkeZAuzaJSBW2bbN2fyrvrzvE0h1JlJTZtPHzZM6MYYzvFXbK7z0+K76njmVHd53bgw9/PsSTi3by+T3jqlRGZ+YXc9Ob69mdlM3cm4ZxXj8FsbR8mhmLCGBmo2/8eIBJL3zPjNfW82PsMW4Z14337xhN+0Bvbn7zZ+atOUBt/ezj0/L4bFPCKWfFx/l5efDw5L5sPpTB19sSK8aQX8xNb6xnZ2IWr9w4jPP7nVmDEJHmQjNjkVZua3wG762LY+G2IxQUlzGkSxuemzaYy84Kx8fTLCH6/N7xPDR/C39ZGMOupGyemDoQL4+qv8vPWRmLm1X7veLqjhdyPbNkFxcO6EBhSRk3lwfx3BuHn3GnLnGSkiLYuQDCB0NoL9B68HpRGIu0UqVlNn/7OoZ5Px3Ez8udXw2N4IbRkQzsfHKv5wBvD169aTgvfLeHl1bGEns0h1duHE67QLPMKD4tj083JnDD6MgqmzacirubxR8v7c8Nr6/nxRV7+TE2lZjELF65QUHcrH37B/j5VfN12x7QZ4r5iBwLHl7OHZsLUxiLtEK5hSU88OFmlu86ysxx3fjtRX0I9Dl1Ew43N4uHJ5t9f3/36VamvvQjr948goGdg3l5VfmsuIEtKsf3CmNSv/bMWbkPT3eLl28YzgUDFMTNVsxXJoiH3wodB8Lub2DDG7DuZfAOgp7nm2DufSH4n7r+oLVRGIu0MomZ+dw+L5rdydn87cqB3DSma4O+//LBnege5s+sd6K5Zu5PPHxRXz6JTmDG6EjCg30bPJ4/XNqfpKwCHpzUmwsVxI0r5yi8Pw3OeRj6X+7Yc6cdgK/uh87D4eJ/mlnwyDugKBf2fw97voE9SyHmS8CCQdfAhX+DoLo3+2gNrNqKMRrbiBEj7OjoaKe8t0hr9cvhTG5/ewO5haW8NGMoE89gyVBKdiH3vLeR6Lh0vNzd+P6RiacVxtKEljwK618BnzZw33oI7OiY85YUwpuTIXU/3P0DhHSr+biyMkjaCr98Duv/B+6ecO7vYfTdreIStmVZG23bHlHjawpjkdbhu5hkHvhwM239vXhj5gj6dQw643MWlpTywnd7CA/yYeb47g4YpTSazAT471DodjbErYEe58H1HzqmwOp4yE9/r/4z7rQD8M1jsGcJhPUxs+me59X9fcUFEPcjJMdAm0hTJNa2B3j5ndnP0AROFca6TC3Swtm2zZtrDvLkohgGdQ7m9VtGnNi84Ux5e7jz2MX9HXKuZq+4AHKSap8VOtsPz5rPl//HVDsv/T/YOh+GXH9m5935tQni0Xc37NJ32+4wY765dL3k9/DuldD/Cpj8d2jTpeqx2cmwd6k5dt9KKM49+XxBERDa04RzWG/zObQnBEeCu+tHnWbGIi1YSWkZf10Yw7vr4pgc1YF/Tx+Kr5d2PGoUi34LWz6Eh/eAd0DjvY9tN3w2m7YfXhoJI26DS56FslKYd6mZXd63DoI6nd5Y0uPgf2dDSHe4/VvwOM1NPIoLYO2L8MPz5vE5vzXFXnuXmXvNRzaZ54MioM9kUwTWeThkJUBqLKTuK/8cC8dioTCz4txunib4j4dzaK+Kj4AOTbr0SpepRVqhIxn5/ObjLazbn8Zd5/Tg91P64ebWgtZ8FhfAwdUVhUFlJTDydhhxO/i1bdqx5B6Df0VBSQFc/xH0neL49ygphE9uhdyjcMtC8GzA/fnP7zKVzg9uqbhPnLoPXhkP3SbADZ80PJRKiuCti+HYHrjrBxN4Zyoj3iyNivmq/AkLIkZUBHCHgXWP07YhLxWO7YW0aiGdth9KCyuOHT7TXCloIrpMLdLKLNx6hD98sZ2SMpvnpg3mmuERzh6SY2QlVlyu3L8KivPA09/cayzOgxVPmtnVkBkw5l4I69U049rwhglidy/Yt9zxYVxaDJ/MhN3l28p/82j9Q+ToLtj2EYy7v2rBVmhPuPCvsOQR2PweDLupYWNa/lc4HA3T3nZMEIO5PH3tO3BwjbnH3fN8CGjgBiGWZZZN+YdB17FVXysrNedNjYVfPoON8yDqKuhxrmPGfwYUxiItSHZBMY9/tYPPNx9maGQb/j19CF1D/Z09rDP382smMBK3mMfBkTD0RjNj6joBPMvvgSfHmDWtm9+F6DfNbGrcbOg6vvEuRxYXwIbXoNeF4OYOscsce/6yUvj8ThPElzwHWYfhx3+ZQqxB19T9/av+Dl4BMP6hk18beSfElN8/7jHx5Hu1tdm9BNa+ZJYuRV3ZkJ+mfrqNd/w5wfz3CelqPrqOM4Vsi34L96w5/UvsjhqaU99dRBwm+mAaF/9nNV9uOcyDk3rzyV1jW0YQr38VFj8MlhtMehzuXQcPbTP3PntdUBHEAB0GwNSX4Nc7zJKZhJ/NvdFXzzVhnrwDivMdO77tH0Nuign9npPMpdC0/Y45d1kZfDUbdnxh1uSOuhPO+yN0GQMLHzSXXk8lcau55Dv2XvAPPfl1Nzfz51VWCgvuN5d465K6D764GzqeBRc9dXo/lyvw9DW/3KTuhZ9edPZodM9YpLkrLi3jv8v3MmdlLBEhfvxr+hCGdw1x9rAcI3aZaVLRZ4pZNuPWwOKz4nxziXbtHHNv87jgLicX84T1bngltG3Dy2NMkdDdq00IvzjM/CM/6s6Gnaumcy/6jZnhT/w/mPj7itcyE2DuBFPQdMeyqr+QVPb+tRC/3vzy4nNym9MTNrxh3uuyf5kir5oc2Wz+HHd8AZ5+MGuV+TNs7j66CfZ+a9ZdN3IlvO4Zi7RAtm2z92gOv/tkK1sTMpk2PILHr4giwLuF/G99dJcpWGofBVe91vAgBjP7GT4Tht4Myb+YWVDlytttn1StvL3wbzD+gfqfP3Y5pOyCX/3PXAYP7Wkqi2OXn1kY2zYs/YMJ4vEPwbmPVH09OMK85wfXwtLHTIhWF/+zub8+6fFTBzGYAN65AJb+0dynPR5KZWWmQG7tHLO21yvQLGEafXf9L2m7uinPmP9eix+BGR85bWOLFvJ/rUjLV1Bcyo4jmWyMS2dTXAabDqVzNLuQYF9PXr5hGJcMakFtBXOPmaDx9DVrUc90qZCbG4SfZT4qs23zXqmxsObfsPwJ6DUJOkTV77xrX4TAcFMEdFyvC2DLB6b6+XTvQ658CtbNgVF3wQV/qTkg+kw2RVk/vWjuHw+8qurrK54E/3Yw+q6638+y4IoX4eVx5rL49fNh23xY+7KpSA7uYi5JD7up7mBvboI7w3mPwbd/hF2LoP9lThmGwljEhRQUl5KRV0xabhEZeUWk5BSyNT6TTYfS2XEkk+JSc1upS1tfxvUMZVjXEKZEdaR9HfsHN5qcFNg0zxRUDZ7umHOWFMJHN0JOMsxcbGaBjcWyTLVuQDtzmXrOaPjyHrhjuWnVeCpJ201F96THq7Zy7DXJFHQdWnd6Vbo/PGcadAy72czaTjVTm/S4eZ8FD5RvWVh+2fjAD3Dge5j8NHjVs26gTSRMfgoWPgDP9TbV6Z2HwzVvmWYczaBxxmkbfbdZI77k96Yyv75/Zg7Ugv90RVzbsphk3l57kNQcE7zpecXkF5eedJy3hxuDI9pw24TuDIsMYVhkyImtC53m6C4zc9v6UcW6zeI8GHHrmZ3XtmHhQ3BoLVzzJkQMP/Ox1pd/GFz+b/OLwOoXqt6jrcnaOWZZVfWfudvZ5h7yvuUND+N1c2HF32DQtXDZv82M/lTcPU1Yzp1glj7d/p2Zja94EgI71X7/tzbDboaEDVCYZZaGdRndOvYjdveEy14w/bW//wdc+ESTD0FhLNLESstsXvhuN3NW7qNrqB+92wcwoFMQIX6etPHzIsTP68TXbf296NHOH093F1j4YNtmJrj2JVNY5eEDQ28wy1uW/QW+/rW5rDz4utN/jzX/hq0fwMTHYODVjhp5/fW/HAZNgx/+CX0vPvmy9nFZibD9UxN2vtWK5bwDzPrW2OUN+0c9K9E0vOh7CVz5Sv3vkbfpYo6ff7251Nr7IlO0ddm/ai/sqo1lmerq1ihyjFkut3YODL4e2jdtm1eFsUgTSsst4oEPN/Nj7DGuHxXJ45cPwMfTxdtTlhSa4Fk7B47uAP/2ZnnNiNsqlstc+y58MM1c4vXwhqhfNfx9dn4Ny/5qQvjcOmaljenif5rLvF/eA3eurHk3oZ9fBbsUxtxT8zl6ToJlj5uAre8WgRvnmSVGk59q+CXhfpfAmPvM1YqdC0wB1tAGNvEQuOAJc9940W9h5qImvSqgMBZpIlviM7j3vY0cyy3in1efxbUjG1CNWlIE6QcrqoCP9+NNP2hmopP+dOYDLMozS3Mqnz81FlJ2m4rj9lEw9WXTaKJ6YZKnD1z3Ibx3NXx2B3j4NqwLVeJW09ii83CYOse5l0b92ppLxPOvN/duz/9D1deLck2Vc7/Lau881esCE8b7lpvZVl1KimDjW9D7QrMD0em44C8Qvw4Ob4Qr59Z9z1tO5h8KF/zV3Dd3xCYaDaAwFmlktm3z4c/x/GXBDtoHefP5PeMY2LkeFan56fD1b8z6zow4sMsqXvMLK986rjusfs60OTydpTRJv8B3fzaBm5VQ9bXATqYgaOBV5vJtz/NPHZLeAXDDx/DOVPj4JrNMpOf5p37/slLTWWrxI+DbFq77oGE9lxtLv0vMpcrVz5vL1Z2HVby25QMoyICxs2v//g5RENDRXKquTxjvWmgK1kbNOv0xe3jB9PfNn+dZ157+eVq7oTeZDm7f/tFUrDdRn3M1/RA5DWVlNgnp+fh4uRHm713rBgwFxaX86ctf+GRjAuf0acd/pg8hxL+em6gvfAg2vQMDroDQ3pUaVPSouE9ZVgrzZ8De7+DGT+sOv8pSdsNbl5iA7Tmp6q42bXuc/nKivDR4+3Izs77xs5pbGxbmwJb3TevK9IPmsur096HjwNN7z8aQn2EaevgEm40QPLzNn/eLw02x1x11tL388j7Y9TU8sr/u+79vToHsJLh/U91FW9L4EreZrm3DZ9a8hvs0qemHyBnIKyphV1I2OxOziDmSRUxiFruTsskrMpXPnu4WHYJ86BTsS8dgH8Lb+BAe5EO7QB9eXhXLjiNZPDCpNw9O6o17fXdNSog29xDH3AtT/l77cW7ucPXr5h/zj2fCHd9Bu751nz9tP7x9hWkxeesSx26o4NcWbvoS5l1i1grf/JXZeQcg8zD8/D+InmcufXcZbRpt9Lv09Jp6NCbfNmbt7fvXwKqnzWXg3Ysh/YD5ui69zoct78HhTdBlZO3HJW031eMXPaUgdhXhZ5nlTju+NJetfYIa/S01Mxapwdb4DF7/8QA7DmdyIDX3RMveQB8P+ocHMSA8iH4dAykqLeNIRgFJmfkcySwgqfyjqNRcUg7y8eBf04cwqX+H+r95aQm8dp7pdzx7A3gH1v09GYfgtfPN+sg7VtTch/jEsfFmRlyUY4pUOgyo/9gaIuuI2aDW+VQAACAASURBVGIvP93MLnYvMa0U7TIYMNUUHJ0qpFzFV7PNLP7270xXrOxEeGBz3b885KXBsz3hnEdMU4naLHgAtn0Mv915cmW2OE9hjinSc2CTE82MReopLbeIZ5fuYv6GeNr4ejKyW1uuGNLpRABHhPhi1VFcVFZmk5ZXRGJGAZ3a+BAa0MA1wRteh6RtMG1e/YIYTMOG6z40myJ8fJOZmdZUBZydBO9cAQWZcMuCxgtiMBvW37zABP+nt1W0Uhw1y+ya01xMfgr2rTS3A3KSYco/6jeL92trCtL2La89jPPTTRCfNU1B7GrOtOtbAymMRTBrfz9YH8dz3+4ht7CEOyZ054FJvQn0aXhFqpubRViAN2ENDWEwYbniSXPvd0ADt6brMtJUIn9+h1nzO/WlqgVXucdMcVXOURPWnYY0fHwNFdIVbl1s1idHXdk8Wyn6BMPUF+HdX4F3sFlbXV+9LjBNJPLSai4E2vIBlOSbrQylVVMYS6sXfTCNP3+1g5jELMb1DOWvV0TRu0M9Z6SOtvT/oLTI7PpzOst7zppmNkP4/h/Qrg+Mf9A8n58O715piqVu/KxpLw+HdIXhtzTd+zWGnueb/yZ+ofW/WgGmMG7V07B/5clNTMrKzD7NXcbU3lxEWg2FsbRaR7MLeGbJLj7fdJjwYB/mzBjGJYM61nkZutHsWwm/fAbnPnpmW9Od+6jZLvC7x00VdrcJZv1vym64/kPzWBrudJaOdR4GPm3MEqfqYbxvuSkGO/+PjhmfNGsKY2mVlu9M5qH5WygsKeO+83py33m98PNy4v8OJYWw+GGz/d6EX5/ZudzcTHvE9DjTgKNdX9NUY/q75rKpNB03dzOrjl1u2olW/kXv59dMN7P+VzhvfOIyVEcvrU5pmc0TX8fQMdiHpb8+h99N7ufcIAZY81/T7erS5xreT7gmnr5mFuzbBhK3wNWvmeVD0vR6TYKcJEjeUfFc2gGzof2IW2sutJNWRzNjaXW+i0kmLjWPl28YRvewpt8q7SRpB0wXrQFXOnbmGtgRbvvG9EeOHO2480rD9JxkPscuq2hqEv2GmTUPP8NdrqTF0MxYWp3XV++nS1tfJkd1dPZQzKXLJY+AmwdMedrx528TqSB2tqBw6DDQhDGYHuCb3jW9reu7iYS0eApjaVU2H0onOi6d28Z3r383rNNRXGDuE+74wnRYKsqt+bidC83lyvP+z6zLlZap1yQ4tM40kvjlU9Pb+kz6UEuLo8vU0qq8vvoAgT4eTBvRgB2T6is7yQTrnqWmMrq4WgAHda7o/Xy8//M3j5pZ06i7HD8ecR09J8Ga/5itGX9+FdoPgK7jnD0qcSEKY2k14tPyWPJLInee04MAbwf81S8rg6StJnz3fGN2VwIIijBbr/WZAgEdIK18K8Jj5VsT/vKZ6YB13DVvNXz/WmleIseAp7/ZkjFpu2kP6sxtIsXl6F8AaTXeWnMQN8ti5rhuZ36yrMTynYn2AhZEjIRJfzYB3H5A1X9oqzd0sG3TkSk11hzXZdSZj0dcm4c3dD8H9iwB7yAYpC0OpSqFsbQKmfnFfLThEJcP7kR48Bnul1uUBx9eZzYMmDrHBLB/WP2/37LMRg6n2sxBWp5ek0wYD7mhyfsei+tTGEurMP/nQ+QWlXL7hO5ndqKyMvjybtNE4/oPzcbzIvUx4EpT1Df2PmePRFyQwlhavOLSMub9dJCxPUIZ2Ll8o4LifCjIgsAGbG0IsPIpiPkKLnpSQSwNE9AOZsx39ijERdVraZNlWVMsy9ptWVasZVmP1vB6pGVZKy3L2mxZ1jbLsi5x/FBFTs/i7YkkZhZw5zmVZsVf/wb+FQXr5kJ99/Te+pFpzjHsZhg7u3EGKyKtUp1hbFmWOzAHuBgYAFxvWVb1TVD/CHxs2/ZQ4DrgZUcPVOR02LbNa6v307OdPxP7tDdP5qWZtZ7egfDN7+GjG82uRqdyaD0smA3dzoZLnlclrIg4VH1mxqOAWNu299u2XQTMB6ZWO8YGgsq/DgaOOG6IIqdv3f40fjmcxe0TeuB2vMnH1vlmm8JbFsBFT5llSXPPgfgNNZ8kPc5sLB8cAde+o17CIuJw9QnjzkB8pccJ5c9V9hfgRsuyEoDFwP0OGZ3IGXp99X5C/b24alj5X1nbhk1vQ+fh0HEQjJsNt30LFvDWFLNhQ1lZxQkKskzldFkxzPi45g3iRUTOkKPaYV4PzLNtOwK4BHjXsqyTzm1Z1izLsqIty4pOSUlx0FtLq1NabFoLlpWe8rDYozks33WUG8d0xcfT3TwZ/zOk7ILhMysOjBgOd602BVnf/cmEb26qOf9nd5h9gKe9DWG9G+9nEpFWrT5hfBio3Dswovy5ym4HPgawbXst4AOctPDStu1XbdseYdv2iHbt2p3eiKX1Ksg0M9f/DIE3J5v2gqfwxo8H8PJw46axXSue3DgPvAIg6qqqB/u2gWvfhUueg/0rYe4EE8R7l8Ilz0LP8xz/84iIlKtPGG8AeluW1d2yLC9MgdaCasccAiYBWJbVHxPGmvqKY6QfhG8egxcGwHd/ItM3gn3e/clb9QLvrdzKuv2pZOYXV/mW1JxCPt+UwFVDOxMW4G2ezM8wGzcMuqbmpguWBaPuhDuWmT2Fd3wOo++Bkbc3/s8oIq1aneuMbdsusSxrNrAUcAfetG17h2VZTwDRtm0vAH4LvGZZ1q8xxVwzbbu+60WkVSkrg2WPQ9waaHt804RKn70DK46N3wBrXzQ7G1lu5PeZykv5FzFndyDjAw7zfunvSF3+b/5Ycg0Andv40j88iAGdgkhIz6OwpIw7zq60nGn7J1CSX/USdU3CB8Os72H/KuirVXoi0vgsZ2XmiBEj7OjoaKe8tziJbcPih2HD69BpGOSmQGZ81WMCw00wF+fB4Y3gHUzpsFv4gIt5ek0WJWU2d5/bk3vO7Ynvl7dStncZay9fwbY0T2ISs9iZmMX+lBzKbJjUrz1vzBxZ8d5zzzaz37tXN/3PLiKtnmVZG23bHlHTa+rAJU3DtuHbP5ogHv8gXPBXE4xFeZB+wGyakBoLqeU7HJUWwcX/5MeAyfz5m4PsT0nngv4d+PNlA4gM9TPnnPh/uMUsYHzS+4y/6G8n3iq/qJTYozkVxwEc2QTJ2+HS55v4BxcRqZvCWJrGqqdh7UtmQ/XjQQzg5QcdosxHJQnpeTz59U6+2bGDrqF+vDVzJOf1a1/1nO37wVnXws+vmX6/gR0B8PVyZ1BEcNVjN74Nnn4waFpj/YQiIqdNYSyNb/UL8P0/YOhNMOUfVbpXlZSWkZCez4Fjuew/lsuBYzkcOJbLxjjTEet3k/ty+4TuFUuTqpv4KGz/FFY/b6qea1KYbfYQjroKfIJrPkZExIkUxtK41s2F5X81M9LL/wNubiyLSWb+hkPsP5bLodQ8Ssoq6haCfDzo3i6Aq4ZFcN95vejcpo7tDtv2gKE3QvRbMO5+aBN58jG/fAZFOTD8Fgf/cCIijqEwlsazcZ7p/dz/crhyLri5c/BYLvd9sIlQfy/OimjD5KiOdA/zp0eYP93D/Gnr74XV0L7P5z4CWz+E7/8JU1+qYRxvQ7v+EDHSIT+WiIijKYylcWz9CBY+BL0vgqvfBHcPbNvm/77Yjpe7G5/fO56OwT6Oea/gCBhxm7l3POHXZonUcYnbTPHWlGe0uYOIuCxHtcMUqbDjS/jybuh+dpWNFT7ZmMBP+1L5/cX9HBfEx034DXh4w6pnqj6/6W1w94azpjv2/UREHEhhLI5VkAlf3GUuCV/3IXiae74p2YU8tWgnI7uFMGNUDfd1z1RgB1Opvf0TSI4xzxXlwbZPYMBUbfAgIi5NYSyOtedbKCmAi56s0nLyrwt3kF9UytNXnVWxlaGjjX/QdPBa9XfzOOZLKMxU4ZaIuDyFsTjWzgUQ0BE6VzSZWRaTzNfbEpl9fi96ta+hJ7Sj+LU16413LoQjm03hVmgv6Dq+8d5TRMQBFMbiOEV5ELsM+l8GbuavVnZBMX/66hf6dAjg7nN71nECBxhzL/iGwFezIX4dDLtFhVsi4vIUxuI4+1aYntL9Lz/x1LNLd5OUVcAzV5+Fl0cT/HXzCYLxD0HyL+DmCUNmNP57ioicIYWxOM7OhWZWWn5ZOPpgGu+ui+OWsd0YFhnSdOMYdScEdoKoK8H/pG21RURcjtYZS4PlFpbg5+VetTlHSRHsWQL9LgN3TwpLSnn08+10Cvbl4cl9m3aAXv5wz5oTldwiIq5OYSwN8t/le3nhuz1EhPgyqV97zu/fgdHd2+ITt9osayq/RP3yyn3EHs3hrVtHEuDthL9mWsokIs2Iwljq7XgQX9Df7J70UXQ8b6+Nw8/LnVeC32Wcuy/p7caSkZzNy6timTqkE+f1bV/HWUVERGEs9fJieRBfNbQzz04bjLubRUFxKWv3pbJi5xEGbv2RpaWDmf3PNQT6eBDg7cGfLxvg7GGLiDQLCmOp00sr9vJ8tSAG8PF057x+7TnPdx9szWDIRTfxcHEffow9xq3juxMa4O3kkYuINA8KYzmlOStjee7bPfyqWhBXsXMhuHsRMWoqs70DmX1+76YfqIhIM6alTVKrOStjeXbpbq4c0onnagti2zZh3PN804pSREQaTGEsNXp5VUUQP3/tkJqDGCBxK2QeqtLoQ0REGkZhLCd5ZdU+/vnNbqbWFcRgZsWWG/S5uOkGKCLSwuiesZxg2zYvr9rHs0t3c8XgTjxf26XpynYuNB23/EObZpAiIi2QZsYCmCB+esmuE5emX7h2MB7udfz1SNkNx3ZD/yuaZpAiIi2UZsZCaZnN/32+nY+i47l5bFf+cnlU/fYc3rnQfO53aeMOUESkhVMYt3KFJaU8NH8LS35J4oHze/HrC/tU7Tl9KjsXmn2Lgzs37iBFRFo4XaZuxXILS7h9XjRLfkniT5cN4DcX9cVK2w9bP4KyslN/c8YhSNyiKmoREQdQGLdSGXlF3PD6etbuT+W5aYO5fUJ388Ki38AXs+D9qyEnpfYT7PzafFYYi4icMYVxK5ScVcD0/60j5kgWL98wjGuGR5gX0vbD/lXQ/RyI+wnmjocDP9R8kp0LoX0UhPZssnGLiLRUCuNWJi41l2vm/kRCeh7zbh3J5KiOFS9uetesGb5yLtyxHLyD4J2psOoZKCutOC7nKBxaq1mxiIiDKIxbkZLSMma+tYGcghI+uHMM43qFVbxYWgxb3ofeF5mCrI4DYdYqOGs6rHrahHJWojl292LAVhiLiDiIwrgV+WrLEQ4cy+UfV5/F4C5tqr64ZynkJMPwmRXPeQfAr+bCla/A4Y0wdwLELjeXqEO6Q4eoJh2/iEhLpTBuJUrLbOasjKV/eBAXDuhw8gEb50FgOPS68OTXhsyAO1dCQHt47yrYt8LMiuu7BEpERE5JYdxKLNqeyP5judx/fq+T1xFnxEPsMhh6I7jXsvS8fT9zH3nYLea+8qBrGn/QIiKthJp+tAJlZTYvrdhL7/YBTKlcsHXc5vfM56E3nfpEXn5wxX9hyjPmaxERcQjNjFuBpTuS2JOcw+zze53c5rKsFDa/a/YjDulavxMqiEVEHEph3MLZts2LK2LpEebPZWd1OvmA2GWQdRiG39L0gxMREUBh3OIt33mUmMQs7j2vV83bIW58G/zbaT9iEREnUhi3YGZWvJcubX2ZOqSGWXF2Euz5xlRLe3g1/QBFRARQGLdo3+9JYWtCJvdN7IVnTXsTb34P7FJTIS0iIk6jMG5ODqyGF4fD7iV1Hnr8XnHnNr5cNSzi5APKymDTO9DtbPWXFhFxMoVxc7L3W0iNhQ+vg6V/gJKiWg9duy+VjXHp3H1uD7w8avjPfGAVZMRV7bglIiJOoTBuTlJ2Q1hfGDUL1r4Eb02B9IM1HvrfFXtpH+jNtBFdaj7XxrfBNwT6XdZ44xURkXpRGDcnKbug4yC45Fm49l04Fgtzz4GYr6oc9vOBNNbtT+Ouc3vi4+l+8nlyUmDXIhg8Azx9mmjwIiJSG4Vxc1GUCxmHoF0/83jAFXD3DxDWCz6+GRY9DMUFALy4Yi9hAV7MGBVZ87m2fgBlxVpbLCLiIhTGzcWxvYAN7fpWPBfSDW79BsbOhg2vwRsXErN9E6v3HuPOs3vg61XDrNi2TeFWlzFVzyUiIk5Tr97UlmVNAf4DuAOv27b9TLXX/wWcV/7QD2hv23a1PfrkjKTsNp+Pz4yP8/CCyU+Zqugv76bH55cwzyeKcXljILoXhJZ/BHY0uyzFrTFFYGc/3PQ/g4iI1KjOMLYsyx2YA1wIJAAbLMtaYNt2zPFjbNv+daXj7weGNsJYW7eUXeDmCW271/x63yns+dU3/PLubzknKAmvja9DSUHF614B0LaHudztHQwDpjbNuEVEpE71mRmPAmJt294PYFnWfGAqEFPL8dcDjztmeHJCym4zw3X3rPWQuVsK+Ma6n7X3TQIfd8hKMLPg1H3ln8u/Hv+ANnsQEXEh9QnjzkB8pccJwOiaDrQsqyvQHVhx5kOTKo5XUtciOauAhVuPcMPorgT7lQd2m0jz0fP8JhqkiIicDkcXcF0HfGrbdmlNL1qWNcuyrGjLsqJTUlIc/NYtWHEBpB84+X5xJfN+Okhpmc1t42u5jC0iIi6rPmF8GKjcOSKi/LmaXAd8WNuJbNt+1bbtEbZtj2jXrl39R9napcaCXVZr9XNuYQnvr4tjclRHIkN1+VlEpLmpTxhvAHpbltXdsiwvTOAuqH6QZVn9gBBgrWOHKKTsMp9rmRl/ujGBrIIS7ji7RxMOSkREHKXOMLZtuwSYDSwFdgIf27a9w7KsJyzLuqLSodcB823bthtnqK1Yym6w3Gvc0KG0zOaNHw8wLLINw7uGOGFwIiJypuq1zti27cXA4mrP/bna4784blhSRcousyzJw/ukl76LSeJQWh6PXlz7/WQREXFt6sDVHKTsrvV+8eurD9ClrS+Tozo28aBERMRRFMaurqQI0vbVGMabD6UTHZfObeO74+5mOWFwIiLiCApjV5e2H8pKaizeen31AQJ9PLi2tm0SRUSkWVAYu7oTldRVZ8bxaXks+SWRGaMj8feu161/ERFxUQpjV5eyG7AgtHeVp99acxA3y2LmuG5OGZaIiDiOwtjVpeyCkK5Vekln5hfz0YZDXD64E+HBvk4cnIiIOILC2NWl7D7pfvH8nw+RW1TK7RPU+lJEpCVQGLuy0hJI3VvlfnFxaRnzfjrI2B6hDOwc7MTBiYiIoyiMXVn6QSgtqjIzXrw9kcTMAu48R7NiEZGWQmHsyqpVUtu2zWur99OznT8T+7R34sBERMSRFMau7HgYh/UB4NuYZH45nMUdZ/fATU0+RERaDIWxK0vZDcFdwDuQr7YcZvYHm+jXMZBfDe3s7JGJiIgDqVuEK0vZhd2uL6/9sI+/L97FqO5tee3mEfh4ujt7ZCIi4kAKY1dVVop9bA/ryqL4++JdXDoonOevHawgFhFpgRTGLqow9SDeJQV8cTiAmeO68efLBug+sYhIC6V7xi4oM7+Y/3y4EIBxo8fx+OUKYhGRlkxh7GKSMguY/r+1Jyqpr7zwfCxLQSwi0pLpMrULiT2azc1v/ExWQQm39C6ElHDwbePsYYmISCPTzNiFPPH1TgpKyvjorjF0KDh40raJIiLSMimMXURpmc3Gg2lcOiicqPCgGjeIEBGRlklh7CJ2J2WTW1TK8K4hkJkAxbmaGYuItBIKYxex8VA6gAnjlN3mSc2MRURaBYWxi9gUl067QG8iQnwrbRChMBYRaQ0Uxi5i06F0hkeGmGVMKbvAvx34tXX2sEREpAkojF1ASnYhcal55hI1qHhLRKSVURi7gE3l94uHdW0Dtl0exireEhFpLRTGLmBTXDpe7m5EdQqG7CQozNTMWESkFVEYu4CNcekM7BxkdmQ6UbylmbGISGuhMHaywpJSth3OrHq/GDQzFhFpRRTGTrbjSBZFJWWVwngX+IaYamoREWkVFMZOtimuvHgrsloltXZqEhFpNRTGTrYxLp0ubX1pH+RTXkm9U/eLRURaGYWxE9m2zcY40+wDgNxjkJ+u+8UiIq2MwtiJDmfkczS7kGGV7xeDZsYiIq2MwtiJNp50v1g9qUVEWiMPZw+gVYlZAMf2nHjovzOZh7yy6B8bA/ssiF0O3kEQGO7EQYqISFNTGDeV0hL49FYoKznx1AXABW7AykrH9btMldQiIq2Mwrip5KaYIL74nzD8VnKLShj+5DLuOqcHv76gT8Vx7p7OG6OIiDiFwrip5CSZz8ER4OHF1rgsCsrcGdK9PXh4OXdsIiLiVCrgairZyeZzQEegUrOPLiHOGpGIiLgIhXFTyU40nwM7AKaSulf7AIL9dFlaRKS1Uxg3lZzymbF/e8rKbDYdyqho9iEiIq2awripZCeBXxh4eLH/WC6Z+cUVm0OIiEirpjBuKtlJEFjtfrHCWEREUBg3nZwkCKi4X9zGz5MeYf5OHpSIiLgChXFTyU4+0Vlr46F0hkWG4Oam5h4iIlLPMLYsa4plWbsty4q1LOvRWo651rKsGMuydliW9YFjh9nMlZWaAq7ADmTkFRF7NEf3i0VE5IQ6m35YluUOzAEuBBKADZZlLbBtO6bSMb2Bx4Dxtm2nW5bVvrEG3CzlpYJdCgEd2XwoA6i0OYSIiLR69ZkZjwJibdveb9t2ETAfmFrtmDuBObZtpwPYtn3UscNs5rLLu28FdmRjXDrubhaDuwQ7d0wiIuIy6hPGnYH4So8Typ+rrA/Qx7KsNZZlrbMsa4qjBtgiVAvj/uGB+HmpE6mIiBiOKuDyAHoDE4Hrgdcsy2pT/SDLsmZZlhVtWVZ0SkqKg966GSjvS13i154t8Wr2ISIiVdUnjA8DXSo9jih/rrIEYIFt28W2bR8A9mDCuQrbtl+1bXuEbdsj2rVrd7pjbn7K+1LvzvUjv7hU64tFRKSK+oTxBqC3ZVndLcvyAq4DFlQ75kvMrBjLssIwl633O3CczVt2IviGsPFwHoAqqUVEpIo6w9i27RJgNrAU2Al8bNv2DsuynrAs64ryw5YCqZZlxQArgd/Ztp3aWINudnKSIcDcL+4Q5E3nNr7OHpGIiLiQelUR2ba9GFhc7bk/V/raBn5T/iHVlbfC3BiXzvCuIViWmn2IiEgFdeBqCtlJ5Hu3IyE9X+uLRUTkJArjxmbbkJPM4dIgAEZ0a+vkAYmIiKtRGDe2vDQoK2ZvXgA+nm5EdQpy9ohERMTFKIwbW3YiAFszfBjSpQ2e7vojFxGRqpQMja284cemdG9GdNUlahEROZnCuLGVN/xIKmvDiG4q3hIRkZMpjBtb+WXqFNqo85aIiNRIYdzYcpLJtfzp2iGMIB9PZ49GRERckMK4kdnZSSTrErWIiJyCwriR5acdJrGsjYq3RESkVgrjRlaamchR2mhzCBERqZXCuDHZNj6FKeR6hhERos0hRESkZgrjxlSQgaddjE9oZ20OISIitVIYN6LkIwcBaNcx0rkDERERl6YwbkT79sUC0KVrDyePREREXJnCuBElHo4DIDJSYSwiIrVTGDeizKPxAHgEhzt5JCIi4soUxo0ku6AYKzuJIjc/8A5w9nBERMSFKYwbyeZDGbSzMijx7+DsoYiIiItTGDeS6INpdLDS8Q7p5OyhiIiIi1MYN5LouHQ6e2ThHtTR2UMREREXpzBuBMWlZWyJTyeMdAhQGIuIyKkpjBvBzsQs3Ity8CorgECFsYiInJrCuBFEH0ynvZVuHiiMRUSkDgrjRrAxLp0BAfnmQYCqqUVE5NQUxg5m2zYbDqYxMqzQPBGohh8iInJqCmMHS0jP52h2IVFB5TPjQM2MRUTk1BTGDhYdlwZAN+9s8PAF7yAnj0hERFydwtjBNhxMJ9Dbg5DSNFO8pX2MRUSkDgpjB9t4MJ1hXUNwy0lWJbWIiNSLwtiBMvOK2XM0mxFdQyAnSZXUIiJSLwpjB9p0KB3bhuHdQiA7SZXUIiJSLwpjB4qOS8PDzWJIBw8oylEltYiI1IvC2IGiD6YT1SkIv8JU84T6UouISD0ojB2kqKSMLfEZjOjWFrITzZMq4BIRkXpQGDvItoQMCkvKTPFWdpJ5UmEsIiL1oDB2kE+iE/D1dGd87zDISTZPqppaRETqQWHsAFkFxSzYeoQrBnciyMfTXKZ29wbfEGcPTUREmgGFsQN8teUI+cWlzBgdaZ7ITjaV1Oq+JSIi9aAwPkO2bfPB+kNEdQrirIhg82ROkiqpRUSk3hTGZ2hLfAY7E7O4flQk1vGZ8PGZsYiISD0ojM/QB+sP4eflztQhnSqeVPctERFpAIXxGcjML2bhtiNMHdKJQB9P82RxPhRmqpJaRETqTWF8Br7acpiC4jJmjOpa8aTWGIuISAMpjE/T8cKtgZ2DGHS8cAsUxiIi0mAK49O06VAGu5Kyq86KwVRSg6qpRUSk3uoVxpZlTbEsa7dlWbGWZT1aw+szLctKsSxrS/nHHY4fqmv5YP0h/L3cuaJy4RaYSmrQzFhEROrNo64DLMtyB+YAFwIJwAbLshbYth1T7dCPbNue3QhjdDmZecV8ve0IVw+PIMC72h9hdiK4eYJvW+cMTkREmp36zIxHAbG2be+3bbsImA9MbdxhubYvNidQWFLGjFGRJ7+Yk2wqqd10B0BEROqnPonRGYiv9Dih/LnqrrYsa5tlWZ9altXFIaNzQbZt88HPhzgrIpiBnYNPPiA7SQ0/RESkQRw1fVsIdLNt+yzgO+Dtmg6yLGuWZVnRlmVFp6SkOOitm9bGuHT2JOfUPCsGNfwQEZEGq08YHwYqz3Qjyp87wbbtVNu2C8sfvg4Mr+lEtm2/atv2CNu2R7Rr1+50xut0H6w/RIC3B5cP7lTzATlJavghIiINUp8w3gD0tiyru2VZXsB1wILKMiiuiAAAEu1JREFUB1iWVXkqeAWw03FDdB0ZeUV8vT2RK4d2wr964RZASSHkp6uSWkREGqTOamrbtkssy5oNLAXcgTdt295hWdYTQLRt2wuAByzLugIoAdKAmY04Zuf45TOO/fg5lEzj+lNdogaFsYiINEidYQxg2/ZiYHG15/5c6evHgMccOzQXkrQd+4t76FVayPMhbkR1urLm43LK1xir4YeIiDSA1t/UpTAbPr6FYq9gPi45l8vzF0DMgpqPPTEz1j1jERGpP4Xxqdg2fP1rSD/A253+xN/dZlEaPhS+mg3pB08+/kQYq5paRETqT2F8Kpvege2fYE98jLePRDCyV0fcr51nXvvkVigpqnp8ThJY7uAX1uRDFRGR5kthXJvkHbDkEegxkYP97yYhPZ9zeodBSDeY+hIc2QTLHq/6PdnJENBe3bdERKRBlBo1KcyBj28Bn2C46jVW70sD4Oze5WujB1wBo2bBupdh16KK78tO1BpjERFpMIVxdbYNi34Lafvg6tchoD0/7Ekhsq0f3cL8K4676EkIHwxf3gMZh8xzOcm6XywiIg2mMK5uy/uwbT6c+3vofg5FJWWs3ZfK2b2r3Qf28IZp80x4H79/rL7UIiJyGhTGlR3dCYsehu7nwDm/A2DzoXRyi0orLlFX1rYHXPFfOBwN3/0J8o5pjbGIiDRYvZp+tApFufDJTPAOhKteBzd3AFbvPYa7m8W4XqE1f1/Ur+Dgj7B+rnms7lsiItJAmhkf9+2fIGU3XP1alUvNP+xNYWiXNgT5eNb+vRc9BR0Hma8Vxv/f3t3H1nXfdRx/f23HcZxHO0nTLg9N0qbrstIWKFURHSqVmLoHtZMG0yaQ9gfShLSJIoagIDTE0P4YQgP+6D8TTAyJUSpgUEGlMY2hFqR1y1jThzVpMidV7DaNHdtxnNiOH778cY8dO00TJ7nJuffc90uK7j3nHtnf/tSbT37n93AkSVfIMJ538Fm46+Ow+6GFU8NnzvHywKmL36JebEUX/OrXYe9jsO3+61qmJKl6vE0NtVvUp9+Cm9635PT/Hh4iEz5wxzI28dh4G3zi765TgZKkKrNnDOe3tuzdveT084cGWdfVwd1b19/4miRJLcMwBhjuq70uCuPM5LnXh3hwzyY62m0mSdL1Y8rAojDetXDq8Ilxjo9NXn68WJKka2QYQy2MuzfWtr8sPHdoCOCdm31IklRnhjHUwvgi48W7N69mW093SUVJklqFYQwwfGRJGE9Oz/K9vpP8oreoJUk3gGE8MwWn+peE8Q/fGGFyes5b1JKkG8IwHnkDyCVh/NyhQVa0Bw/sfpctMCVJqiPD+CLLmp57fYifvbWH1SvdE0WSdP0ZxvNh3FNb1nTi9CSvvTXmkiZJ0g1jGA/3wcr10N0L1LbABJy8JUm6YQzj4b7aZh8RADz/+hC9qzt5/3vWlVyYJKlVGMaL1hjPzSXPHRriwds30dYWJRcmSWoVrR3Gs9Nw6thCGB84fpqh8SmXNEmSbqjWDuNTx2BuZiGMnz80CODkLUnSDdXaYXzBsqbnDw3x3i1ruXl9V4lFSZJaTYuH8ZHaa+8uJs7N8v2jw96iliTdcC0exn2wohvWbOGFIyc5NzPHB+7wFrUk6cYyjHt3QwTPHxqis6ON+3f2ll2VJKnFGMa9tZ23vtd3kp/b2cOqzvaSi5IktZrWDeO5WRg5Cr27yUzeOHmWPTetLbsqSVILat0wHnsTZs9B727GJmYYn5ph64ZVZVclSWpBrRvGi5Y1HRs5C8C2HsNYknTjGcY9uxgYnQBgq2EsSSpBa4dx+0pYt5WBkSKMvU0tSSpBa4dxz05oa6N/ZIJVK9rpXd1ZdlWSpBbUwmF8ZGEbzIHRs2ztWUWET2qSJN14rRnGmTByPoz7RyacvCVJKk1rhvH42zB9dmHDj4HRCceLJUmlac0wXljWtIvxqRlGz06zrae73JokSS2rxcN49/mZ1N6mliSVpHXDuK0D1u+g3w0/JEkla90w3rAD2jsWNvzY5pixJKkkrRvG88uaRibo7Ghj05qVJRclSWpVywrjiHgkIg5GxOGIeOIS1308IjIi7qtfiXWWuWSNcf9IbSZ1W5trjCVJ5bhsGEdEO/Ak8CFgL/CpiNh7kevWAo8DL9S7yLo6OwxTY+fD2GVNkqSSLadnfD9wODP7MvMc8BTw2EWu+1Pgy8BkHeurv0UPiAAYGDnr5C1JUqmWE8ZbgWOLjvuLcwsi4meA7Zn5H5f6QRHxmYjYFxH7BgcHr7jYuli0rGlyepah8XP2jCVJpbrmCVwR0QZ8Bfj85a7NzK9m5n2Zed/mzZuv9VdfneE+IKDnVvqLNcbbeg1jSVJ5lhPGA8D2RcfbinPz1gJ3Af8dEUeBB4BnGnYS13AfrN8OHSvPP8d4g7tvSZLKs5ww/gGwJyJ2RUQn8EngmfkPM/NUZm7KzJ2ZuRP4HvBoZu67LhVfq+G+hT2p3fBDktQILhvGmTkDfA74FvAa8HRmvhoRX4yIR693gXV3wRrjjrZgy7qukouSJLWyjuVclJnPAs9ecO4L73LtQ9de1nUyMQoTw4ueYzzBLRu6aHeNsSSpRK21A9fIkdrrwm1q1xhLksrXWmG8aFkT1G5TO3lLklS21gzjnp2cm5nj7dOTTt6SJJWuxcL4CKy9BTpX89apCTJ9jrEkqXwtFsZ9Sx4QAS5rkiSVrwXDeH5P6vnnGDtmLEkqV+uE8dQ4jL+95GlNbQE3r3eNsSSpXK0TxiNHa68953ff2rKui86O1mkCSVJjap0kusiyJseLJUmNoAXD2A0/JEmNpbXCuHsTdK1nZnaO42OTbOtx8pYkqXytFcbFLerjY5PMzqVrjCVJDaGFwvjIkvFiwNvUkqSG0BphPD0BY/1u+CFJakitEcbDxdOaNt4G1B6dCPAee8aSpAbQGmE8eKD2uvm9QO029ea1K+la0V5iUZIk1bRIGB+EaIONtwPQP3rW8WJJUsNokTA+AD07YUUtgN3wQ5LUSFokjA/C5jsBmJtL3hyddFmTJKlhVD+MZ6fh5OGF8eLB8SnOzc654YckqWFUP4yHj8Dc9ELPuH/kLADbHDOWJDWI6ofxBTOpXWMsSWo0LRDGB2uvm+4AzoexY8aSpEbRAmF8ADbsgM7VQG3Dj57uFXR3dpRcmCRJNS0QxudnUsP8siYnb0mSGke1w3huFoZeXxgvhtoELjf8kCQ1kmqH8chRmJ1a6BlnJgOjbvghSWos1Q7j+clbRRifPHOOyek5J29JkhpKxcO4WNZUzKQeWFjW5JixJKlxVDyMD8K6rdC1Dli0rMkxY0lSA6l4GB9YMnlrYLS2+5a3qSVJjaS6YTw3V8ykXrqsaW1XB+tXrSixMEmSlqpuGJ86BtNnL1jWNOEtaklSw6luGF8wkxooljU5eUuS1FgqHMZLZ1JnJv0jrjGWJDWeCofxQVizBbp7ARibmGF8asYwliQ1nAqH8dKZ1MeK5xg7ZixJajTVDOPMdz4gYtQNPyRJjamaYTz2Jpw7/Y6Z1OAaY0lS46lmGM9P3rpgjXF3Zzs93a4xliQ1loqG8cWWNdUenRgRJRUlSdLFVTSMD0D3Rli9aeGUy5okSY2qomG8dPIW1CZwOV4sSWpE1QvjzHcsaxqfmmH07DRbNziTWpLUeKoXxuMnYHJ0Sc/44PExAG7bvLqsqiRJelfVC+OFmdTne8b7j50C4J7tG8qoSJKkS1pWGEfEIxFxMCIOR8QTF/n8NyPi5Yh4MSL+JyL21r/UZbrITOqX+kfZsm4lW9Z1lVSUJEnv7rJhHBHtwJPAh4C9wKcuErbfyMyfysx7gT8DvlL3Spdr8AB0ra/tS114qf8Ud2+zVyxJakzL6RnfDxzOzL7MPAc8BTy2+ILMHFt0uBrI+pV4heZnUhfriccmp+kbOsPdW9eXVpIkSZeynDDeChxbdNxfnFsiIj4bET+h1jP+rYv9oIj4TETsi4h9g4ODV1Pv5V0wk/qV/tp48d2OF0uSGlTdJnBl5pOZeRvw+8Afvcs1X83M+zLzvs2bN9frV593ZgjODi0ZL94/H8b2jCVJDWo5YTwAbF90vK04926eAj52LUVdtYXJW+d7xi/1j7Kjt5ue1Z2llCRJ0uUsJ4x/AOyJiF0R0Ql8Enhm8QURsWfR4UeAQ/Ur8Qpc5AERtclb9oolSY3rsmGcmTPA54BvAa8BT2fmqxHxxYh4tLjscxHxakS8CPwO8OnrVvGlDB6EzjWwrjakPTQ+xcDoBPc4k1qS1MA6lnNRZj4LPHvBuS8sev94neu6OvOTt4qZ1C/1jwLYM5YkNbRq7cB1wQMi9h87RVvAXU7ekiQ1sOqE8cQIjB9/x+St229aw+qVy7oBIElSKaoTxoOv116LnnFmuvOWJKkpVCiMlz4gYmB0gpNnznGP48WSpAZXoTA+CB2rYP0OoLakCbBnLElqeBUK4wOw+Q5oq/0n7e8fZUV7cOcta0suTJKkS6tQGC+dSf3SsVO875Z1rOxoL7EoSZIurxphPDkGY/0L48Vzc8krA+68JUlqDtUI46Fi982iZ9w3dIbTUzOOF0uSmkI1wviCPannd95yG0xJUjOoRhjveAA+/Oew4VagNpO6u7Od229aU3JhkiRdXjW2ptp4W+1PYX//KHe9Zz3tbVFiUZIkLU81esaLTM/O8eM3x5y8JUlqGpUL44PHTzM1M8fd2x0vliQ1h8qF8fzOW26DKUlqFhUM41HWr1rBjt7uskuRJGlZKhfG+/trm31EOHlLktQcKhXGk9OzvP72aSdvSZKaSqXC+NU3x5idS3fekiQ1lUqFsTtvSZKaUcXC+BQ3rV3Jzeu7yi5FkqRlq1QY7+8f9Ra1JKnpVCaMxyan6Rs84/piSVLTqUwYv1Js9uHOW5KkZlOZMN4/H8Zb7RlLkppLZcL4pf5RdvR207O6s+xSJEm6IhUK41Nu9iFJakqVCOOh8SkGRidcXyxJakqVCOP5zT7sGUuSmlElwnhmNrnz5rXc5eQtSVIT6ii7gHr44Ptv5oPvv7nsMiRJuiqV6BlLktTMDGNJkkpmGEuSVDLDWJKkkhnGkiSVzDCWJKlkhrEkSSUzjCVJKplhLElSyQxjSZJKZhhLklQyw1iSpJIZxpIklcwwliSpZIaxJEklM4wlSSqZYSxJUskMY0mSShaZWc4vjhgE3qjjj9wEDNXx57Uy27J+bMv6sS3rx7asjyttx1szc/PFPigtjOstIvZl5n1l11EFtmX92Jb1Y1vWj21ZH/VsR29TS5JUMsNYkqSSVSmMv1p2ARViW9aPbVk/tmX92Jb1Ubd2rMyYsSRJzapKPWNJkppSJcI4Ih6JiIMRcTginii7nmYSEV+LiBMR8cqic70R8e2IOFS89pRZYzOIiO0R8d2I+HFEvBoRjxfnbcsrFBFdEfH9iNhftOWfFOd3RcQLxff8HyOis+xam0VEtEfEjyLi34tj2/IqRMTRiHg5Il6MiH3Fubp8x5s+jCOiHXgS+BCwF/hUROwtt6qm8rfAIxecewL4TmbuAb5THOvSZoDPZ+Ze4AHgs8X/h7bllZsCHs7Me4B7gUci4gHgy8BfZObtwAjwGyXW2GweB15bdGxbXr1fysx7Fy1pqst3vOnDGLgfOJyZfZl5DngKeKzkmppGZj4HDF9w+jHg68X7rwMfu6FFNaHMfCsz/694f5raX3xbsS2vWNaMF4crij8JPAz8U3HetlymiNgGfAT46+I4sC3rqS7f8SqE8Vbg2KLj/uKcrt6WzHyreH8c2FJmMc0mInYCPw28gG15VYrbqi8CJ4BvAz8BRjNzprjE7/ny/SXwe8BccbwR2/JqJfCfEfHDiPhMca4u3/GOelSn6srMjAin3C9TRKwB/hn47cwcq3VCamzL5cvMWeDeiNgAfBO4s+SSmlJEfBQ4kZk/jIiHyq6nAh7MzIGIuAn4dkQcWPzhtXzHq9AzHgC2LzreVpzT1Xs7Im4BKF5PlFxPU4iIFdSC+O8z81+K07blNcjMUeC7wM8DGyJivgPh93x5fgF4NCKOUhvCexj4K2zLq5KZA8XrCWr/SLyfOn3HqxDGPwD2FLMDO4FPAs+UXFOzewb4dPH+08C/lVhLUyjG4f4GeC0zv7LoI9vyCkXE5qJHTESsAn6Z2hj8d4FfKS6zLZchM/8gM7dl5k5qfzf+V2b+GrblFYuI1RGxdv498EHgFer0Ha/Eph8R8WFq4yLtwNcy80sll9Q0IuIfgIeoPX3kbeCPgX8FngZ2UHuy1icy88JJXlokIh4Engde5vzY3B9SGze2La9ARNxNbSJMO7UOw9OZ+cWI2E2td9cL/Aj49cycKq/S5lLcpv7dzPyobXnlijb7ZnHYAXwjM78UERupw3e8EmEsSVIzq8JtakmSmpphLElSyQxjSZJKZhhLklQyw1iSpJIZxpIklcwwliSpZIaxJEkl+39Me7LXUJNU/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(8, 8))\n",
    "ax.plot(history.history[\"acc\"])\n",
    "ax.legend(\"acc\")\n",
    "ax.plot(history.history[\"val_acc\"])\n",
    "ax.legend(\"val_acc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}