{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nqrTKIyfYRRa"
      },
      "source": [
        "# Build Named Entity Recognation Model with BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KJbZYeNbYfkt"
      },
      "source": [
        "# **Objective**\n",
        "\n",
        "The objective of this notebook is to be able to build ner  model using BERT for the given dataset .\n",
        "\n",
        "The main section of this notebook organize as follows:\n",
        "\n",
        "- Load NER Data from kaggle.\n",
        "- Select the relevant columns.\n",
        "- Identify unique entity tagers in the dataset.\n",
        "- Remove Duplicate Sentences.\n",
        "- Finetune Bert Model\n",
        "- Apply the model to a new sentences.\n",
        "- Conclusion\n",
        "- Acknowledgements\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GX1Gm0sVTU4O"
      },
      "source": [
        "# Imports and Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qIFLx0_wimTB"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "pd.set_option('max_colwidth',150)\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import re\n",
        "import pickle\n",
        "import os\n",
        "from tqdm import tqdm, trange\n",
        "from IPython.core.interactiveshell import InteractiveShell\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QqvaLRjVjIj3"
      },
      "source": [
        "# Download the Datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T8aMPXC7t_VX"
      },
      "outputs": [],
      "source": [
        "pathdir = \"/content/data\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x3CI3PtUp2lW"
      },
      "outputs": [],
      "source": [
        "# def download_dataset():\n",
        "  \n",
        "#   if not os.path.isfile('ner.csv'):\n",
        "\n",
        "#     # Downloading Annotated Corpus for Named Entity Recognition dataset\n",
        "#     !gdown https://drive.google.com/uc?id=13y8JNgL5TQ4x-yufpBOv3QBsEiE051sE\n",
        "\n",
        "#   if not os.path.exists(pathdir):\n",
        "#     # Make a data folder to store the data\n",
        "#     !mkdir data\n",
        "\n",
        "#     !mv /content/ner.csv ./data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zS6WbHz8wHzu"
      },
      "outputs": [],
      "source": [
        "# download_dataset()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "liJiX3Xf2hQh"
      },
      "source": [
        "# Load Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LMwtt2rJnNhB"
      },
      "outputs": [],
      "source": [
        "#specify the path to data location\n",
        "\n",
        "filepath = '/content/ner.csv'\n",
        "data = pd.read_csv(filepath, encoding = \"latin1\", on_bad_lines='skip')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g4VoxOSnnOs9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        },
        "outputId": "943dc9f9-26f1-4c41-be13-af52fe2390e0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Unnamed: 0     lemma next-lemma next-next-lemma next-next-pos  \\\n",
              "0           0  thousand         of        demonstr           NNS   \n",
              "1           1        of   demonstr            have           VBP   \n",
              "2           2  demonstr       have           march           VBN   \n",
              "3           3      have      march         through            IN   \n",
              "4           4     march    through          london           NNP   \n",
              "\n",
              "  next-next-shape next-next-word next-pos next-shape      next-word  ...  \\\n",
              "0       lowercase  demonstrators       IN  lowercase             of  ...   \n",
              "1       lowercase           have      NNS  lowercase  demonstrators  ...   \n",
              "2       lowercase        marched      VBP  lowercase           have  ...   \n",
              "3       lowercase        through      VBN  lowercase        marched  ...   \n",
              "4     capitalized         London       IN  lowercase        through  ...   \n",
              "\n",
              "  prev-prev-lemma prev-prev-pos prev-prev-shape prev-prev-word   prev-shape  \\\n",
              "0      __start2__    __START2__        wildcard     __START2__     wildcard   \n",
              "1      __start1__    __START1__        wildcard     __START1__  capitalized   \n",
              "2        thousand           NNS     capitalized      Thousands    lowercase   \n",
              "3              of            IN       lowercase             of    lowercase   \n",
              "4        demonstr           NNS       lowercase  demonstrators    lowercase   \n",
              "\n",
              "       prev-word sentence_idx        shape           word tag  \n",
              "0     __START1__          1.0  capitalized      Thousands   O  \n",
              "1      Thousands          1.0    lowercase             of   O  \n",
              "2             of          1.0    lowercase  demonstrators   O  \n",
              "3  demonstrators          1.0    lowercase           have   O  \n",
              "4           have          1.0    lowercase        marched   O  \n",
              "\n",
              "[5 rows x 25 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-510934d7-169e-4138-9ea1-3dfbbfaa4742\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>lemma</th>\n",
              "      <th>next-lemma</th>\n",
              "      <th>next-next-lemma</th>\n",
              "      <th>next-next-pos</th>\n",
              "      <th>next-next-shape</th>\n",
              "      <th>next-next-word</th>\n",
              "      <th>next-pos</th>\n",
              "      <th>next-shape</th>\n",
              "      <th>next-word</th>\n",
              "      <th>...</th>\n",
              "      <th>prev-prev-lemma</th>\n",
              "      <th>prev-prev-pos</th>\n",
              "      <th>prev-prev-shape</th>\n",
              "      <th>prev-prev-word</th>\n",
              "      <th>prev-shape</th>\n",
              "      <th>prev-word</th>\n",
              "      <th>sentence_idx</th>\n",
              "      <th>shape</th>\n",
              "      <th>word</th>\n",
              "      <th>tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>thousand</td>\n",
              "      <td>of</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>NNS</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>...</td>\n",
              "      <td>__start2__</td>\n",
              "      <td>__START2__</td>\n",
              "      <td>wildcard</td>\n",
              "      <td>__START2__</td>\n",
              "      <td>wildcard</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>1.0</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>of</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>have</td>\n",
              "      <td>VBP</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>NNS</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>...</td>\n",
              "      <td>__start1__</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>wildcard</td>\n",
              "      <td>__START1__</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>have</td>\n",
              "      <td>march</td>\n",
              "      <td>VBN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>VBP</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>...</td>\n",
              "      <td>thousand</td>\n",
              "      <td>NNS</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>have</td>\n",
              "      <td>march</td>\n",
              "      <td>through</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>VBN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>...</td>\n",
              "      <td>of</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>of</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>march</td>\n",
              "      <td>through</td>\n",
              "      <td>london</td>\n",
              "      <td>NNP</td>\n",
              "      <td>capitalized</td>\n",
              "      <td>London</td>\n",
              "      <td>IN</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>through</td>\n",
              "      <td>...</td>\n",
              "      <td>demonstr</td>\n",
              "      <td>NNS</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>have</td>\n",
              "      <td>1.0</td>\n",
              "      <td>lowercase</td>\n",
              "      <td>marched</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 25 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-510934d7-169e-4138-9ea1-3dfbbfaa4742')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-510934d7-169e-4138-9ea1-3dfbbfaa4742 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-510934d7-169e-4138-9ea1-3dfbbfaa4742');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "#Verify that the data is loaded correctly\n",
        "data.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iJZa9dP1vGeN",
        "outputId": "b9ff3af2-0d8b-4536-e7a4-0d546b4e82f5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1050795, 25)"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "#totally the data have 1050795 rows and 25 columns\n",
        "data.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XwYxq7Wqx8QH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b97e927d-88b5-44a6-a9dd-3d1bf4a9fd25"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 1050795 entries, 0 to 1050794\n",
            "Data columns (total 25 columns):\n",
            " #   Column           Non-Null Count    Dtype  \n",
            "---  ------           --------------    -----  \n",
            " 0   Unnamed: 0       1050795 non-null  int64  \n",
            " 1   lemma            1050795 non-null  object \n",
            " 2   next-lemma       1050795 non-null  object \n",
            " 3   next-next-lemma  1050795 non-null  object \n",
            " 4   next-next-pos    1050795 non-null  object \n",
            " 5   next-next-shape  1050795 non-null  object \n",
            " 6   next-next-word   1050795 non-null  object \n",
            " 7   next-pos         1050795 non-null  object \n",
            " 8   next-shape       1050794 non-null  object \n",
            " 9   next-word        1050794 non-null  object \n",
            " 10  pos              1050794 non-null  object \n",
            " 11  prev-iob         1050794 non-null  object \n",
            " 12  prev-lemma       1050794 non-null  object \n",
            " 13  prev-pos         1050794 non-null  object \n",
            " 14  prev-prev-iob    1050794 non-null  object \n",
            " 15  prev-prev-lemma  1050794 non-null  object \n",
            " 16  prev-prev-pos    1050794 non-null  object \n",
            " 17  prev-prev-shape  1050794 non-null  object \n",
            " 18  prev-prev-word   1050794 non-null  object \n",
            " 19  prev-shape       1050794 non-null  object \n",
            " 20  prev-word        1050794 non-null  object \n",
            " 21  sentence_idx     1050794 non-null  float64\n",
            " 22  shape            1050794 non-null  object \n",
            " 23  word             1050794 non-null  object \n",
            " 24  tag              1050794 non-null  object \n",
            "dtypes: float64(1), int64(1), object(23)\n",
            "memory usage: 200.4+ MB\n"
          ]
        }
      ],
      "source": [
        "data.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o6obun2r48jC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "510a2e3c-05da-42d6-ed36-9a30b5635dc4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Unnamed: 0         0\n",
              "lemma              0\n",
              "next-lemma         0\n",
              "next-next-lemma    0\n",
              "next-next-pos      0\n",
              "next-next-shape    0\n",
              "next-next-word     0\n",
              "next-pos           0\n",
              "next-shape         1\n",
              "next-word          1\n",
              "pos                1\n",
              "prev-iob           1\n",
              "prev-lemma         1\n",
              "prev-pos           1\n",
              "prev-prev-iob      1\n",
              "prev-prev-lemma    1\n",
              "prev-prev-pos      1\n",
              "prev-prev-shape    1\n",
              "prev-prev-word     1\n",
              "prev-shape         1\n",
              "prev-word          1\n",
              "sentence_idx       1\n",
              "shape              1\n",
              "word               1\n",
              "tag                1\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "data.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "id": "d5aY-xEJ6-ke",
        "outputId": "8da135c0-5996-4f48-c77e-9fc7267c58de"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         Unnamed: 0  sentence_idx\n",
              "count  1.050795e+06  1.050794e+06\n",
              "mean   4.153109e+05  1.898184e+04\n",
              "std    3.447835e+05  1.576237e+04\n",
              "min    0.000000e+00  1.000000e+00\n",
              "25%    1.313490e+05  5.997000e+03\n",
              "50%    2.626980e+05  1.201700e+04\n",
              "75%    7.858755e+05  3.592600e+04\n",
              "max    1.048574e+06  4.795900e+04"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b3fdafd6-9ec1-40de-ab94-29a9408f7b83\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>sentence_idx</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>1.050795e+06</td>\n",
              "      <td>1.050794e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>4.153109e+05</td>\n",
              "      <td>1.898184e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>3.447835e+05</td>\n",
              "      <td>1.576237e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000e+00</td>\n",
              "      <td>1.000000e+00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>1.313490e+05</td>\n",
              "      <td>5.997000e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>2.626980e+05</td>\n",
              "      <td>1.201700e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>7.858755e+05</td>\n",
              "      <td>3.592600e+04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1.048574e+06</td>\n",
              "      <td>4.795900e+04</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b3fdafd6-9ec1-40de-ab94-29a9408f7b83')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b3fdafd6-9ec1-40de-ab94-29a9408f7b83 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b3fdafd6-9ec1-40de-ab94-29a9408f7b83');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "data.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9QsrxPE0SPS"
      },
      "source": [
        "# Select the data which contains only Sentence, Word, Pos and tag columns"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dWK0fXlR0jek"
      },
      "outputs": [],
      "source": [
        "ner_data = data[['sentence_idx', 'word', 'pos','tag']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jatQyuv654PV",
        "outputId": "62c673f2-a4a0-4554-a750-1c6b50e4721f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1050795, 4)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "ner_data.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FerXPCTA59DG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "a93ec506-4d61-4ae3-808c-1b20f30b7978"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   sentence_idx           word  pos tag\n",
              "0           1.0      Thousands  NNS   O\n",
              "1           1.0             of   IN   O\n",
              "2           1.0  demonstrators  NNS   O\n",
              "3           1.0           have  VBP   O\n",
              "4           1.0        marched  VBN   O"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d31ef711-8424-4056-ab93-7527868b7233\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence_idx</th>\n",
              "      <th>word</th>\n",
              "      <th>pos</th>\n",
              "      <th>tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>Thousands</td>\n",
              "      <td>NNS</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>of</td>\n",
              "      <td>IN</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1.0</td>\n",
              "      <td>demonstrators</td>\n",
              "      <td>NNS</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1.0</td>\n",
              "      <td>have</td>\n",
              "      <td>VBP</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1.0</td>\n",
              "      <td>marched</td>\n",
              "      <td>VBN</td>\n",
              "      <td>O</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d31ef711-8424-4056-ab93-7527868b7233')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d31ef711-8424-4056-ab93-7527868b7233 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d31ef711-8424-4056-ab93-7527868b7233');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "\n",
        "ner_data.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R1R7mjz91LgG",
        "outputId": "788c54e5-bc50-4468-d2dc-f300697c4cc3"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "sentence_idx    1\n",
              "word            1\n",
              "pos             1\n",
              "tag             1\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "ner_data.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2MQUCtH71R3Y"
      },
      "outputs": [],
      "source": [
        "#drop null value\n",
        "ner_data = ner_data.dropna()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d9yp68G95lYQ",
        "outputId": "15de11e3-e1a7-4850-cc55-cd805a0e36d7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "35177"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "# the total number of unique sentence\n",
        "len(ner_data['sentence_idx'].unique())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_RDZ0EwW2Kzo",
        "outputId": "c4175579-2c95-4e62-d209-1bb300cc9d00"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "30172"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "# the total number of unique word\n",
        "len(ner_data['word'].unique())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CYF3NaEo2ZCl",
        "outputId": "f68db507-1647-4086-b48c-cbda744a6cce"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "17"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "# the total number of unique tag\n",
        "len(ner_data['tag'].unique())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGxYry7F1rtz"
      },
      "source": [
        "# Remove Duplicated Sentences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hQX8UodKt6Jy"
      },
      "outputs": [],
      "source": [
        "class Build_sentence(object):\n",
        "\n",
        "    def __init__(self, data):\n",
        "        self.n_sent = 1\n",
        "        self.data = data\n",
        "        self.empty = False\n",
        "        agg_func = lambda s: [(w, p, t) for w, p, t in zip(s[\"word\"].values.tolist(),\n",
        "                                                           s[\"pos\"].values.tolist(),\n",
        "                                                           s[\"tag\"].values.tolist())]\n",
        "        self.grouped = self.data.groupby(\"sentence_idx\").apply(agg_func)\n",
        "        self.sentences = [s for s in self.grouped]\n",
        "\n",
        "    def get_next(self):\n",
        "        try:\n",
        "            s = self.grouped[\"Sentence: {}\".format(self.n_sent)]\n",
        "            self.n_sent += 1\n",
        "            return s\n",
        "        except:\n",
        "            return None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vpwFE5R8t7ha"
      },
      "outputs": [],
      "source": [
        "getter = Build_sentence(ner_data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eZrfg7pNufah"
      },
      "outputs": [],
      "source": [
        "sentences = [[word[0] for word in sentence] for sentence in getter.sentences]\n",
        "tag = [[tag[2] for tag in tags] for tags in getter.sentences]\n",
        "\n",
        "for ind in range(len(sentences)):\n",
        "  sent=' '.join(sentences[ind])\n",
        "  sent = sent.split('.')[0]\n",
        "  sentences[ind] = sent.split(\" \")\n",
        "  sentences[ind][-1] = sentences[ind][-1].replace('', '.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q77M0i6hOZOr",
        "outputId": "c11e91b2-e193-4bc4-8b4f-57ebcdc608f1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "92\n"
          ]
        }
      ],
      "source": [
        "max=0\n",
        "for i in range(len(sentences)):\n",
        "  if len(sentences[i]) > max:\n",
        "    max = len(sentences[i])\n",
        "print(max)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zFIliok0AK9K",
        "outputId": "92b28a4b-f02b-4fd6-b1e5-1a7fc38fb591"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['O', 'O', 'O', 'O', 'O', 'O', 'B-geo', 'O', 'O', 'O', 'O', 'O', 'B-geo', 'O', 'O', 'O', 'O', 'O', 'B-gpe', 'O', 'O', 'O', 'O', 'O']\n"
          ]
        }
      ],
      "source": [
        "labels=[]\n",
        "for ind in range(len(sentences)):\n",
        "  labels.append(tag[ind][0:len(sentences[ind])])\n",
        "print(labels[0])"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "CfxErQETuoPy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i9sqMMDRPa2t",
        "outputId": "431669c9-cf6f-45ed-df85-1a814bc606f3"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'I-per': 0,\n",
              " 'I-gpe': 1,\n",
              " 'I-eve': 2,\n",
              " 'I-geo': 3,\n",
              " 'I-art': 4,\n",
              " 'B-eve': 5,\n",
              " 'I-tim': 6,\n",
              " 'B-gpe': 7,\n",
              " 'B-geo': 8,\n",
              " 'B-org': 9,\n",
              " 'B-tim': 10,\n",
              " 'B-per': 11,\n",
              " 'O': 12,\n",
              " 'I-nat': 13,\n",
              " 'B-art': 14,\n",
              " 'I-org': 15,\n",
              " 'B-nat': 16,\n",
              " 'PAD': 17}"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "tag_values = list(set(ner_data[\"tag\"].values))\n",
        "tag_values.append(\"PAD\")\n",
        "tag2idx = {t: i for i, t in enumerate(tag_values)}\n",
        "tag2idx"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tag_values = list(set(ner_data[\"tag\"].values))\n",
        "tag_values.append(\"PAD\")\n",
        "idx2tag = {i: t for i, t in enumerate(tag_values)}\n",
        "idx2tag"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3bgovK-DJitP",
        "outputId": "7cf06a98-0f92-4503-d457-b507103f600b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 'I-per',\n",
              " 1: 'I-gpe',\n",
              " 2: 'I-eve',\n",
              " 3: 'I-geo',\n",
              " 4: 'I-art',\n",
              " 5: 'B-eve',\n",
              " 6: 'I-tim',\n",
              " 7: 'B-gpe',\n",
              " 8: 'B-geo',\n",
              " 9: 'B-org',\n",
              " 10: 'B-tim',\n",
              " 11: 'B-per',\n",
              " 12: 'O',\n",
              " 13: 'I-nat',\n",
              " 14: 'B-art',\n",
              " 15: 'I-org',\n",
              " 16: 'B-nat',\n",
              " 17: 'PAD'}"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tag_values[12]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "S-SIswPXU1rz",
        "outputId": "8b637fb9-24c2-4d86-9bb2-74387016f54b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'O'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# save idx2tag dictionary using the package pickel \n",
        "path_file = '/content/idx2tag.pkl'\n",
        "pickle.dump(idx2tag, open(path_file, 'wb')) "
      ],
      "metadata": {
        "id": "t_CI1Q8hKMEr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yNVF2iPvRXF1"
      },
      "source": [
        "# Apply Bert\n",
        "- Prepare the sentences and labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nHgkem_JRnd9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45f6eeff-ded4-40cc-8fc3-224614fc2d98"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers==2.6.0\n",
            "  Downloading transformers-2.6.0-py3-none-any.whl (540 kB)\n",
            "\u001b[K     |████████████████████████████████| 540 kB 4.1 MB/s \n",
            "\u001b[?25hCollecting tokenizers==0.5.2\n",
            "  Downloading tokenizers-0.5.2-cp37-cp37m-manylinux1_x86_64.whl (5.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 5.6 MB 32.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from transformers==2.6.0) (1.21.6)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==2.6.0) (2022.6.2)\n",
            "Collecting boto3\n",
            "  Downloading boto3-1.26.8-py3-none-any.whl (132 kB)\n",
            "\u001b[K     |████████████████████████████████| 132 kB 64.8 MB/s \n",
            "\u001b[?25hCollecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.3 MB 53.0 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.53.tar.gz (880 kB)\n",
            "\u001b[K     |████████████████████████████████| 880 kB 67.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==2.6.0) (3.8.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers==2.6.0) (2.23.0)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers==2.6.0) (4.64.1)\n",
            "Collecting s3transfer<0.7.0,>=0.6.0\n",
            "  Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB)\n",
            "\u001b[K     |████████████████████████████████| 79 kB 9.1 MB/s \n",
            "\u001b[?25hCollecting botocore<1.30.0,>=1.29.8\n",
            "  Downloading botocore-1.29.8-py3-none-any.whl (9.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 9.9 MB 35.3 MB/s \n",
            "\u001b[?25hCollecting jmespath<2.0.0,>=0.7.1\n",
            "  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)\n",
            "Collecting urllib3<1.27,>=1.25.4\n",
            "  Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)\n",
            "\u001b[K     |████████████████████████████████| 140 kB 80.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.30.0,>=1.29.8->boto3->transformers==2.6.0) (2.8.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.8->boto3->transformers==2.6.0) (1.15.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.6.0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.6.0) (2022.9.24)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers==2.6.0) (3.0.4)\n",
            "  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)\n",
            "\u001b[K     |████████████████████████████████| 127 kB 69.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==2.6.0) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers==2.6.0) (1.2.0)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.53-py3-none-any.whl size=895259 sha256=53811b0fa5a699bea43cc69ffa3c4b84f7bcc820e83e860f951b15ca98d687e4\n",
            "  Stored in directory: /root/.cache/pip/wheels/87/39/dd/a83eeef36d0bf98e7a4d1933a4ad2d660295a40613079bafc9\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: urllib3, jmespath, botocore, s3transfer, tokenizers, sentencepiece, sacremoses, boto3, transformers\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 1.24.3\n",
            "    Uninstalling urllib3-1.24.3:\n",
            "      Successfully uninstalled urllib3-1.24.3\n",
            "Successfully installed boto3-1.26.8 botocore-1.29.8 jmespath-1.0.1 s3transfer-0.6.0 sacremoses-0.0.53 sentencepiece-0.1.97 tokenizers-0.5.2 transformers-2.6.0 urllib3-1.25.11\n"
          ]
        }
      ],
      "source": [
        "pip install transformers==2.6.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "LG9aY-laRWFv",
        "outputId": "884a1357-d4c3-4141-c040-c5bfb14605d4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'1.12.1+cu113'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "import torch\n",
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
        "from transformers import BertTokenizer, BertConfig\n",
        "\n",
        "from keras_preprocessing.sequence import pad_sequences\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "torch.__version__\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q3hxi1CvRWJF"
      },
      "outputs": [],
      "source": [
        "# Define maximumn sequence length and batch size\n",
        "MAX_LEN = 95\n",
        "bs = 32"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IrdtFGGaRWMJ"
      },
      "outputs": [],
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "n_gpu = torch.cuda.device_count()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "7QnTbysiRWPd",
        "outputId": "962f3361-0929-4714-95b4-ea4b693738f7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Tesla T4'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 32
        }
      ],
      "source": [
        "torch.cuda.get_device_name(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SegEq-U2RWTa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "37f8492753364534bdfd6cce7d3b9879",
            "2a9a4eec9e894e2c820606c2f19ab7fc",
            "52f4974caffb47e8bf2f1d7a2531ed75",
            "040f040bb3204454ab2e50765f76addf",
            "41a19caf7f834097b081f42eb03a9b6d",
            "076ec77a50cb4495a22b678fa87578cf",
            "be47d8bf7c354111a852e2a1aeea2bb3",
            "d84dce6797924e648eb15756abb32972",
            "0625d3b1a61746f68a6784e4760a1369",
            "8c37d9b2f9684048890691d38597ed98",
            "1106a61ccbb341349f64aaf5b7993edd"
          ]
        },
        "outputId": "3dfb77bf-b1f6-4236-bfba-1a0a1cd077a2"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/213k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "37f8492753364534bdfd6cce7d3b9879"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8ASrVnv5WKX-"
      },
      "outputs": [],
      "source": [
        "# Now we tokenize all sentences. Since the BERT tokenizer is based a Wordpiece tokenizer it will split tokens in subword tokens\n",
        "\n",
        "def tokenize_and_preserve_labels(sentence, text_labels):\n",
        "    tokenized_sentence = []\n",
        "    labels = []\n",
        "\n",
        "    for word, label in zip(sentence, text_labels):\n",
        "\n",
        "        # Tokenize the word and count # of subwords the word is broken into\n",
        "        tokenized_word = tokenizer.tokenize(word)\n",
        "        n_subwords = len(tokenized_word)\n",
        "\n",
        "        # Add the tokenized word to the final tokenized word list\n",
        "        tokenized_sentence.extend(tokenized_word)\n",
        "\n",
        "        # Add the same label to the new list of labels `n_subwords` times\n",
        "        labels.extend([label] * n_subwords)\n",
        "\n",
        "    return tokenized_sentence, labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U3Ng8-lNWKcd"
      },
      "outputs": [],
      "source": [
        "tokenized_texts_and_labels = [tokenize_and_preserve_labels(sent, labs) for sent, labs in zip(sentences, labels)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bnku7QsdXlF2"
      },
      "outputs": [],
      "source": [
        "tokenized_texts = [token_label_pair[0] for token_label_pair in tokenized_texts_and_labels]\n",
        "labels = [token_label_pair[1] for token_label_pair in tokenized_texts_and_labels]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iCY59xlnWKfx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9d1a7501-56c6-4f7c-f4fa-a711d91bd6a9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['thousands',\n",
              " 'of',\n",
              " 'demons',\n",
              " '##tra',\n",
              " '##tors',\n",
              " 'have',\n",
              " 'marched',\n",
              " 'through',\n",
              " 'lo',\n",
              " '##ndon',\n",
              " 'to',\n",
              " 'protest',\n",
              " 'the',\n",
              " 'war',\n",
              " 'in',\n",
              " 'i',\n",
              " '##ra',\n",
              " '##q',\n",
              " 'and',\n",
              " 'demand',\n",
              " 'the',\n",
              " 'withdrawal',\n",
              " 'of',\n",
              " 'br',\n",
              " '##itis',\n",
              " '##h',\n",
              " 'troops',\n",
              " 'from',\n",
              " 'that',\n",
              " 'country',\n",
              " '.']"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ],
      "source": [
        "tokenized_texts[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1kK_I60-Z8dm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "59d8320a-6fc3-4515-aba5-35281b5bdb18"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'B-geo',\n",
              " 'B-geo',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'B-geo',\n",
              " 'B-geo',\n",
              " 'B-geo',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'B-gpe',\n",
              " 'B-gpe',\n",
              " 'B-gpe',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O',\n",
              " 'O']"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ],
      "source": [
        "labels[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QPGJWcZxWKjz"
      },
      "outputs": [],
      "source": [
        "# cut and pad the token and label sequences to our desired length\n",
        "input_ids = pad_sequences([tokenizer.convert_tokens_to_ids(txt) for txt in tokenized_texts],\n",
        "                          maxlen=MAX_LEN, dtype=\"long\", value=0.0,\n",
        "                          truncating=\"post\", padding=\"post\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g2hx6kTZZHLM"
      },
      "outputs": [],
      "source": [
        "tags = pad_sequences([[tag2idx.get(l) for l in lab] for lab in labels],\n",
        "                     maxlen=MAX_LEN, value=tag2idx[\"PAD\"], padding=\"post\",\n",
        "                     dtype=\"long\", truncating=\"post\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "2iDm9Ys6s8cn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2W84DLmcZHTE"
      },
      "outputs": [],
      "source": [
        "# create the mask to ignore the padded elements in the sequences.\n",
        "attention_masks = [[float(i != 0.0) for i in ii] for ii in input_ids]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VIUgIXlKasmo"
      },
      "source": [
        "# Split the datasets\n",
        "- split the dataset to use 10% to validate the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xq7S0zgdaKw3"
      },
      "outputs": [],
      "source": [
        "tr_inputs, val_inputs, tr_tags, val_tags = train_test_split(input_ids, tags,\n",
        "                                                            random_state=42, test_size=0.1)\n",
        "tr_masks, val_masks, _, _ = train_test_split(attention_masks, input_ids,\n",
        "                                             random_state=42, test_size=0.1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-5AhmtiraK1c",
        "outputId": "8a01776c-c2c2-4300-da2b-ba0faee3730f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "31659"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ],
      "source": [
        "len(tr_inputs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Al3DAYStb_wL"
      },
      "outputs": [],
      "source": [
        "#  convert the dataset to torch tensors.\n",
        "tr_inputs = torch.tensor(tr_inputs)\n",
        "val_inputs = torch.tensor(val_inputs)\n",
        "tr_tags = torch.tensor(tr_tags)\n",
        "val_tags = torch.tensor(val_tags)\n",
        "tr_masks = torch.tensor(tr_masks)\n",
        "val_masks = torch.tensor(val_masks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I2GrT2l3b_z3"
      },
      "outputs": [],
      "source": [
        "# define the dataloaders. \n",
        "train_data = TensorDataset(tr_inputs, tr_masks, tr_tags)\n",
        "train_sampler = RandomSampler(train_data)\n",
        "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=bs)\n",
        "\n",
        "valid_data = TensorDataset(val_inputs, val_masks, val_tags)\n",
        "valid_sampler = SequentialSampler(valid_data)\n",
        "valid_dataloader = DataLoader(valid_data, sampler=valid_sampler, batch_size=bs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wRKhPfOXdzpT"
      },
      "source": [
        "# Setup the Bert model for finetuning\n",
        "- BertForTokenClassification class for token-level predictions.\n",
        "- BertForTokenClassification is a fine-tuning model that wraps BertModel and adds token-level classifier on top of the BertModel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "2OI57oHYb_4G",
        "outputId": "d3e42f34-e6a9-402b-97ed-8376dd06b2a6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'2.6.0'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 46
        }
      ],
      "source": [
        "import transformers\n",
        "from transformers import BertForTokenClassification, AdamW\n",
        "transformers.__version__"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-bAKXuM-dPm4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81,
          "referenced_widgets": [
            "ddb2c5db384b4d2e93a11e97059e162b",
            "2f640d4012de4902b8f7f9f2784f1f25",
            "4abff02aca52432b988e4ee00195903f",
            "690cb7b841994afabfc1b6611af13884",
            "c37df6ff54894fe68d233b1bc549c16f",
            "1b08b0d7849a47c082cb5fdef9572cef",
            "476328dc592344cea67a9f3896aa5846",
            "a4d64fddc53b4c878626a5c0e61a74b5",
            "c21323df97e44d0489dde003a93df4be",
            "b99ccd1a1b904fe7a92e41e6eb42ca2f",
            "6dacafdcd3444641be139401dd4d308f",
            "71c9b23be1d64c5dbe45805009ef4c9b",
            "bd18553e30ba43319f383bfbfabedd34",
            "e02267a4c81a43ef9e32ade86ce9edfb",
            "0e8ccbc5076942beb9ca9c007df141a2",
            "ddb9642fe2d74c2eb1605d31aa480910",
            "6b93d34ecefd45688797b366f3ed1ebb",
            "88ca5ad0d4384d3b9eead27b4045ad93",
            "e439eb19190e41da87abc4daaae6797c",
            "d9ac365ce8964a1f95baa28716aee951",
            "b0f17c04b2e0458e911f6d6dfb1dec6d",
            "e712b244d60743ffa67df4e7e113de78"
          ]
        },
        "outputId": "2eac0a89-c423-4ead-c482-1ad290f8701e"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/433 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ddb2c5db384b4d2e93a11e97059e162b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/436M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "71c9b23be1d64c5dbe45805009ef4c9b"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "model = BertForTokenClassification.from_pretrained(\n",
        "    \"bert-base-cased\",\n",
        "    num_labels=len(tag2idx),\n",
        "    output_attentions = False,\n",
        "    output_hidden_states = False\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "THLkdijDdPq8"
      },
      "outputs": [],
      "source": [
        "# pass model parameter to GPU\n",
        "model.cuda();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZZ7zMnI3dPuT"
      },
      "outputs": [],
      "source": [
        "# setup the optimizer and add the parameters it should update\n",
        "# Add some weight_decay as regularization to the main weight matrices.\n",
        "\n",
        "FULL_FINETUNING = True\n",
        "if FULL_FINETUNING:\n",
        "    param_optimizer = list(model.named_parameters())\n",
        "    no_decay = ['bias', 'gamma', 'beta']\n",
        "    optimizer_grouped_parameters = [\n",
        "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
        "         'weight_decay_rate': 0.01},\n",
        "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
        "         'weight_decay_rate': 0.0}\n",
        "    ]\n",
        "else:\n",
        "    param_optimizer = list(model.classifier.named_parameters())\n",
        "    optimizer_grouped_parameters = [{\"params\": [p for n, p in param_optimizer]}]\n",
        "\n",
        "optimizer = AdamW(\n",
        "    optimizer_grouped_parameters,\n",
        "    lr=3e-5,\n",
        "    eps=1e-8\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "egLkdD4ng5FE"
      },
      "outputs": [],
      "source": [
        "# add a scheduler to linearly reduce the learning rate throughout the epochs.\n",
        "from transformers import get_linear_schedule_with_warmup\n",
        "\n",
        "epochs = 3\n",
        "max_grad_norm = 1.0\n",
        "\n",
        "# Total number of training steps is number of batches * number of epochs.\n",
        "total_steps = len(train_dataloader) * epochs\n",
        "\n",
        "# Create the learning rate scheduler.\n",
        "scheduler = get_linear_schedule_with_warmup(\n",
        "    optimizer,\n",
        "    num_warmup_steps=0,\n",
        "    num_training_steps=total_steps\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9jSiNuPsh38X",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be72080a-b099-43bf-e9fb-4d9087937eaf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting seqeval\n",
            "  Downloading seqeval-1.2.2.tar.gz (43 kB)\n",
            "\u001b[K     |████████████████████████████████| 43 kB 1.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from seqeval) (1.21.6)\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval) (1.0.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (1.2.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (3.1.0)\n",
            "Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval) (1.7.3)\n",
            "Building wheels for collected packages: seqeval\n",
            "  Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16182 sha256=2693e09cd12b241a69160f87078b12ae25f78d0afb936132b7447e8012075532\n",
            "  Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7\n",
            "Successfully built seqeval\n",
            "Installing collected packages: seqeval\n",
            "Successfully installed seqeval-1.2.2\n"
          ]
        }
      ],
      "source": [
        "# define evaluation metrics\n",
        "!pip install seqeval\n",
        "from seqeval.metrics import f1_score, accuracy_score, precision_score, recall_score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uXdkZH4ikGhl"
      },
      "source": [
        "## Finetune Bert Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BAnZVWJ1g5IZ",
        "outputId": "1e8d3df3-3dda-46b9-9c5b-db09ced3b52e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\rEpoch:   0%|          | 0/3 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:155: UserWarning: This overload of add_ is deprecated:\n",
            "\tadd_(Number alpha, Tensor other)\n",
            "Consider using one of the following signatures instead:\n",
            "\tadd_(Tensor other, *, Number alpha) (Triggered internally at  ../torch/csrc/utils/python_arg_parser.cpp:1174.)\n",
            "  exp_avg.mul_(beta1).add_(1.0 - beta1, grad)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average train loss: 0.2906156464207052\n",
            "Validation loss: 0.19834004586393184\n",
            "Validation Accuracy: 0.9403530555588094\n",
            "Validation F1-Score: 0.8254750561305515\n",
            "Precision Score: 0.8214713896457766\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\rEpoch:  33%|███▎      | 1/3 [09:09<18:19, 549.92s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Recall Score: 0.8295179396874313\n",
            "\n",
            "Average train loss: 0.16679781007796826\n",
            "Validation loss: 0.18206528838385236\n",
            "Validation Accuracy: 0.9451908772505242\n",
            "Validation F1-Score: 0.8377568139937842\n",
            "Precision Score: 0.8374113377687359\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\rEpoch:  67%|██████▋   | 2/3 [18:27<09:14, 554.20s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Recall Score: 0.838102575390711\n",
            "\n",
            "Average train loss: 0.1261197534323943\n",
            "Validation loss: 0.1860737373883074\n",
            "Validation Accuracy: 0.9451791633965491\n",
            "Validation F1-Score: 0.8396624472573839\n",
            "Precision Score: 0.8361344537815126\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Epoch: 100%|██████████| 3/3 [27:44<00:00, 554.79s/it]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Recall Score: 0.8432203389830508\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "## Store the average loss after each epoch so we can plot them.\n",
        "loss_values, validation_loss_values = [], []\n",
        "\n",
        "for _ in trange(epochs, desc=\"Epoch\"):\n",
        "    # ========================================\n",
        "    #               Training\n",
        "    # ========================================\n",
        "    # Perform one full pass over the training set.\n",
        "\n",
        "    # Put the model into training mode.\n",
        "    model.train()\n",
        "    # Reset the total loss for this epoch.\n",
        "    total_loss = 0\n",
        "\n",
        "    # Training loop\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "        # add batch to gpu\n",
        "        batch = tuple(t.to(device) for t in batch)\n",
        "        b_input_ids, b_input_mask, b_labels = batch\n",
        "        # Always clear any previously calculated gradients before performing a backward pass.\n",
        "        model.zero_grad()\n",
        "        # forward pass\n",
        "        # This will return the loss (rather than the model output)\n",
        "        # because we have provided the `labels`.\n",
        "        outputs = model(b_input_ids, token_type_ids=None,\n",
        "                        attention_mask=b_input_mask, labels=b_labels)\n",
        "        # get the loss\n",
        "        loss = outputs[0]\n",
        "        # Perform a backward pass to calculate the gradients.\n",
        "        loss.backward()\n",
        "        # track train loss\n",
        "        total_loss += loss.item()\n",
        "        # Clip the norm of the gradient\n",
        "        # This is to help prevent the \"exploding gradients\" problem.\n",
        "        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)\n",
        "        # update parameters\n",
        "        optimizer.step()\n",
        "        # Update the learning rate.\n",
        "        scheduler.step()\n",
        "\n",
        "    # Calculate the average loss over the training data.\n",
        "    avg_train_loss = total_loss / len(train_dataloader)\n",
        "    print(\"Average train loss: {}\".format(avg_train_loss))\n",
        "\n",
        "    # Store the loss value for plotting the learning curve.\n",
        "    loss_values.append(avg_train_loss)\n",
        "\n",
        "\n",
        "    # ========================================\n",
        "    #               Validation\n",
        "    # ========================================\n",
        "    # After the completion of each training epoch, measure our performance on\n",
        "    # our validation set.\n",
        "\n",
        "    # Put the model into evaluation mode\n",
        "    model.eval()\n",
        "    # Reset the validation loss for this epoch.\n",
        "    eval_loss, eval_accuracy = 0, 0\n",
        "    nb_eval_steps, nb_eval_examples = 0, 0\n",
        "    predictions , true_labels = [], []\n",
        "    for batch in valid_dataloader:\n",
        "        batch = tuple(t.to(device) for t in batch)\n",
        "        b_input_ids, b_input_mask, b_labels = batch\n",
        "\n",
        "        # Telling the model not to compute or store gradients,\n",
        "        # saving memory and speeding up validation\n",
        "        with torch.no_grad():\n",
        "            # Forward pass, calculate logit predictions.\n",
        "            # This will return the logits rather than the loss because we have not provided labels.\n",
        "            outputs = model(b_input_ids, token_type_ids=None,\n",
        "                            attention_mask=b_input_mask, labels=b_labels)\n",
        "        # Move logits and labels to CPU\n",
        "        logits = outputs[1].detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "        # Calculate the accuracy for this batch of test sentences.\n",
        "        eval_loss += outputs[0].mean().item()\n",
        "        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])\n",
        "        true_labels.extend(label_ids)\n",
        "\n",
        "    eval_loss = eval_loss / len(valid_dataloader)\n",
        "    validation_loss_values.append(eval_loss)\n",
        "    print(\"Validation loss: {}\".format(eval_loss))\n",
        "    pred_tags = [tag_values[p_i] for p, l in zip(predictions, true_labels)\n",
        "                                 for p_i, l_i in zip(p, l) if tag_values[l_i] != \"PAD\"]\n",
        "    valid_tags = [tag_values[l_i] for l in true_labels\n",
        "                                  for l_i in l if tag_values[l_i] != \"PAD\"]\n",
        "    print(\"Validation Accuracy: {}\".format(accuracy_score(valid_tags, pred_tags)))\n",
        "    print(\"Validation F1-Score: {}\".format(f1_score([valid_tags], [pred_tags])))\n",
        "    print(\"Precision Score: {}\".format(precision_score([valid_tags], [pred_tags])))\n",
        "    print(\"Recall Score: {}\".format(recall_score([valid_tags], [pred_tags])))\n",
        "    print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W87z3w_03tuK"
      },
      "source": [
        "# Visualize the training loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "id": "173H9l8Kk5bt",
        "outputId": "b435d497-7a25-4ae5-b7d1-a2672d6bbcb3"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvoAAAGaCAYAAAB+A+cSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVxV1f7/8dc5zIMIKiIOiBOIA4ooKuKYmilooA1qTqSZdbtWdjNvdsV+VtehrGy4DqhFmiYOhPNsphmKleY8i6mIM5gyCL8//HoSAeUYehDfz8ejR5y119r7sz8cH3zOPmuvbcjJyclBRERERERKFKOlAxARERERkaKnQl9EREREpARSoS8iIiIiUgKp0BcRERERKYFU6IuIiIiIlEAq9EVERERESiAV+iIiUqTatWtHnz59LB2GiMgjT4W+iEgx8PPPP+Pr60t0dLSlQxERkRLC2tIBiIhIybJ8+XJLhyAiIuiKvoiIFCAzM5P09HSzx9na2mJra3sfIiqe0tLSLB2CiEi+VOiLiDxkjh49yr/+9S9CQkKoV68e7dq1Y+zYsfz555+5+h06dIioqCi6dOlCQEAADRo0ICIignnz5uXZ56RJk/D19eXAgQN88MEHtGrVCn9/f3799VcWLFiAr68vP/30E9HR0bRv35569erx+OOPs3Dhwjz7ym+O/s22Q4cO8cILLxAQEEBgYCD//Oc/SUlJybOPvXv3EhkZScOGDWnatCnDhw/n/Pnz+Pr68tZbbxUqTxkZGUydOpVu3brRoEEDAgMDiYiI4JtvvjH1eeutt/D19c13/O3HOnHiBL6+vkyaNImlS5cSERGBv78/Y8aMYfz48fj6+rJ37948+0lNTcXf35+XXnopV/vmzZuJjIykcePG1K9fn7CwML799ttCnZuISGFo6o6IyEPk999/p1+/fri4uPDMM8/g4eHB3r17iYmJ4ZdffiEmJgYbGxsAEhIS2LZtG23atKFy5cpcvXqV5cuXM3LkSM6fP8/gwYPz7P+NN97A3t6eyMhIANzd3fnjjz8AmDhxIteuXeOZZ57B1taWb7/9lrfeegsvLy8CAwPvGntycjJ9+/alffv2vPnmm+zdu5e5c+eSlpbG9OnTTf2OHj1K7969yc7Opk+fPnh4eLBhwwYGDhxY6DxlZGTw/PPPk5CQQEhICF27dsXOzo79+/ezcuVKnnvuuULv63arV68mJiaGnj178uyzz+Ls7IyPjw/Tpk0jLi6O2rVr5+q/bNky0tPTCQ8PN7XNnTuXUaNG0bBhQ1588UUcHBzYvHkzUVFRHD9+nOHDh99zfCIiN6nQFxF5iPz73//G3d2d2NhYnJ2dTe3NmzfnH//4B/Hx8URERADQrVs3evbsmWt8//796devH1OmTCEyMtL0oeAmFxcXZsyYgbX1X38efv31V+BG8RwbG2ualtOpUycee+wxZs2aVahC/9ixY0ycOJHOnTub2oxGI7Nnz+bw4cNUr14duPGBIi0tjdmzZ5v2+9xzz/Hqq6+ya9euQuXpq6++IiEhgcGDB/P666/n2padnV2ofRTk4MGDfP/999SoUSNXe7169YiPj+eNN97AysrK1L5o0SJcXV1p3bo1AGfOnGHMmDF06dKFDz/80NSvd+/ejBkzhpkzZ9KrVy+qVKnyt+IUEdHUHRGRh8S+ffvYt28foaGhZGRkcP78edN/gYGBODo6smnTJlN/R0dH08/p6elcuHCBixcv0qJFC9LS0jh8+HCeY/Tr1y9XkX+rXr165Zp77+HhQbVq1Th69Gih4i9fvnyuIh+gWbNmwI0PAQDXr1/nhx9+wN/fP8+Hh5vfMhRGfHw8pUuX5uWXX86zzWj8e3/6WrdunafIBwgPDyclJSXX7yApKYnt27cTGhpqyt2KFSvIyMigR48euX6H58+fp127dmRnZ7N58+a/FaOICOiKvojIQ+PQoUPAjfn0kyZNyrfP2bNnTT9fuXKFzz77jGXLlnHq1Kk8fS9fvpynzdvbu8Dj53eF2dXV1TS1524KGg9w8eJFAM6fP8+ff/5JtWrV8vTNr60gx44dw8/PDzs7u0KPKayCctSlSxf++9//EhcXR6tWrQCIi4sjJyeHbt26mfrd/D3279+/wGPc+nsUEblXKvRFRB4ykZGRtGzZMt9tLi4upp+HDRvG+vXrefrpp2nSpAmurq5YWVmxYcMGZs6cme8UFnt7+wKP+3evhN86neV2OTk5f2vf98pgMOTbnpWVVeAYBweHfNvd3Nxo3bo1q1evJi0tDWdnZ+Li4qhRowb+/v6mfjfPdezYsZQvXz7ffWnajogUBRX6IiIPiapVqwI3Cu7g4OA79r18+TLr16+nW7duvPvuu7m2FedpIWXKlMHR0ZEjR47k2ZZfW0G8vb05fPgwGRkZd1zqs3Tp0sCNbxRufrsAN6bc3Ivw8HBWr17N8uXLqVatGsePH2fYsGF5YoMbHwzu9nsUEfk7NEdfROQhUadOHXx8fJgzZ06+hWhWVpZpCszNq++3Xyk/c+ZMvstrFhdWVla0bNmSHTt2kJiYmGvbrSvz3E1YWBiXLl3iiy++yLPt1pzcLLpv//AzY8YMM6L+S+vWrXFzcyMuLo64uDiMRmOuaTsATzzxBLa2tkyaNIlr167l2UdqaioZGRn3dHwRkVvpir6ISDHy008/5fuQKjc3N3r27Mm4cePo168fXbt2pXv37tSsWZNr165x7NgxVq1axeuvv05ERATOzs60aNGC77//Hnt7e+rXr88ff/zB3LlzqVy5sukDQXH06quv8uOPPzJw4ECee+45KlSowPr16zl//jxQ8HSbW/Xt25d169bx5ZdfsnPnTkJCQrC1teXgwYMcOXKEmTNnAhAaGsrEiRP5z3/+w+HDh3F1dWXjxo1cuHDhnmK3sbEhNDSUb775ht9//53g4GA8PDxy9alQoQJRUVGMHDmSzp0707VrVypVqsT58+fZv38/q1evZsmSJVSuXPmeYhARuUmFvohIMbJx40Y2btyYp71atWr07NkTPz8/Fi5cyOTJk1m7di1z5szBycmJSpUqER4eTvPmzU1jxo8fz4cffsjatWtZuHAh3t7evPbaa1hbWzNixIgHeVpmqV69OrNmzWLs2LF8/fXX2NnZ0aZNG/7zn//Qvn37Qt1ga2try/Tp05k+fTqLFy/mo48+ws7OjqpVq5qWHwVwdnZmypQpfPDBB0yePBlHR0c6duzI+PHjadKkyT3F/+STTxITE8Off/6Z52r+Td27d8fb25vp06czd+5cUlNTcXV1pVq1agwdOhR3d/d7OraIyK0MOZa6A0pERMQMv//+O927d2fYsGG88MILlg5HRKTY0xx9EREpdm6fu56Tk8O0adMAdAOriEghaeqOiIgUO926daNZs2b4+Phw9epV1q1bx7Zt2+jcuTP16tWzdHgiIg8FTd0REZFiZ9y4caxbt47Tp0+TlZVF5cqVCQsLY9CgQdjY2Fg6PBGRh4IKfRERERGREkhz9EVERERESiAV+iIiIiIiJZBuxr2PLly4Qnb2g50ZVbasM+fOpT3QYz7MlC/zKWfmUb7Mo3yZR/kyj/JlHuXLPJbKl9FowM3NKd9tKvTvo+zsnAde6N88rhSe8mU+5cw8ypd5lC/zKF/mUb7Mo3yZp7jlS1N3RERERERKIBX6IiIiIiIlkAp9EREREZESSIW+iIiIiEgJpEJfRERERKQE0qo7IiIiUqJdvXqFtLRLnDlznezsbEuH89A4c8aofJmhqPNlZWWDs3NpHBzyXzqzMFToi4iISImVmZlBauoFXF3L4eDgwPXrxWv5w+LM2tpIVpYK/cIqynzl5OSQmZnOxYtnsba2wcbG9p72o6k7IiIiUmKlpl7E2bk0trb2GAwGS4cjUigGgwFbW3ucnEqTlnbxnvejQl9ERERKrKysDOzsHCwdhsg9sbd3IDMz457Ha+pOCfHTrtMs2HCI85fTKeNiR0TrGjSvW8HSYYmIiFhUdvZ1jEYrS4chck+MRiuys6/f83gV+iXAT7tO89WyvWT837ywc5fT+WrZXgAV+yIi8sjTlB15WP3d966m7pQACzYcMhX5N2VkZbNgwyELRSQiIiIilqZCvwQ4dzndrHYRERGROwkJaczUqf+757HR0ZOLOKK72759GyEhjdm+fdsDP3ZxpUK/BCjrYpdvu421kbSrmQ84GhEREbnffv99J9HRk0lNTbV0KFKMqdAvASJa18DWOvev0spoICsrm6gZCRw4ce/LMomIiEjxs3v3TmbMmEpa2v0p9Nes2cSAAQPveWy/fs8XcURyL1TolwDN61ag3xO1Ketih4EbV/gju/gxsl9jrIwGxs76hfhNR8jO1kNCREREHjXXr18nI8O8JRrt7Oywtr63NVv+zlgpWvotlBDN61aged0KuLuXIiXlr0/3UQOC+HrFPhZuPMKeYxcYFFYXt1L5T/URERGR4i86ejIzZkwF4Kmnupra5837Hk/PioSENOapp3ri4+NLTMwM/vjjBBMnfk6jRo2ZPTuGH35Yx/Hjx7h27Rre3tXo06c/bdu2z3WMkJDGPP/8CwwY8EKuY373XRzR0ZP58ccNALRu3Y7XXx+Ovb19rrEDBgzi+ecHmz02Pf0aX345iVWrlpORkUmjRoG88cYIwsM759qnOdasWck338zk2LGjODo60aJFS4YM+Seurq6mPklJx/nf/yaxc+cO0tJSKV3aFX//BvzrX2/j7OwMwOrVK5g9O4akpOMYDAYqVKhAaOiTPP10T7NjelBU6JdwDnbWvBBWhzrebsxatZ9R0xMYGOqHf41ylg5NRETkoXTz2TXnLqdT1gLPrmnduh0nT55gxYpl/POfr1O69I2C1dXVzdRn69YtrF27kvDwpyhVqhTlyt34ux8bO4cWLVrRoUMnsrIyWb16Je+88xbjxn1McHDIXY89cuSbVKxYmRdffIX9+/cSH78IV1c3Xnrpn0Uy9r33RrN27SqeeCIUP7+6/Prrdv71r1fNTZHJ0qXxvP/+aOrWrc+QIf/kzJlk5s+fy549u5g69Wvs7OzIzMzk9ddfwcrKyDPP9KJ06dIkJyezefOPpKWl4uzszNatW4iKepvWrdvStWs4169f5+jRI+zc+ZsKfbEsg8FAS/+K1KhYmv/F7eLjeTvo2KQKPdrUwNpKs7dEREQKqzg8u6ZmzVr4+vqxYsUyWrZsg6dnxTx9kpKOExPzHV5eVXO1f/vtfOzs/rqC3r37M0RG9mbu3FmFKvRr167Dm2++bXp96dIlliyJK1Shf7ex+/btZe3aVfTs2YeXXx4KQETEU7z//mgOHtx/1/3fLisriy+/nETNmj5MmjQZW1tbAHx9axMV9Tbx8Qvp0eNZjh49zKlTfzB16lf4+dU1jb/124PNmzdRrVp13ntvvNlxWJIK/UdIxXJOjOwbyNx1B1m5NYn9SRcZ3K0uHm6Olg5NRETkgdq08xQ/7jhl9rhDJy+RdT33PW8ZWdnMWLqHH349afb+Qvw9aVHf0+xxd9OoUeM8RT6Qq8i/fPky2dnZ+PsHsHr1ikLt98knu+d63aBBQ374YR1XrqTh5OT8t8b+/PNmAMLDe+Tq1737MyxdGl+o+G61d+9uLlw4z6BBQ0xFPkC7dh34/PNP2Lx5Ez16PGuKe9OmjdSs6YONjU2efTk7O3PmTDK7dv1O3br1zI7FUlToP2Jsbazo09GXOlXLMGPpHkbP2ErfTr40q6Mn6IqIiNzN7UX+3dotJb+r/HCjmP3qq2gOHtyf6wbdwj6B1cMjd71QqpQLAKmpqXct9O829vTpU1hZWVGhQu4PPpUrVy5UbLc7ffrGB7nbP/AYjUYqV65CcvKN7RUrVuKZZ3ozc+Y05s6dTUBAI4KDW9KxYyccHZ2AG98srFu3msGD++PpWYkmTYJo27Y9TZo0vafYHhQV+o+oQF93qlZwZsr3u5ny/W52H7lA7w4+2NlaWTo0ERGR+65F/Xu7kv6vLzbl+0DKsi52DO/dqChCKxK3Xrm/6bfffuGtt16nQYMAXn99OGXLlsPa2pqlS+NZtWp5ofZrNOZfJ+Tk3P2Dzt8Ze7+98sprdOkSxsaNG0hI2MJHH43l66+nM3nyDNzdy+PmVoYZM2aTkLCFLVs2s2XLZr7/fiFdunRlxIj/WDr8AmmC9iOsXGkHhvcOIDS4Kpt2nuLdr7ZyPFkP3hARESlIfs+usbU2EtG6xgOOpHBX4G+1fv1abG1t+eijzwgN7Ubz5i2K1RXpChU8uX79uulK/E0nTpy45/0BHD9+LFd7Tk4OJ04k4eGR+4Ne9eo16dfveT7/fCpffjmdM2eSWbRovmm7jY0NLVq0ZNiw4Xz33SIiIp5iyZLv+eOPe4vvQVCh/4izMhqJaFWDN55tyJ/pWYz5OpG1208Ui0/XIiIixc2tz66BG1fy+z1R+4GuugPg4OAAYNYDs4xGIwaDgezsbFPbqVMn2bhxfVGHd0+CgpoDsHBhbK72+fPn3tP+ateug5tbGRYtiiUzM9PUvm7dGlJSzhAc3AKAK1fSyMrKyjW2evUaWFlZmaY3XbqU++GjBoOBGjVqAZCenvcbnuJCU3cEAD/vMoyODCJ68R6+Wbmf3Ucv0P+J2jg75L0hRURE5FF289k1luTrWxuAKVO+4LHHOmJtbU2LFq1MHwDyExwcwty5sxg27BU6dHicCxcusGDBPCpVqsKhQwceVOgFql3bjzZt2vHttzFcvHjBtLxmUtKNK/KFvY/gJmtra4YMeYX33x/NK68Mpn37jpw5k0xs7FyqV69BWFg4AImJ25g4cRxt2jyGl1dVsrOvs2LFMgwGA61btwPgv/8dQ2rqZRo1akz58uVJTr6xn1q1fPD2rla0iShCKvTFxMXRlqFP+bNqaxKx6w8RNSOBF8Lq4lPF9e6DRURE5IHx8anN4MEvs2DBPH7++Seys7OZN+/7Oxb6gYFNeOutd/jmm6/49NOP8PSsyJAhr3Dq1MliUegDjBz5LmXKlGX16pWsX7+Wxo2DGD36A3r16p5r5ZzC6tw5DFtbW2bN+orPP/8EJycnOnToxIsvvoKd3Y1vZWrWrEVQUDM2b95IXNwC7O3tqVmzFhMmfEq9evUBePzxJ/j++4UsXBhLWloqZcqUpV279kRGvoDRWHwnyBhyNEfjvjl3Lo3s7Aeb3tufjHuvjpy6zP/ifufspWs8GVKNLs29MRrNnw9Y3BVVvh4lypl5lC/zKF/mUb7u7vTpY1SocGPVFWtrI1lZ2XcZITcVl3wdOLCPAQN685///D86dnzC0uEU6H7l69b3cH6MRgNly+a/4lHx/QgiFlXN04WoAUEE+XmwcOMRJsz5hQupxXcOmoiIiDz80tOv5Wn77rtvMRqNNGgQYIGIHm6auiMFcrCz5oWwOtTxdmPWqv2Mmp7AwFA//GuUs3RoIiIiUgLFxMzk4MH9NGrUGIPByM8/31jKsmvX8Dzr8MvdqdCXOzIYDLT0r0iNiqX5X9wuPp63g45NqtCjTQ2srfSFkIiIiBSdevX8SUxMYMaMaVy9+iceHhV4/vnB9OkzwNKhPZRU6EuhVCznxMi+gcxdd5CVW5PYn3SRwd3q4uHmaOnQREREpIRo1iyYZs2CLR1GiaFLslJotjZW9Onoy8vh9Tlz4SqjZ2xly+7Tlg5LRERERPKhQl/MFujrzujIICqXd2bK97uZvmQP6RnXLR2WiIiIiNxChb7ck7Kl7RneK4DQYG827TzFu19t5XiylngTERERKS4sOkc/IyODTz75hLi4OC5fvkzt2rV57bXXaN68+R3HrVy5kqVLl7Jjxw7OnTuHp6cnbdu25aWXXqJUqVKmfgsWLGDEiBEF7mf8+PF07doVgEmTJvHZZ5/l6VOuXDk2bdp0j2dYslkZjUS0qo6flytTFu9mzNeJPNOuJu0aVTL76XUiIiIiUrQsWui/9dZbrFy5kr59+1K1alUWLlzIoEGDiImJISCg4LVS33nnHcqXL0+3bt2oWLEi+/btIyYmho0bNzJ//nzTk86aNGnCuHHj8oz/6quv2Lt3b74fKN59913s7e1Nr2/9WfLn512G0ZFBRC/ew6xV+9l99DwDOvvh7GBj6dBEREREHlkWK/R37NjBkiVLGDFiBP379wfgySefJDQ0lAkTJjBr1qwCx3766ac0bdo0V1u9evUYPnw4S5YsISIiAoAqVapQpUqVXP2uXbvG6NGjadasGe7u7nn2/cQTT+Di4vI3z+7R4+Joy9Cn/Fm1NYnY9YeImpHAC2F18aniaunQRERERB5JFpujv3z5cmxsbHjqqadMbXZ2dvTo0YPExETOnDlT4Njbi3yA9u3bA3Do0KE7Hnft2rVcuXKFsLCwfLfn5OSQlpZGTk5OYU5DbmE0GHg8yIt/9wnEymhg7OztxG86Qna2cikiIiLyoFms0N+zZw/VqlXDyckpV7u/vz85OTns2bPHrP2dPXsWADc3tzv2i4+Px97eng4dOuS7vU2bNgQGBhIYGMiIESO4ePGiWXEIVPN0IWpAEEF+HizceIQJc37hQmq6pcMSERGRArz3XhQ9evx1EfTUqZM0a9aIpUvjzR5bFKKjJxMS0rhI91kY27dvIySkMdu3b3vgx74fLDZ1JyUlBQ8PjzztN6fT3OmKfn6mTp2KlZUVHTt2LLDPxYsX2bhxI+3bt8fZ2TnXNhcXF/r06UODBg2wsbFhy5YtzJ07l927dzNv3jxsbW3NiudR52BnzQthdajj7casVfsZNT2BgaF++NcoZ+nQREREpJiIiZlJ1aretGrVxtKhlEgWK/SvXbuGjU3emzVv3kibnl74K8Dx8fHExsYyePBgvLy8Cuy3YsUKMjMz8522069fv1yvO3XqRK1atXj33XdZtGgRTz/9dKHjualsWee7d7oP3N1L3b3TAxLxmAtN6lVkXMw2Pp63gydb16Bv5zrYWBeflV2LU74eFsqZeZQv8yhf5lG+7uzMGSPWt/zNsS5Gf3+Kk5ur5d3MT+XKldiw4Sesra2xsrpzzm4fa45Zs2bSqlVb2rVrl6v9+ecH0b9/5AP/fd08Vysr4z0d+37EazQa7/nfucUKfXt7ezIzM/O03yzwbxb8d7Nt2zbefvtt2rRpw9ChQ+/YNz4+HldXV1q1alWofffs2ZPx48fz008/3VOhf+5c2gOfn+7uXoqUlOK1nr29EUb0DmDu2oMs2nCIX/edYXC3uni4OVo6tGKZr+JOOTOP8mUe5cs8ytfdZWdnk5WVDdwowm7+LLndvDfx1vzY2dmRlZV915zlN9bcY+cda8TK6sH/vq5fzzb939xj36/3V3Z29h3/nRuNhgIvLlvsY627u3u+03NSUlIAKF++/F33sXfvXoYMGYKvry8TJ07EysqqwL4nT55k27ZtPP744/l+k5Afo9GIh4cHly5dKlR/KZiNtRXPdfTl5fD6pFy8StSMrWzZddrSYYmIiDx01q5dTUhIY3bs+DXPtm++mUnLlk1ITr7xN/a3335h5MjhRER0oW3b5kREdOHTTz8kPf3aHY9R0Bz9H35YT58+T9OuXTB9+jzNhg3r8h0/e3YML74YSefOj9GuXQsiI59j3brVufqEhDQmLS2NZcsWExLSmJCQxrz3XhSQ/xz9rKwspk+fwlNPdaNt2+Y8/XQ3Zs6cxvXr1/Ps95NPPmT9+jU899zTtG3bnOeee5otWzbf8ZzvZM2alQwY0It27YIJDe3ABx+8m+c+zuPHj/P22/+ia9fHadcumPDwzowaNYK0tDRTn9WrVxAZ+RwdOrSiY8fW9O37DN999+09x3U3FruiX7t2bWJiYrhy5UquG3J/++030/Y7OX78OAMHDqRMmTJMnjwZR8c7Xx1evHgxOTk5pgdkFUZmZianTp2iXr16hR4jdxbo6453hVJMjt/FlPjd7D56gd4dfLCzLfhDmoiISHFyectmzi6YT9b5c1iXKUu5iO64NAt+YMcPDg7BwcGBtWtX4+/fMNe2tWtXU69efTw8KgCwbt1q0tOvER7eAxeX0uzZs4v587/jzJkzjBkz1qzjJiRsYeTIN6lWrTqDB7/MpUuX+OCD0bi75704Gxs7hxYtWtGhQyeysjJZvXol77zzFuPGfUxwcAgA77zzLuPHv4+vrx9du4YDUKlS5QKPP3bsGJYtW8xjj3XE3783O3b8wrRp/yM5+TTDh4/M1ffXXxNZt2414eE9cHBwJDZ2DiNHvsn8+YspXdq8pb+XLo3n/fdHU7dufYYM+SdnziQzf/5c9uzZxdSpX2NnZ0dmZiavvvoyRqORZ57pRenSpUlOTmbz5h9JS0vF2dmZrVu3EBX1Nq1bt6Vr13CuX7/O0aNH2LnzN55+uqdZMRWWxQr9Tp06MX36dObNm2daRz8jI4MFCxbQqFEj0426J0+e5OrVq9SoUcM0NiUlhcjISAwGA9HR0ZQpU+aux1u8eDEVK1YkMDAw3+3nz5/Ps5/o6GjS09Np2bLlPZ6l5KdsaXuG9wog7sejLNl8lIN/XOLFbnXx8tA8UxERKd4ub9lM8tczycnIACDr/DmSv54J8MCKfXt7e4KDQ1i/fg1Dhw4zzZH/448T7N+/l6FD3zD1HTLkFezs/nr4Z7duEVSqVIUpUz7n9OnTVKhQodDH/fLLTylXzp0vv4zG0fHGRdqAgEa89to/qFDBM1ffb7+dn+u43bs/Q2Rkb+bOnWUq9B9/vDMTJ46jYsVKPP545zse+8CB/Sxbtpgnn+zOG2+M+L99Po2zcyni4hbQvfsz1KxZy9T/2LGjfPPNPCpWrARAo0aN6d+/J6tXr6B792cKfc5ZWVl8+eUkatb0YdKkyabFWXx9axMV9Tbx8Qvp0eNZjh49zMmTfzB16lf4+dU1jX/++cGmnzdv3kS1atV5773xhT7+32WxQr9BgwZ06tSJCRMmkJKSgpeXFwsXLuTkyZN88MEHpn7Dhw8nISGBffv2mdoGDhxIUlISAwcOJDExkcTERNM2Ly+vPE/V3b9/P/v27eOFF14w/WO4Xdu2bencuTM+Pj7Y2try888/s2LFCgIDAwkNDS3isxcroyoKk/EAACAASURBVJGIVtXx83JlyuLdjPk6kWfa1aRdo0oF/o5ERESKyuXNm7j04w9mj7t2+BA5WVm52nIyMkieOZ1LP2wwe3+lQ1rhEtzC7HHt2nVgzZpV7NjxKw0a3Kh71q5dhdFopG3b9qZ+txbbV69eJT09nfr1byxlfuDA3kIX+mfPnuXAgf306/e8qcgHaNKkGd7e1bl27Wqu/rce9/Lly2RnZ+PvH8Dq1SvMPleALVs2AfDMM71ztT/zTC/i4hbw00+bchX6QUHNTEU+QM2atXBycuLkyT/MOu7evbu5cOE8gwYNybUCY7t2Hfj880/YvHkTPXo8i5PTjTnymzZtpGZNn3yniTs7O3PmTDK7dv1O3boPZraIxQp9gHHjxvHxxx8TFxfHpUuX8PX1ZcqUKQVedb9p7969AEybNi3PtvDw8DyFfnz8jflldyrYw8LC2L59O8uXLyczM5NKlSrx0ksvMXjwYKytLZqmEs3PuwyjI4OIXryHWav2s/voeQZ09sPZoXD3UYiIiDxItxf5d2u/X5o1a4GjoxNr1qzMVej7+zekXLm/lrI+ffo00dH/48cffyA19XKufdw6d/xuTp8+BUDlylXybPPyqsr+/XtztW3atJGvvorm4MH9ZPzftx/APV/MO336FFZWVnmm9lSqVAUrKyuSk0/lar85delWpUq5kJpq3s3rN8/by6tqrnaj0UjlylVMx61YsRI9ez7HzJnTmDt3NgEBjQgObknHjp1MH4wiIp5i3brVDB7cH0/PSjRpEkTbtu1p0iTvg2CLikUrWDs7O4YPH87w4cML7BMTE5On7dar+4UxbNgwhg0bdsc+Y8aMMWufUnRcHG0Z+pQ/q7YmEbv+EFEzEnghrC4+VcybQyciIlJYLsEt7ulK+uE3h5F1/lyedusyZany5oiiCK1Q7OzsaNGiJRs2rOXVV//FyZN/cODAfl5//a+a6vr167z22kukpl6md+++VK3qjb29A2fPpvDee1Gm1XKK2m+//cJbb71OgwYBvP76cMqWLYe1tTVLl8azatXy+3LM2xmN+d/7d7/OGWDo0Nd54olQNm7cQELCFj76aCxffz2dyZNn4O5eHje3MsyYMZuEhC1s2bKZLVs28/33C+nSpSsjRvznvsSkxWSlWDAaDDwe5MW/+wRiZTQwdvZ24jcdeeDLk4qIiNxJuYjuGG57iKbB1pZyEd0feCzt2rXn3Llz/PrrdtauXYWVlRVt2z5m2n748EGSko7z8suv8txz/WnZsg1NmjTNdcW/sG7OwT9xIinPtuPHj+V6vX79Wmxtbfnoo88IDe1G8+Yt7nDVunBX+CtU8OT69ev88ceJXO1//HGC69ev4+HhWcDIv+fmed9+jjk5OZw4kZTnuNWr16Rfv+f5/POpfPnldM6cSWbRovmm7TY2NrRo0ZJhw4bz3XeLiIh4iiVLvs9zXkVFhb4UK9U8XYgaEERTPw8WbjzChDm/cCG18A9PExERuZ9cmgXj0bc/1mXKAjeu5Hv07f9AV925qWnTYJycnFi7dhVr166mYcNA3Nz+Wlgkv6vaOTk5zJs3x+xjlStXjlq1fFi2bDF//nnF1L516xaOHj2cq6/RaMRgMJCd/dea8qdOnWTjxvV59uvg4EBa2t2n0zRrduPbl9uXorx5Ljdv8C1qtWvXwc2tDIsWxeZ6/tO6dWtISTlD8P99K3TlShpZt03fql69BlZWVqapS5cu5V6O02AwUKPGjfsKzHlQrDk0+VyKHQc7awaF1aGOdxm+WbWPUdMTeL6LHw1qmn8FQkREpKi5NAu2SGF/O1tbW0JCWrNixTKuXv2TN998O9f2qlW9qVSpMp9//jEpKWdwcnJi/fq1Zs9Tv2nw4H/w5puvMmTI83TuHMbly5eZP38u1apV5+rVv27GDQ4OYe7cWQwb9godOjzOhQsXWLBgHpUqVeHQoQO59unrW5tt2xKYM+cbypVzx9OzUr43qtaq5cMTT4SycOE8UlMv4+/fkB07fmX16hWEhnajRo2a93ROd2Ntbc2QIa/w/vujeeWVwbRv35EzZ5KJjZ1L9eo1CAu7sSxoYuI2Jk4cR5s2j+HlVZXs7OusWLEMg8FA69Y3nvr73/+OITX1Mo0aNaZ8+fIkJ9/YT61aPnh7V7s/8d+XvYr8TQaDgRB/T2pUcuHLRbv4JHYHHZtUoUebGljf5VHcIiIij4rHHuvIihVLsbKyonXrtrm2WVtbM3bsRD7+eDwxMTOxs7OlZcu2dO/+NP37m79ue7Nmwfy///dfpk79ksmTP6dixcqMGDGKH3/cwC+//LUCYmBgE9566x2++eYrPv30Izw9KzJkyCucOnUyT6H/8suvMnbsGKZO/ZL09HSeeCK0wBVphg8fiadnRZYtW8z69Wtwdy/PwIEv0qfPALPPxRydO4dha2vLrFlf8fnnn+Dk5ESHDp148cVXsLOzA26s6tOsWXM2b95IXNwC7O3tqVmzFhMmfEq9evUBePzxJ/j++4UsXBhLWloqZcqUpV279kRGvoDReH9qG0PO/bwr4RF37lzaA59jXhIfh56ZdZ25aw+ydvsfVK1Qihe71cXD7c4PSCuskpiv+005M4/yZR7lyzzK192dPn2MChVurJhibW0kKyv7LiPkJuXLPPcrX7e+h/NjNBooW9Y5/21FHo1IEbOxtuK5jr68HF6fsxevEjVjK1t2nbZ0WCIiIiLFmgp9eWgE+roTNSCIKuWdmRK/m+lL9pCecd3SYYmIiIgUSyr05aFStrQ9w3sFEBrszaadpxg9cyvHk/W1tYiIiMjtVOjLQ8fKaCSiVXXeeLYhVzOyGPN1ImsST9zXh2CIiIiIPGxU6MtDy8+7DKMjg/Cr6sasVfv5bMFO0q5m3n2giIiIyCNAhb481FwcbRn6lD/PtKvJjkPniJqRwP6ki3cfKCIiIlLCqdCXh57RYODxIC/+3ScQa6ORsbO3E7/pyANf2lRERESkOFGhLyVGNU8XRg1oQlM/DxZuPMKEOb9wIfX+PFJaREQeHrqHSx5Wf/e9q0JfShQHO2sGhdUhsrMfh09dZtT0BH47eNbSYYmIiIVYWVmTmZlh6TBE7klmZgZWVtb3PF6FvpQ4BoOBEH9PRvVvgquzHZ/E7mDOmgNkXdfT/UREHjXOzq5cvJhCRka6ruzLQyMnJ4eMjHQuXkzB2dn1nvdz7x8RRIo5z7JOvNMvkLlrD7JyaxL7ki7yYre6eLg5Wjo0ERF5QBwcnAC4dOksFy9mk52tiz6FZTQalS8zFHW+rKysKVXKzfQevhcq9KVEs7G24rmOvvhVLcPMZXuImrGVfo/70qxuBUuHJiIiD4iDgxMODk64u5ciJUUPWSws5cs8xTFfmrojj4RAX3eiBgRRpbwzU+J3M33JHtIzrls6LBEREZH7RoW+PDLKlrZneK8AQoO92bTzFKNnbuXIyUuWDktERETkvlChL48UK6ORiFbVeePZhlzNyGLYJz+wJvGEbtASERGREkeFvjyS/LzLMDoyiAa13Jm1aj+fLdhJ2tVMS4clIiIiUmRU6Msjy8XRlncim/Jsu5rsOHSOqBkJ7E+6aOmwRERERIqECn15pBmNBjoGefHvPoFYG42Mnb2d7zcdITtbU3lERETk4aZCXwSo5unCqAFNaOrnwaKNR5gw5xcupKZbOiwRERGRe6ZCX+T/ONhZMyisDpGd/Th86jKjpifw28Gzlg5LRERE5J6o0Be5hcFgIMTfk1H9m+DqbMcnsTuYs+YAWdf1ZEARERF5uKjQF8mHZ1kn3ukXSLtGlVi5NYn3YhJJvvCnpcMSERERKTQV+iIFsLG24rmOvrwcXp+zF68SNWMrW3adtnRYIiIiIoWiQl/kLgJ93YkaEESV8s5Mid/N9CV7SM+4bumwRERERO5Ihb5IIZQtbc/wXgGEBXuzaecpRs/cyvHkVEuHJSIiIlIgFfoihWRlNBLeqjpv9AzgakYWY75OZE3iCXJytOa+iIiIFD8WLfQzMjIYP348ISEh+Pv78/TTT/PTTz/dddzKlSt59dVXadeuHQ0aNKBTp06MHTuW1NS8V1h9fX3z/e/bb7/N0zc5OZmhQ4fSuHFjGjVqxEsvvURSUlKRnKuUHH5V3RgdGUQdbzdmrdrPZwt2knY109JhiYiIiORiyLHg5cjXX3+dlStX0rdvX6pWrcrChQv5/fffiYmJISAgoMBxTZs2pXz58rRv356KFSuyb98+5syZg7e3N/Pnz8fOzs7U19fXl5CQELp27ZprHw0aNMDb29v0+sqVK0RERHDlyhX69++PtbU1M2fOxGAwsGjRIkqXLm32+Z07l/bAn7Dq7l6KlBRNKSmsv5Ov7JwcVm9NYt76Q5R2tuWFsLr4VHEt4giLH73HzKN8mUf5Mo/yZR7lyzzKl3kslS+j0UDZss75brN+wLGY7NixgyVLljBixAj69+8PwJNPPkloaCgTJkxg1qxZBY799NNPadq0aa62evXqMXz4cJYsWUJERESubdWrV6dbt253jGf27NkcO3aMBQsWUKdOHQBatmxJWFgYM2fOZOjQofdwllKSGQ0GOgZ5UauKK5PjdjF29na6hVQjtLk3RqPB0uGJiIjII85iU3eWL1+OjY0NTz31lKnNzs6OHj16kJiYyJkzZwoce3uRD9C+fXsADh06lO+Ya9eukZ6eXuA+V6xYQcOGDU1FPkCNGjVo3rw5y5Ytu+v5yKOrmqcLowY0oamfB4s2HmHCnF+4kFrwe01ERETkQbBYob9nzx6qVauGk5NTrnZ/f39ycnLYs2ePWfs7e/YsAG5ubnm2xcbG0rBhQ/z9/QkLC2PVqlW5tmdnZ7Nv3z7q1auXZ2z9+vU5evQoV69eNSseebQ42FkzKKwOkZ39OHzqMqOmJ/DbwbOWDktEREQeYRYr9FNSUihfvnyednd3d4A7XtHPz9SpU7GysqJjx4652gMCAnjttdf44osv+M9//kNGRgb/+Mc/WLx4sanPxYsXycjIMB379nhycnJISUkxKx559BgMBkL8PRnVvwmuznZ8EruDOWsOkHU929KhiYiIyCPIYnP0r127ho2NTZ72mzfS3mmaze3i4+OJjY1l8ODBeHl55do2Z86cXK/Dw8MJDQ1l/PjxdOnSBYPBYDqWra1tgfFcu3at0PHcVNCNEfebu3spixz3YVXU+XJ3L8Unw9yZHr+LJZuOcPjUZf7VpzEVy1nm/XA/6D1mHuXLPMqXeZQv8yhf5lG+zFPc8mWxQt/e3p7MzLxLEt4sum9dOedOtm3bxttvv02bNm0KdcOso6Mjzz77LB9++CGHDx+mRo0apmNlZGQUGI+9vX2h4rmVVt0p/u5nvrq3rEY1D2dmLN3DPz9cT9/HfWlet8J9OdaDpPeYeZQv8yhf5lG+zKN8mUf5Mk9xXHXHYlN33N3d852ec3OKTH7Tem63d+9ehgwZgq+vLxMnTsTKyqpQx/b09ATg0qVLALi6umJra5vv9JyUlBQMBkO+03pE7qaRjztRA4LwKu/M1PjdRC/ZzbWMLEuHJSIiIo8AixX6tWvX5siRI1y5ciVX+2+//WbafifHjx9n4MCBlClThsmTJ+Po6FjoY998CFaZMmUAMBqN+Pj48Pvvv+fpu2PHDqpWrYqDg0Oh9y9yq7Kl7XmzVwBhwd5s3nmad2du43iyrpCIiIjI/WWxQr9Tp05kZmYyb948U1tGRgYLFiygUaNGeHh4AHDy5Mk8S2ampKQQGRmJwWAgOjraVLDf7vz583naLly4wOzZs6lcuXKuB2Y9/vjj/Prrr+zevdvUdvjwYbZs2UKnTp3+zqmKYGU0Et6qOm/0DOBqRhZjvk5kTeIJLPi8OhERESnhLDZHv0GDBnTq1IkJEyaQkpKCl5cXCxcu5OTJk3zwwQemfsOHDychIYF9+/aZ2gYOHEhSUhIDBw4kMTGRxMRE0zYvLy/TU3VnzZrFmjVraNOmDRUrViQ5OZm5c+dy/vx5Pv/881zx9OrVi3nz5vHCCy8wYMAArKysmDlzJu7u7qYHeon8XX5V3RgdGcT0JXuYtWo/u4+eZ0BnP5wd8t6YLiIiIvJ3WKzQBxg3bhwff/wxcXFxXLp0CV9fX6ZMmUJgYOAdx+3duxeAadOm5dkWHh5uKvQDAgLYvn078+bN49KlSzg6OtKwYUMGDx6c5xjOzs7ExMTw/vvv88UXX5CdnU3Tpk15++23812bX+ReuTja8s8e/qzemsS89YeImpHAC2F18aniaunQREREpAQx5GjuwH2jVXeKP0vn68ipy0yO20XKpat0C6lGaHNvjEaDxeIpDEvn7GGjfJlH+TKP8mUe5cs8ypd5tOqOiORSzdOFUQOa0NTPg0UbjzBhzi9cSC38MyRERERECqJCX8TCHOysGRRWh8jOfhw+dZlR0xP47eBZS4clIiIiDzkV+iLFgMFgIMTfk1H9m+BWyo5PYncwZ80BMrOyLR2aiIiIPKRU6IsUI55lnRjZN5DHGlVm5dYk3v8mkeQLf1o6LBEREXkIqdAXKWZsrK3o3dGHf0TU5+zFq0TN2MpPu05bOiwRERF5yKjQFymmGvm4EzUgCK/yzkyN3030kt1cy8iydFgiIiLykFChL1KMlS1tz5u9AggL9mbzztO8O3Mbx5O11JmIiIjcnQp9kWLOymgkvFV13ugZwNWMLMZ8nciaxBPoERgiIiJyJyr0RR4SflXdGB0ZRB1vN2at2s9nC3aSdjXT0mGJiIhIMaVCX+Qh4uJoyz97+PNsu5rsOHSOqBkJ7E+6aOmwREREpBhSoS/ykDEaDHQM8uLffQKxNhoZO3s73286Qna2pvKIiIjIX1Toizykqnm6MGpAE5r6ebBo4xEmzPmFC6nplg5LREREigkV+iIPMQc7awaF1SGysx+HT11m1PQEfjt41tJhiYiISDGgQl/kIWcwGAjx92RU/ya4lbLjk9gdzFlzgMysbEuHJiIiIhakQl+khPAs68TIvoE81qgyK7cm8f43iSRf+NPSYYmIiIiFqNAXKUFsrK3o3dGHf0TU5+zFq0TN2MpPu05bOiwRERGxABX6IiVQIx93ogYE4VXemanxu4lesptrGVmWDktEREQeIBX6IiVU2dL2vNkrgLBgbzbvPM27M7dxPDnV0mGJiIjIA6JCX6QEszIaCW9VnTd6BnA1I4sxXyeyJvEEOTlac19ERKSkU6Ev8gjwq+rG6Mgg6ni7MWvVfj5bsJO0q5mWDktERETuIxX6Io8IF0dbhvbw59l2Ndlx6BxRMxLYn3TR0mGJiIjIfaJCX+QRYjAY6Bjkxdt9A7E2Ghk7ezvfbzpCdram8oiIiJQ0KvRFHkHeFVwYNaAJTf08WLTxCBPm/MKF1HRLhyUiIiJFSIW+yCPKwc6aQWF1iOzsx+FTlxk1PYHfDp61dFgiIiJSRFToizzCDAYDIf6ejOrfBLdSdnwSu4M5aw6QmZVt6dBERETkb1KhLyJ4lnViZN9AHmtUmZVbk3j/m0SSL/xp6bBERETkb1ChLyIA2Fhb0bujD/+IqM/Zi1eJmrGVn3adtnRYIiIico9U6ItILo183IkaEIRXeWemxu8meslurmVkWTosERERMZMKfRHJo2xpe97sFUBYsDebd57m3ZnbOJ6caumwRERExAwq9EUkX1ZGI+GtqvNGzwCuZWQx5uttrEk8QU6O1twXERF5GKjQF5E78qvqRlRkEHW8yzBr1X7em5FA2tVMS4clIiIid2FtyYNnZGTwySefEBcXx+XLl6lduzavvfYazZs3v+O4lStXsnTpUnbs2MG5c+fw9PSkbdu2vPTSS5QqVcrU79SpU8TGxrJhwwaOHTuG0WjEx8eHl156Kc8xJk2axGeffZbnWOXKlWPTpk1Fc8IiDykXR1uG9vBn1dYkYjccYv/xCwzuWhefKq6WDk1EREQKYNFC/6233mLlypX07duXqlWrsnDhQgYNGkRMTAwBAQEFjnvnnXcoX7483bp1o2LFiuzbt4+YmBg2btzI/PnzsbOzA2DNmjVMmzaN9u3bEx4eTlZWFnFxcfTv35+xY8fy5JNP5tn3u+++i729ven1rT+LPMoMBgMdg7xo6l+JD2YmMHb2drqFVCO0uTdGo8HS4YmIiMhtLFbo79ixgyVLljBixAj69+8PwJNPPkloaCgTJkxg1qxZBY799NNPadq0aa62evXqMXz4cJYsWUJERAQATZs2Zd26dZQpU8bUr2fPnnTr1o1PP/0030L/iSeewMXFpQjOUKRkqlnFlVEDmhCzYh+LNh5h77ELDAqri1spO0uHJiIiIrew2Bz95cuXY2Njw1NPPWVqs7Ozo0ePHiQmJnLmzJkCx95e5AO0b98egEOHDpnaatWqlavIB7C1taV169b88ccfXLt2Lc9+cnJySEtL0w2HInfgYGfNoLA6RHb24/Cpy4yansBvB89aOiwRERG5hcUK/T179lCtWjWcnJxytfv7+5OTk8OePXvM2t/ZszeKDDc3t7v2TUlJwdHR0TTF51Zt2rQhMDCQwMBARowYwcWLF82KQ+RRYTAYCPH3ZFT/JriVsuOT2B3MWXOAzKxsS4cmIiIiWHDqTkpKCh4eHnna3d3dAe54RT8/U6dOxcrKio4dO96x37Fjx1i1ahVdunTBYPhrXrGLiwt9+vShQYMG2NjYsGXLFubOncvu3buZN28etra2ZsUj8qjwLOvEyL6BfLf2ECu3JrEv6SIvdquLh5ujpUMTERF5pFms0L927Ro2NjZ52m9eZU9PTy/0vuLj44mNjWXw4MF4eXkV2O/q1asMHToUBwcHXnvttVzb+vXrl+t1p06dqFWrFu+++y6LFi3i6aefLnQ8N5Ut62z2mKLg7l7q7p3ERPkyX345e7V3IE39K/Lp3F94d+ZWXuregDaBVSwQXfGj95h5lC/zKF/mUb7Mo3yZp7jly2KFvr29PZmZedfivlng5zetJj/btm3j7bffpk2bNgwdOrTAftevX+e1117j0KFDREdHU758+bvuu2fPnowfP56ffvrpngr9c+fSyM5+sHP93d1LkZKiJ5gWlvJlvjvlrGYFZ0b1b8KU+F18OHs7W3acpHdHH+xtLbrAl0XpPWYe5cs8ypd5lC/zKF/msVS+jEZDgReXLTZH393dPd/pOSkpKQCFKsT37t3LkCFD8PX1ZeLEiVhZWRXYd+TIkWzYsIGxY8cSFBRUqBiNRiMeHh5cunSpUP1FBMqWtufNXgF0beHN5t9PM3rmNo4n6w+FiIjIg2axQr927docOXKEK1eu5Gr/7bffTNvv5Pjx4wwcOJAyZcowefJkHB0Lng88duxYFixYwL///W86d+5c6BgzMzM5depUoW7wFZG/WBmNPNmyOm/0DCA9I4sxX29jTeIJrWYlIiLyAFms0O/UqROZmZnMmzfP1JaRkcGCBQto1KiR6UbdkydP5loyE25c9Y+MjMRgMBAdHZ1nCc1bTZs2jenTp/Piiy/Sp0+fAvudP38+T1t0dDTp6em0bNnS3NMTEcCvqhtRkUHU8S7DrFX7+WzBTtKu5p2yJyIiIkXPYhNnGzRoQKdOnZgwYQIpKSl4eXmxcOFCTp48yQcffGDqN3z4cBISEti3b5+pbeDAgSQlJTFw4EASExNJTEw0bfPy8jI9VXfVqlWMHz8eb29vqlevTlxcXK4YOnToYPomoG3btnTu3BkfHx9sbW35+eefWbFiBYGBgYSGht7PVIiUaC6Otgzt4c+qrUnMW3+IUdMTGNy1Lj5VXC0dmoiISIlm0Tvkxo0bx8cff0xcXByXLl3C19eXKVOmEBgYeMdxe/fuBW5crb9deHi4qdC/2e/o0aO8+eabefquWbPGVOiHhYWxfft2li9fTmZmJpUqVeKll15i8ODBWFs/ujcSihQFg8FAxyAvfLxc+d+iXYydvZ1uIdUIbe6N0Wi4+w5ERETEbIYcTZq9b7TqTvGnfJnv7+bsanoWMSv2sWV3MrW9XBkUVhe3UoVbZethpPeYeZQv8yhf5lG+zKN8mUer7ojII8/BzppBYXWI7OzH4VOXGTU9gd8OnrV0WCIiIiWOCn0ReeAMBgMh/p6M6t8Et1J2fBK7gzlrDpCZlW3p0EREREoMFfoiYjGeZZ0Y2TeQxxpVZuXWJN6PSST5/J+WDktERKREUKEvIhZlY21F744+vBJRn7OXrhI1cys//X7a0mGJiIg89FToi0ixEODjzujIILzKOzN18W6iF+/mWkaWpcMSERF5aKnQF5Fio4yLPW/2CqBrC282/36a0TO3cTxZKz6IiIjciyIp9LOyslixYgXfffcdKSkpRbFLEXlEWRmNPNmyOm/0DCA9I4sxX29jTeIJtBKwiIiIecx+EtS4ceP4+eefmT9/PgA5OTkMGDCAbdu2kZOTg6urK9999x1eXl5FHqyIPDr8qroRFRnE9CV7mLVqP7uPnmdAZz+cHWwsHZqIiMhDwewr+hs3bqRx48am12vXrmXr1q08//zzfPjhhwBMmTKl6CIUkUeWi6MtQ3v482y7muw4dI5R0xPYn3TR0mGJiIg8FMy+on/69GmqVq1qer1u3ToqV67MG2+8AcCBAweIj48vughF5JFmMBjoGOSFj5cr/1u0i7Gzt9MtpBqhzb0xGg2WDk9ERKTYMvuKfmZmJtbWf30++PnnnwkODja9rlKliubpi0iR867gwqgBTWjq58GijUeYMOcXLqSmWzosERGRYsvsQr9ChQr88ssvwI2r90lJSTRp0sS0/dy5czg6OhZdhCIi/8fBzppBYXWI7OzH4VOXGTU9gV8PnrV0WCIiIsWS2VN3unTpwhdffMH58+c5cOAAzs7OtG7d2rR9z549uhFXRO4bg8FAiL8nNSq58L+4XXwau4MOjavQo00NbKy1YrCIiMhNZv9VHDx4MOHhVgyfswAAIABJREFU4fz6668YDAbGjh2Li4sLAKmpqaxdu5bmzZsXeaAiIrfyLOvEyL6BPBZYmVXbkng/JpHk839aOiwREZFiw+wr+ra2trz//vv5bnNycuLHH3/E3t7+bwcmInI3NtZW9O7gQ52qbkxfuoeomVvp29GX5vUqWDo0ERERiyvS77mzsrIoVaoUNjZa51pEHpwAH3dGRwbhVd6ZqYt3E714N9cysiwdloiIiEWZXehv2LCBSZMm5WqbNWsWjRo1omHDhgwbNozMzMwiC1BEpDDKuNjzZq8AurbwZvPvpxk9cxvHk1MtHZaIiIjFmF3oR0dHc/jwYdPrQ4cO8f7771O+fHmCg4NZunQps2bNKtIgRUQKw8po5MmW1XmjZwDpGVmM+XobaxJPkJOTY+nQREREHjizC/3Dhw9Tr1490+ulS5diZ2dHbGws06ZNo3PnzixatKhIgxQRMYdfVTeiIoP4/+zde1yUdd4//tc1B2Y4zDAchpOcVE6KoojHtNZzrFJmq7XlLplf3Wp/7ZZbe2vbfX/N9i53O+q2d7/SMFfKTAwiyoysbttIU9QklUBRBERgRBnkOMBc3z8GRscZ0OE0A7yej4cPh898ruv6zNuL8T3XfK73Z3S4N97/shD/TP8JdY38ppGIiIYWuxN9vV4PLy8v88/ff/89pk6dCg8PDwDA5MmTUVZW1nsjJCLqBrWbC55YEodfz45AXlE11m09hMLSGkcPi4iIqN/Yneh7eXmhvLwcAFBXV4effvoJEydOND/f2tqKtra23hshEVE3CYKA+ZND8WxyAuQyCf6+4yg++e4cjEZO5SEiosHP7vKa48ePx86dOxEREYFvv/0WbW1tuOOOO8zPnz9/Hn5+fr06SCKinggPUGPd8klIzS7Ax9+dQ/75K/jd3bHwUikcPTQiIqI+Y/cV/T/+8Y8wGo148sknkZ6ejnvuuQcREREAAFEUsW/fPkyYMKHXB0pE1BOuChlWJY3G/1k4CucqarFu6yH8eOaSo4dFRETUZ+y+oh8REYE9e/bg6NGjUKlUmDRpkvm52tpaPPTQQ5gyZUqvDpKIqDcIgoDpYwMxIkiNtzJP4h+78zBvYgiWzBwJuaxXlxUhIiJyOLsTfQDQaDSYPXu2VbunpyceeuihHg+KiKgvBfq44z+TE7DrmyJ8mVuKwtIaPLooFv7ebo4eGhERUa/pVqIPACUlJfjqq69QWloKAAgJCcGcOXMQGhraa4MjIuorcpkUy+ZFYXSYF7buycdz2w4jeX40po0JcPTQiIiIekW3Ev2NGzdiy5YtVtV1Xn75ZTzyyCN44oknemVwRER9LT5Ki/UBKrz9yUls+fQUThVfxrL5UVC6dPs6CBERkVOw+3+y3bt346233kJ8fDxWrlyJyMhIAMDp06eRkpKCt956CyEhIbj33nt7fbBERH3BW63EfzwYj6ycYmTlFONMeS0eWxSLUH+Vo4dGRETUbXbffbZjxw6MGzcOqamp5qk6oaGhmDNnDrZv3464uDi89957fTFWIqI+I5VIcM/tI/D0A/FoNrTiv7fn4qsjZRBF1twnIqKBye5Ev6ioCAsWLIBMZv1lgEwmw4IFC1BUVNQrgyMi6m+jwrzw3IrJGB3ujfe/LMQ/039CXWOLo4dFRERkN7sTfblcjoaGhk6fr6+vh1wu79GgiIgcSe3mgieWxOHXcyKRV1SNdVsPoaDkiqOHRUREZBe7E/2xY8fiww8/xKVL1gvNVFdXY9euXRg3btwt7ctgMODll1/GjBkzEBcXh/vuuw8HDhy46XbZ2dl48sknMXv2bIwbNw6JiYn4+9//jqtXr9rsn5aWhl/+8pcYO3Ys7rzzTrz//vs2+1VWVuKJJ57AxIkTMWHCBPz+9783VxUioqFFEATMnxSCZ5MTIJdJ8NIHx/DJd+dgNHIqDxERDQyCaOcE1MOHD2P58uVwd3fHr371K/OquGfOnEF6ejrq6+uxbds2TJw48ab7+tOf/oTs7GwkJycjLCwMGRkZOHHiBFJTUxEfH9/pdlOmTIGfnx/mzp2LoKAgFBQUYOfOnQgPD8dHH30EheLasvY7d+7EunXrkJiYiOnTpyM3NxeZmZlYs2YNVqxYYe5XX1+Pe++9F/X19Vi+fDlkMhm2bdsGQRDw8ccfw9PT054wAQCqq+v6PSnQalXQ6Wx/4CFrjJf9hmLMGptbkZpdgIMnKxEdosHv7o6Fl0px8w0xNOPVE4yXfRgv+zBe9mG87OOoeEkkAnx8PGw+Z3eiDwBff/01/vrXv+LixYsW7UFBQfi///f/YubMmTfdR15eHpYuXYpnnnkGy5cvBwA0NzcjKSkJfn5+nV51B4AffvjBavXdjz/+GGvWrMGGDRvMFX+amprwi1/8AgkJCXjzzTfNfZ9++ml8/fXX2L9/P1QqU1WNLVu24NVXX0V6ejpGjx4NwHQ/wl133dXtkqFM9J0f42W/oRozURTx/YkKpGYXwEUmxYqFozA+wvem2w3VeHUX42Ufxss+jJd9GC/7OGOi360132fPno2vvvoKu3btwmuvvYbXXnsNaWlp2LdvHyoqKrBgwYKb7mPv3r2Qy+VYunSpuU2hUGDJkiU4cuQIqqqqOt32xiQfAObOnQsAFjcC//DDD6ipqcGDDz5o0XfZsmWor6/Ht99+a2774osvMH78eHOSDwAjR47EtGnT8Pnnn9/09RDR4CYIAqaPDcS65ZPgpVLgH7vz8MG+02hpNTp6aERERDZ1K9EHAIlEgri4OCxYsAALFizA2LFjIZFIcOXKFZw7d+6m2+fn52P48OFwd3e3aI+Li4MoisjPz7drPB33DHh5eZnbTp06BQAYM2aMRd/Y2FhIJBLz80ajEQUFBVb9ANM9CcXFxWhsbLRrPEQ0OAX6uOM/kxMwJyEYX+aW4sXUI6i83HmBAiIiIkfpdqLfUzqdDn5+flbtWq0WALq8om/Lli1bIJVKMX/+fItjuLi4QKPRWPTtaOs4Rk1NDQwGg/nYN45HFEXodDq7xkNEg5dcJsWyeVH4w71jcUnfiOe2HcaBExWOHhYREZEFh63x3tTUZLMMZ8eNtM3Nzbe8r6ysLOzevRuPPPIIQkNDb3qMjuN0HKPjbxcXl07H09TUdMvj6dDZfKm+ULX/W5Skvo/CS9VQ+Pog9LfL4PeLO/rt+AOZVsvVT+3FmJnM16oQPzoQr7yfiy2fnkJRxVU8em8cXBWWb62Ml30YL/swXvZhvOzDeNnH2eLlsERfqVSipcV6EZqOpPv6yjldyc3NxbPPPouZM2da3TCrVCphMBhsbtfc3Gw+Rsfftvp2jEepVN7SeK7XXzfj1h78HpXbt0FsH3+z7hLO/PP/x9XaRqin3tbnxx/IeKOR/Rgza6uXxiErpxhZOcU4ebYajy2KRai/6c2e8bIP42Ufxss+jJd9GC/7DJqbcXuDVqu1OT2nY4qMrWk9N/r555/x2GOPITo6Gq+//jqkUqnVMVpaWlBTU2PRbjAYUFNTYz6GRqOBi4uLzek5Op0OgiDYnNbjLC6lf2RO8juIBgN0abvQVl+PbhRWIiI7SCUS3HP7CPz5gXg0G1rx39tzsS+3lL97RETkULd0Rf/dd9+95R0ePXr0lvrFxMQgNTUV9fX1FjfkHj9+3Px8V0pKSrBy5Up4e3vj7bffhpubm1WfUaNGAQBOnDiBGTNmmNtPnDgBo9Fofl4ikSAqKgonTpyw2kdeXh7CwsLg6up6S6/LEVovV9tsb9PXoOiJ/w+CTAappydknhrIPDXtjz2ve6yBTOMJqUoN4YYPS0R062LCvLB+xWSkfJaPHftO41TxFfw5eZKjh0VEREPULSX6f//73+3aqSAIN+2TmJiIrVu3Ii0tzVxH32AwID09HRMmTIC/vz8AoLy8HI2NjRg5cqR5W51OhxUrVkAQBKSkpMDb29vmMaZOnQqNRoMdO3ZYJPoffPAB3NzccMcd1+aw33nnnXjttddw6tQpc4nNs2fP4uDBg1i1apVdr7+/ybx9bCb7Eg8P+CxIQqu+Bq01erTV6mGovIjWgp9hbKi33pEgQOqhMiX96o4PABrTY037BwO1J2QaDSS3OLWKaKhRubngiSVx+DK3DGnfnMETr36D/7NwFKJDvW6+MRERUS+6pQWzDh06ZPeOJ0+efNM+TzzxBL766is89NBDCA0NNa+M+69//QsJCQkAgN/+9rc4dOgQCgoKzNstWrQIP//8M1auXImoqCiLfYaGhlqsqvv+++/j+eefR2JiImbMmIHc3Fx8/PHHePrppy0S+Lq6OixevBiNjY14+OGHIZVKsW3bNoiiiI8//tiibOetctQcfQAQXFzgn7y80zn6xpYWtNXq0VpT0/63Hq36GnNbq970waBVrweM1nXCJUrldd8SeELa/veN3xhIPDxu6YOfo3D+of0Ys1tXXFGLLZ/mo6K6HoumD0fSbeGQSJz398EZ8PyyD+NlH8bLPoyXfZxxjn63VsbtLc3Nzdi4cSOysrKg1+sRHR2NP/3pT7jttmvJqa1EPzo6utN9Ll68GH/7298s2nbt2oWtW7eirKwMgYGB+O1vf4vk5GSrbSsqKvDiiy8iJycHRqMRU6ZMwbPPPouQkJBuvb7+XBm39uD3uJT+EVqvXIbMyxu+9/6qV27EFY1GtNXVoU1v+iDQqq9pf6y3eizaqpQklUKm9jQl/xqNjcemaUMytScEWf/fG843MfsxZvZxVynx+o4jOHiyEtEhGvzu7lh4qfiNWGd4ftmH8bIP42Ufxss+TPSHmP5M9Ds48pfS2NTU/mFAf+2DQcc3Bvpr3x60XbU9PomHh+kbArUnpO1ThSwet39zIFEqe+1bAr6J2Y8xs49Wq0JVVS2+P1GB1OwCuMikWLFwFMZH+Dp6aE6J55d9GC/7MF72Ybzs44yJvsPKa9LgI1Eq4aIMgIt/QJf9xNZWtNbWWk8Tqrn2jYGhoAJttXqIra1W2wsuLjdMF+p43PFhoH0KkUoFQeKwwlJEZoIgYPrYQIwIUuOtzJP4x+48zJsYgiUzR0Iu4zlKRER9g4k+9TtBJoPc2xvyTm6i7iCKIoz19ddNE6ppf6w3P26+UIaGUydhbGy03oFEAqlKbfow0MlNxU3GYTC2SSGRWy+WRtTbAn3c8Z/JCdj1TRG+zC1FYWkNHl0UC39v66phREREPcVEn5yWIAiQenhA6uEBxbBhXfY1NjejtVaPtho9WmvbvyWoqUFrx43GV66g6Xwx2mprgetmq5W2/y1xc+uk9KhlGVKJm5tT31xMzk8uk2LZvCiMDvPC1j35eG7bYSTPj8a0MV1/E0ZERGQvJvo0KEgUCrho/QBt1wutiW1taLt6tf0DQA3cjE24UlZp8S1BU1GR6eZiGys3c00C6i3xUVqsD1Bh8ycnseXTUzhZfBm/mR8FpQvflomIqHfwfxQaUgSp1FTxR6MBQsOg1aogsXHjjCiKMDY22pguxDUJqPd4q5X484PxyMopRlZOMYrKa/Ho3bEIC1A5emhERDQIMNEnskEQBEjd3CB1c4NLYFCXfW9lTQJD+YVBvyYBdY9UIsE9t49ATKgXNmedxAupubhvVgTmJATz35uIiHqEiT5RD0nkckh8fCH36bpc4q2sSdB0/jxa9ccH5JoE1DMxYV5Yv2IyUj7Lx459p3Gq+ApWLBwFD1e5o4dGREQDFLMBon4iSCSQqdWQqdVQ3GQRtputSdCi06Gp6IxTrUlAPadyc8ETS+LwZW4Z0r45g3VbD+F3d41GdKj9K3MTEREx0SdyQlyTYOgSBAHzJ4UgKsQTb2WexEsfHMOi6cORdFs4JBJ+KCMiolvHRJ9oAHPEmgQ6P1+0Kd2tbiru+GDANQl6R3iAGuuWT0JqdgE+/u4c8s9fwe/ujoWXijdvExHRrWGiTzQE9OaaBIbqy2i6fMZqTYIOXJOg97gqZFiVNBqx4d5IzS7Auq2HsGLhKIyP6Pp+ECIiIoCJPhHd4GZrEmi1Kuh0V63WJLD1LQHXJOg5QRAwfWwgRgSp8XbmSfxjdx7mTgzG0pkRkMs4nYqIiDrHRJ+IuuXGNQk6wzUJekegjzueTU7Arm+KsC+3DIWlNXhs0Rj4e7s5emhEROSkmOgTUZ/imgS9Ry6TYtm8KIwO88LWPfl4btthJM+PxrQxXd+0TUREQxMTfSJyGlyT4NbER2mxPkCFzZ+cxJZPT+Fk8WX8Zn4UlC7ON1YiosGu9uD3uJT+EQqvXIbMyxu+9/4K6qm3OXpYAJjoE9EAxDUJAG+1En9+MB5ZOcXIyilGUXktHr07FmEBqn4bAxHRUFd78HtUbt8G0WAAALRerkbl9m0A4BTJPhN9IhrUem1NgtrurUlgDA5Ao0TRJ2sSSCUS3HP7CMSEemFz1km8kJqL+2ZFYE5C8ICcmkRE5GiiKAJtbRBbW01/2lohtrZd9/jaH7S1QffhB+Yk37wPgwGX0j9iok9E5Cy6vyaB3uobg+vXJKi+cQfXrUnQ2U3F9q5JEBPmhfUrJiPls3zs2Hcap4qvYMXCUfBwlXcvGEREvUw0GiG2tQE2E+c2qyTalEjfvK9FUm6RnLdCvO65TpP31haLfaKtrVdeb+tlq3d/h2CiT0RkB3vXJPCUtaLqbLnNNQlar1xB0/niXlmTQOXmgieWxOHL3DKkfXMG67Yewu/uGo3oUK++CgUROQFTAm1KXNFV8myR9LZCbLl53zoXCRquNlyXFLd2vU9bSXlrS3uC3zsJtBWpFIJMBkEqgyC77rG8o830B1IpJApF+/PXb9O+nVQGQS6/tj9b+2zvi+v2e/GtN9FWq7calszbp29er52Y6BMR9RGJQgGl1heuUvcu+/XmmgRjPDUYoXHF4QtN2PvWCZSNDce0qVFw8dIM+TUJiOxxfQLdcdW360T6hmT7FhJp3OSKtrlfZ1NI+jCBFmQy6NsTZNtJsalNIpcDSlebCbFVon1LibTMRp/O9+noaYra++63mKMPmKZy+t77KweO6hom+kREDtYraxK0Tx0yVF4E9Hok1LevSfDVAZR91XEgrklAjicajTamULSYktkurkKLba1AJ3OlbSXBsLGvCkGEobHZep+2knAbJXx7gzlJ7eKKsSCVQeKiMF09vkk/i6RXLr9pX3SSXF9LrKXmBLpjgUTqXMc8/EvpH6GVVXeIiKi7urMmwdGjRfh3TgHUxiZMH+4OT6mBaxIMQqIoAh0J9A1JrzmBvmEucpd9b5JI3zQp7yLhtjVNrccEweIKMG5MZNuvEMNVYUqg3dwgyOQ3JNLWybN5O4tkW26j37XkGZ3tSyYDJBL+vgxC6qm3QT31Nqf8YMREn4hoEOpYk2DKPF+EThiDtzNP4tXKOsydGIylD0ZALjNV/nGWNQmctQ61uQJHJzfyWU+9aOn05kF0cpXa1hSPLhPp67YpamuDsaV9DnRfJ9A3JtI3JMQShQKCu7tFQmyVcF+/XXs/y0TbOsnuvK/UIpG/Fc6YiBH1JSb6RESDXKCPO55NTsCub4qwL7cMhaU1eGzRGPh7uznFmgTNuipc/fd+dJQtbb1cjcp/vQtDlQ7uo0ZZTfG4aeWNG6dx3JBIm5Pzrip6tFxL4vuERHJDMnvtKjFuSHQlrq5WSS/aE2l3lSsaDcYu5zxbJeWd9IWtq9C9VAqWiBxDEMW+uARAAFBdXQejsX/Dy6sV9mG87MeY2cfZ4nWsUIete/LRahSRPD8a08Z0vb5Ad9zKmgStNfpO1yToEanUdkWNjmkaNqZldD71out+nSbHt9K3lxJoZzu/nB3jZR/Gyz6OipdEIsDHx8Pmc7yiT0Q0hMRHabE+QIXNn5zElk9P4WTxZfxmfhSULr3334G9axIUPfl4p32G/enPtq982yp3J5XyCjQR0XWY6BMRDTHeaiX+/GA8snKKkZVTjKLyWjx6dyzCAlT9Oo6ONQlk3j42F5eRefvAfXRsv46JiGgw4aUPIqIhSCqR4J7bR+DPD8Sj2dCKF1JzsS+3FI6Yzel7768guFiuAuxMdaiJiAYqJvpERENYTJgX1q+YjNHh3tix7zTe+Ogn1DVaL8rVl9RTb4N/8nLTSpKCAJm3D/yTlztF1R0iooGMU3eIiIY4lZsLnlgShy9zy5D2zRms23oIv7trNKJDvfptDM5ch5qIaKDiFX0iIoIgCJg/KQTPJidALpPgpQ+OIfO7c/1eOYyIiHqPQ6/oGwwGbNq0CZmZmaitrUVMTAxWr16NadOmdbldXl4e0tPTkZeXh8LCQrS0tKCgoMCq3xtvvIF//vOfne5nx44dSEhIAACsXbsWGRkZVn3GjRuHXbt22fnKiIgGpvAANdYtn4T3sguQ+d05/Hz+ClbdNRreaqWjh0ZERHZyaKK/du1aZGdnIzk5GWFhYcjIyMCqVauQmpqK+Pj4Trfbv38/0tLSEB0djZCQEJw9e9Zmv3nz5iE0NNSq/fXXX0dDQwPGjh1r0e7q6or169dbtHnfpDwcEdFg46qQYdVdsRgd7o33sgvx3LuHsWLhKIyP8HX00IiIyA4OS/Tz8vLw2Wef4ZlnnsHy5csBAPfccw+SkpLwyiuv4P333+902wceeACrVq2CUqnECy+80GmiHxMTg5iYGIu2ixcvoqKiAkuXLoXLDVUeZDIZFi1a1LMXRkQ0SEwfG4gRQWq8nXkS/9idh7kTg7F0ZgTkMs76JCIaCBz2br13717I5XIsXbrU3KZQKLBkyRIcOXIEVVVVnW7r6+sLpbJ7XyN/+umnEEURd911l83n29raUFdX1619ExENNoE+7ng2OQFzEoKxL7cML6TmovJyg6OHRUREt8BhiX5+fj6GDx8Od3d3i/a4uDiIooj8/Pw+OW5WVhYCAwMxadIkq+fq6+uRkJCAhIQETJkyBRs2bEBzc3OfjIOIaKCQy6RYNi8Kf7h3LKr1TXhu22EcOFHh6GEREdFNOGzqjk6ng7+/v1W7VqsFgC6v6HfX6dOnUVBQgJUrV0IQBKvjrly5EqNGjYLRaMQ333yDbdu2oaioCO+8806vj4WIaKCJj9JifYAKmz85iS2fnsLJ4sv4zfwoKF1YqZmIyBk57N25qakJcrncql2hUABAn1xJz8rKAgCb03aeeuopi5+TkpLg7++PlJQU5OTkYPr06XYfz8fHo3sD7SGttn+XsR/oGC/7MWb2GUzx0mpVeOmPd2Dnl4X4cF8BzldexZ9/MxEjgzW9egy6dYyXfRgv+zBe9nG2eDks0VcqlWhpsV59sSPB70j4e4soivj0008RFRVldYNuZ1asWIGUlBQcOHCgW4l+dXVdv9eg5mIz9mG87MeY2Wewxmt+wjCE+rphc9ZJPP2Pb7F0VgTmJgRbfVtqr8Ear77CeNmH8bIP42UfR8VLIhE6vbjssDn6Wq3W5vQcnU4HAPDz8+vV4x05cgQXLlzo9CZcW3x9fSGXy6HX63t1LEREg0FMmBfWr5iM2HBvfLDvNN746CfUNVpfwCEiIsdwWKIfExODc+fOob6+3qL9+PHj5ud7U1ZWFgRBQFJS0i1vU1FRgZaWFtbSJyLqhMrNBX9cEocH5kTip7PVWLf1EApKrjh6WEREBAcm+omJiWhpaUFaWpq5zWAwID09HRMmTDDfqFteXo6ioqIeHaulpQV79+5FQkICgoKCrJ5vbm62WVLzzTffBADMmDGjR8cnIhrMBEHAvEkheDY5AXKZBC99cAyZ353r96mLRERkyWFz9MeNG4fExES88sor0Ol0CA0NRUZGBsrLy7FhwwZzvzVr1uDQoUMoKCgwt124cAGZmZkAgJ9++gnAtaQ8JiYGs2fPtjjWd999h5qamk6n7eh0OixevBhJSUkYMWKEuerOgQMHsGDBApulOImIyFJ4gBrrlk/Ce9kFyPzuHH4+fwWr7hoNb3X31j0hIqKecWhNtJdeegkbN25EZmYm9Ho9oqOjsXnzZiQkJHS5XVlZGTZt2mTR1vHz4sWLrRL9rKwsyOVyJCYm2tyfWq3GzJkzkZOTg4yMDBiNRoSHh2Pt2rVITk7uwSskIhpaXBUyrLorFqPDvfFediGee/cwViwchfERvo4eGhHRkCOIosjvVvsIq+44P8bLfoyZfYZyvC5W1+PtzJMoqarD3InBWDozAnJZ1zNGh3K8uoPxsg/jZR/Gyz6sukNERENGoI87nk1OwJyEYOzLLcMLqbmovNzg6GEREQ0ZTPSJiKjPyGVSLJsXhT/cOxbV+iY8t+0wDpyocPSwiIiGBCb6RETU5+KjtFi/YjLC/Dyw5dNTeOfTU2gytDp6WEREgxoTfSIi6hfeaiX+/GA87p4ejgMnK7D+3cM4X8H5v0REfcWhVXeIiGhokUokuOf2EYgJ9cLmrJN4ITUXS2dFwF0pQ8a3Z3G5thneagXu/cVITIsNcPRwiYgGNCb6RETU72LCvLB+xWRs/SwfH+w7DUEAOmrAVdc241+f/wwATPaJiHqAU3eIiMghVG4u+OOSOLgpZbix0LOh1Yj0/T1bFZ2IaKhjok9ERA4jCAIammzflFtd24xPvy9GYWkNWlrb+nlkREQDH6fuEBGRQ/moFaiubbZql0oEpH97FgAgk0owIkiNqBBPRIVoMDLIE64K/hdGRNQVvksSEZFD3fuLkfjX5z/D0Go0t7nIJHjolzEYO8IHp8tqUFhq+rPnQAk+/f48JIKAUH8PRIVoEB2iQWSIBh6ucge+CiIi58NEn4iIHKrjhtv0/UU2q+7ER2oRH6kFADQZWlF0oRYFpTU4XVqDr49eQPbhUgDAMF93RIVoEBniiegQL3ipFI55QUREToKJPhEROdy02ABMiw2AVquCTtd5bX2liwyxw70RO9wbANDSasS5i7U4XVaDgtIaHDhZgW+OXQAAaDVKRAVrEBWiQVSoBn4aVwiC0C+vh4jIGTDRJyImcoEfAAAgAElEQVSiAUsuk5gS+RANFk4D2oxGlFXVo6B9qs/xomrknKgAAHi6u5j7RodoEKR1h4SJPxENYkz0iYho0JBKJAgLUCEsQIX5k0IgiiIuVjeY5viX1aCgpAaHf64CALgrZYgMNk31iQrRIMxfBZmUxeiIaPBgok9ERIOWIAgI8nVHkK87ZsYPgyiKqNY3obD9Bt+CUj1+PHMJAOAil2BkkCei26/6jwhSw0UudfArICLqPib6REQ0ZAiCAF+NK3w1rrhtTCAAQF9vwOnSGvMNvpnfnYMIU3nP4YFq83SfiGGecFPyv00iGjj4jkVEREOap7sLJsb4YWKMHwCgoakFp8v05qv+XxwqwZ6D5yEIQIifqaRnx02+ancXB4+eiKhzTPSJiIiu46aUY1yEL8ZF+AIAmlvacLa81lzL/9sfy7EvtwwAEOjjhsjgjlr+nvD1dHXk0ImILDDRJyIi6oJCLsWoMC+MCvMCALS2GXG+4qo58T/8cxW+PV4OwLTKb8dUn6gQDQK83VjSk4gchok+ERGRHWRSCUYO88TIYZ745dQwGI0iynR17ZV99DhZfAUHTlYCAFRucoupPiF+HpBImPgTUf9gok9ERNQDEomAUH8VQv1VmDvRVNKz6kqjuZZ/YWkNjhToAACuCikihmkQ1b56b3ggS3oSUd9hok9ERNSLBEGAv7cb/L3dcMe4IADA5domc9JfWKbHR/urAZgW/BoZZKrsExmiQUSQJxQuLOlJRL2DiT4REVEf81YrMTU2AFNjAwAAtQ0GnC7V43SZqaxn1vfFEEVTSc+wAJV5qk9kiCfclXIHj56IBiom+kRERP1M7eaChGgtEqK1AIDG5lYUXdCbp/vsO1KKvYdKIAAYpnW3uMFX46Fw7OCJaMBgok9ERORgrgoZxozwwZgRPgCAltbrSnqW6ZHzUwW+PnoBAODn5Wq+wXfaeAkkosjKPkRkExN9IiIiJyOXSREd6oXoUFNJzzajESWVdeZ5/scKdfgu7yK27smHl0qByGBPRLdf8Q/0dYeEiT8RgYk+ERGR05NKJBgeqMbwQDXunBwKoyji4qV6lNc04cipChSW1uBQfhUAwF0ps5jqE+rvAamElX2IhiIm+kRERAOMRBAwTOuB8aMDMSnSF6IoQqdvQmFJDQrL2q/6n74EAFC4SBExzLN9uo8nRgSpIZexsg/RUMBEn4iIaIATBAF+Glf4aVwxIy4QAFBT13ytpGdpDTK+PQsAkEkFDA80lfSMDtFg5DBPuCqYDhANRvzNJiIiGoQ0HgpMHuWPyaP8AQB1jS04U6Zvv8G3Bp8fLMFnB85DEIBQf5V5jn9ksCdUbi4OHj0R9QYm+kREREOAh6sc4yN9MT7SFwDQZGhFUXktTrdf8f/m2AVkHy4FAAT5upun+kSFaOCtVjpy6ETUTQ5N9A0GAzZt2oTMzEzU1tYiJiYGq1evxrRp07rcLi8vD+np6cjLy0NhYSFaWlpQUFBg1a+srAxz5syxuY8tW7bgjjvusGgrKirCiy++iKNHj0Iul2PWrFlYs2YNvL29u/8iiYiInJDSRYbYcG/Ehpv+j2tpNeJ8xVUUlF5BYakeB09W4H+PmUp6+noqLW7w9fdyZUlPogHAoYn+2rVrkZ2djeTkZISFhSEjIwOrVq1Camoq4uPjO91u//79SEtLQ3R0NEJCQnD27Nkuj3P33XdjxowZFm0xMTEWP1dUVGDZsmVQq9VYvXo1GhoasHXrVhQWFmLXrl2Qy7kyIRERDV5ymQQRwZ6ICPbEwmmA0SiitOpaSc+fzlbj+xMVAAC1u4t5jn9ksCeC/TxY0pPICTks0c/Ly8Nnn32GZ555BsuXLwcA3HPPPUhKSsIrr7yC999/v9NtH3jgAaxatQpKpRIvvPDCTRP92NhYLFq0qMs+b731Fpqbm5Gamgp/f9N8xri4ODz88MPIzMzEkiVL7HuBREREA5hEIiAsQIWwABXmTQqBKIqouNyAgtIa83Sf3J9NJT3dFDJEtk/ziQrRICxABZmUJT2JHM1hif7evXshl8uxdOlSc5tCocCSJUvw+uuvo6qqCn5+fja39fX1tft4DQ0NkMlkcHGxfYNRdnY2Zs+ebU7yAeC2225DeHg4Pv/8cyb6REQ0pAmCgEAfdwT6uGPm+GEAgEv6Rpwu1aOgPfE/XlQNAHCRSzAy6FriPyJIDYWcJT2J+pvDEv38/HwMHz4c7u7uFu1xcXEQRRH5+fmdJvr22rRpEzZs2ABBEDBu3Dg8/fTTmDRpkvn5yspKVFdXY8yYMVbbxsXFIScnp1fGQURENJj4errC19MV08YEAAD09Qbz1f7Cshp88t05iACkEgHhgar2G3xN033clJwSS9TXHJbo63Q6i6vnHbRaLQCgqqqqx8eQSCSYMWMG5s2bBz8/P5w/fx4pKSl4+OGHsW3bNkycONHiWB3HvnE81dXVaGtrg1TKqxFERESd8XR3wcQYP0yMMV2oa2hqwZkLehSWmsp6Zh8qxecHSyAACPHzQGTHPP8QDTzdWdKTqLc5LNFvamqyeYOrQqEAADQ3N/f4GEFBQUhJSbFoW7BgARYuXIhXXnkFO3futDiWrWk9HeNpamqy+vbhZnx8PLoz7B7TalUOOe5AxXjZjzGzD+NlH8bLPs4er7AQb8yZanrcZGjF6ZIanDhbjZNnL+G7ny7iqyNlAIBhWnfEjvBF7AgfjBnhAz9vtz4Zj7PHy9kwXvZxtng5LNFXKpVoaWmxau9IujsS7N7m7++PhQsXYteuXWhsbISrq6v5WAaDodPxKJX21xCurq6D0Sj2bMB20mpV0Omu9usxBzLGy36MmX0YL/swXvYZiPEK8FQgID4Ic+OD0NpmxPnKqygsrcHpUj2++/ECsn84DwDwVivMU32iQjQI9HHrcUnPgRgvR2K87OOoeEkkQqcXlx2W6Gu1WpvTc3Q6HQD02vx8WwIDA2E0GlFbWwtXV1fzsTqOfeN4fHx8OG2HiIiol8mkppt2RwZ54pdTAKMo4oKu3lzSM7/4Cg6erAQAqNzkiGxP+qNDNAjx84BEwpKeRF1xWKIfExOD1NRU1NfXW0yJOX78uPn5vlJaWgqpVApPT08Apqv83t7eOHHihFXfvLw8jBo1qs/GQkRERCYSQUCInwdC/DwwJyEYoiiiqqYRhSXXbvA9Wmi6KKd0kSIi2BPR7ZV9wgPUkMtY0pPoeg5L9BMTE7F161akpaWZ6+gbDAakp6djwoQJ5ht1y8vL0djYiJEjR9p9jMuXL1utanv+/Hl89tlnmDhxosV0nPnz5+OTTz5BZWWl+dgHDhxAcXExVq5c2c1XSURERN0lCAL8vdzg7+WG28cFAQAu1zahsMw01aewtAYf7TetpSOXSTAiUG0u6TlymBpKF4euC0rkcA77DRg3bhwSExPxyiuvQKfTITQ0FBkZGSgvL8eGDRvM/dasWYNDhw6hoKDA3HbhwgVkZmYCAH766ScAwJtvvgnA9E3A7NmzAQAvv/wySktLMXXqVPj5+aGkpMR8A+6aNWssxvPoo49i7969SE5Oxm9+8xs0NDQgJSUFMTExN11si4iIiPqHt1qJqaMDMHW0qaTn1QYDzpRdq+X/6YFiiN+bvh0IC/AwJ/6RwRpY19YjGtwc+lH3pZdewsaNG5GZmQm9Xo/o6Ghs3rwZCQkJXW5XVlaGTZs2WbR1/Lx48WJzoj99+nTs3LkT7733Hq5evQq1Wo3p06fj8ccfR2RkpMX2gYGBeO+99/C3v/0Nr776KuRyOWbOnIlnnnmm00W2iIiIyLFUbi6Ij9IiPsqUxjc2t6Ko3HS1v7CkBl8duYAvDpUCAMICVBgRpDaV9AzWwEvVN4U/iJyFIIpi/5aFGUJYdcf5MV72Y8zsw3jZh/GyD+N1cy2tbTh30VTZp7iyDifPVaPZ0AYA8NO4mq72h5jm+ms1rj2u7DOY8PyyD6vuEBEREfUjuUxqnr6j1apQUalHaVUdCktqUFBagx/PmOr5A4DGw8XcNypYgyCtOyRM/GkAY6JPREREQ4ZUIkF4gBrhAWrMnxwKoyjiYnWDuaRnYWkNDuWbyn+7K2Xmkp5RIRqE+ntAJmVlHxo4mOgTERHRkCURBAzzdccwX3fMih8GURRxSd9kkfj/eOYSAEAhlyJimBqR7bX8hweq4SLnOjvkvJjoExEREbUTBAFajSu0GldMHxsIANDXNaOwTG+e7pP573MQAcikAsID1eZa/hHDPOGqYGpFzoNnIxEREVEXPD0UmBTjh0kxfgCA+qYWnC4zVfY5XVqDvT+U4LMD5yEIQKifqn2qjyciQzRQu7FyHzkOE30iIiIiO7gr5Rgf4YvxEb4AgGZD27WSnqU1+N8fL+DLXFNJz0AfN/Mc/+gQDbzVyq52TdSrmOgTERER9YDCRYrR4d4YHe4NAGhtM6K44qrFzb37fywHAPiolaakP9SU/Pt7saQn9R0m+kRERES9SCaVIGKYJyKGeWLB1DAYjSLKdHUoaJ/qc/JcNQ6crAAAqN3k7bX8TVf8g7UekEiY+FPvYKJPRERE1IckEgGh/iqE+qswb2IIRFFExeUGnC7To6DEdNU/t0AHAHBVyBAZ7Gme7hMeoGJJT+o2JvpERERE/UgQBAT6uCPQxx13jAsCAFTrm1BYdq2kZ15RNQDARSbBiCC1eY7/iGGeULCkJ90iJvpEREREDubjqcQ0zwBMiw0AANTWG3C6rAaFpaabfLO+L8YnIiCVCAgPUJmn+0QGe8JdKXfw6MlZMdEnIiIicjJqdxckRPshIdpU0rOxuRVnLpiS/oLSGmQfLsXnP5RAADBM62Gq5R+qQVSwJzw9FI4dPDkNJvpERERETs5VIcPYET4YO8IHAGBoacO5i7UoaJ/q891PF/HV0TIAgL+Xq3mOf1SIBr6eSlb2GaKY6BMRERENMC5yKaJDvRAd6gXAVNKzpLLOPMf/aKEO/867CADwUikQ3T7VJypEgyAfNyb+QwQTfSIiIqIBTiY13bQ7IkiNxCmhMIoiyi/VmxP//JIrOHiqEgDg4SpHZLCnebpPiJ8HpBJW9hmMmOgTERERDTISQUCw1gPBWg/MnhAMURShq2k0T/U5XarHsdOXAJgW/Iocdq2k5/BAFeQyVvYZDJjoExEREQ1ygiDAz8sNfl5uuD3OVNLzytVm0xX/9rKe6d+eBXDt24Hx0X4I9nHFyCBPuCqYMg5E/FcjIiIiGoK8VApMGe2PKaP9AQB1jS3tJT1Nf3Z/fRpGowiJICDU38Ncyz8yRAMPV5b0HAiY6BMRERERPFzliI/UIj5Sa/pZ7Yofjl9AQWkNTpfW4OujF5B9uBQAMMzXvb2WvyeiQ7zgpWJJT2fERJ+IiIiIrLgqZIgd7o3Y4d4AgJZWI85drMXpMlMt/wMnK/DNsQsAAK1Giajg9pKeoRr4aVxZ2ccJMNEnIiIiopuSyyTmG3YXTgPajEaUVdWbb/A9XlSNnBMVAABPdxdz3+gQDYK07pAw8e93TPSJiIiIyG5SiQRhASqEBagwf1IIRFHExeoG8w2+BSU1OPxzFQDAXSlDZLBpqk9UiAZh/irIpCzp2deY6BMRERFRjwmCgCBfdwT5umNm/DCIoohqfZO5qk9BqR4/njGV9HSRSzAyqL2Wf4gGI4LUcJGzpGdvY6JPRERERL1OEAT4alzhq3HFbWMCAQD6umacLtObp/tkfncOIgCpRMDwQLV5uk/EME+4KZmm9hQjSERERET9wtNDgYkxfpgY4wcAaGhqwekyvfmq/xeHSrDn4HkIAhDiZyrp2XGTr9rdxcGjH3iY6BMRERGRQ7gp5RgX4YtxEb4AgOaWNpwtrzXX8v/2x3Lsyy0DAAT6uCEyuKOWvyd8PV0dOfQBgYk+ERERETkFhVyKUWFeGBXmBQBobTPifMXV9jn+ppt7vz1eDgDwUSvMU32iQjQI8HZjSc8bMNEnIiIiIqckk0owcpgnRg7zxC+nhsFoFFGmq2uv7KPHyeIrOHCyEgCgcpNbTPUJ8fOARDK0E38m+kREREQ0IEgkAkL9VQj1V2HuRFNJz6orjeabewtLa3CkQAcAcFVIETFMg6j21XvDA4deSU8m+kREREQ0IAmCAH9vN/h7u+GOcUEAgMu1Teakv7BMj4/2VwMwLfg1MshU2ScyRIOIIE8oXAZ3SU8m+kREREQ0aHirlZgaG4CpsQEAgNoGA06X6nG6zDTPP+v7YoiiqaRnWIDKPNUnMsQT7kq5g0ffu5joExEREdGgpXZzQUK0FgnRWgBAY3Mrii5cq+W/70gp9h4qgQBgmNbd4gZfjYfCsYPvIYcm+gaDAZs2bUJmZiZqa2sRExOD1atXY9q0aV1ul5eXh/T0dOTl5aGwsBAtLS0oKCiw6ldUVISPPvoIOTk5KCkpgbu7O2JjY/HHP/4RsbGxFn3Xrl2LjIwMq32MGzcOu3bt6tkLJSIiIiKn4KqQYcwIH4wZ4QMAaGm9rqRnmR45P1Xg66MXAAB+Xq7XbvAN1UDrqbSq7HPgZAXS9xfhcm0zvNUK3PuLkZjW/m2Cozk00V+7di2ys7ORnJyMsLAwZGRkYNWqVUhNTUV8fHyn2+3fvx9paWmIjo5GSEgIzp49a7Pf7t27sXv3bsyfPx8PPvggrl69ig8//BD33XcfUlJSMHXqVIv+rq6uWL9+vUWbt7d3z18oERERETkluUyK6FAvRIeaSnq2GY0oqawzz/M/VqjDd3kXAQBeKgUigz0R3X7F/3zlVWzfWwBDqxEAUF3bjH99/jMAOEWyL4iiKDriwHl5eVi6dCmeeeYZLF++HADQ3NyMpKQk+Pn54f333+9020uXLsHDwwNKpRIvvPACtm/fbvOK/okTJzB8+HC4u7ub265cuYIFCxYgIiICqamp5va1a9di3759yM3N7bXXWF1dB6Oxf8Or1aqg013t12MOZIyX/Rgz+zBe9mG87MN42Yfxsg/jZWIURZRfqsfp9lr+haU1qKkzAAAEALYyPR+1Ai//fnq/jE8iEeDj42HzOYdd0d+7dy/kcjmWLl1qblMoFFiyZAlef/11VFVVwc/Pz+a2vr6+t3SMMWPGWLV5eXlh4sSJOHLkiM1t2tra0NjYCA8P2wEjIiIioqFDIggI1nogWOuBWROCIYoidPomFJbUYOuefJvbVNc29/MobXNYMdH8/Hyrq+0AEBcXB1EUkZ9vO3C9QafTwcvLy6q9vr4eCQkJSEhIwJQpU7BhwwY0NzvHPxQREREROZ4gCPDTuGJGXCB81LZv1u2svb857Iq+TqeDv7+/VbtWa7ojuqqqqk+Om5ubix9//BGPP/641XFXrlyJUaNGwWg04ptvvsG2bdtQVFSEd955p1vH6uxrlL6m1aocctyBivGyH2NmH8bLPoyXfRgv+zBe9mG8urY8KRb/TDuO5pY2c5tCLsXypFiniJ3DEv2mpibI5da1ShUK0yegvriSXl1djaeeegqhoaFYsWKFxXNPPfWUxc9JSUnw9/dHSkoKcnJyMH26/fOsOEff+TFe9mPM7MN42Yfxsg/jZR/Gyz6M183FhmqQnBhtVXUnNlTTb7Fzyjn6SqUSLS0tVu0dCX5Hwt9bGhoa8Mgjj6CxsREpKSlwc3O76TYrVqxASkoKDhw40K1En4iIiIgGt2mxAZgWG+CUH4wcluhrtVqb03N0Oh0AdHojbncYDAb84Q9/QGFhIbZu3YqIiIhb2s7X1xdyuRx6vb7XxkJERERE1B8cdjNuTEwMzp07h/r6eov248ePm5/vDUajEWvWrMGBAwfw2muvYeLEibe8bUVFBVpaWlhLn4iIiIgGHIcl+omJiWhpaUFaWpq5zWAwID09HRMmTDDfqFteXo6ioqJuH+evf/0r9uzZg3Xr1mHu3Lk2+zQ3N6Ours6q/c033wQAzJgxo9vHJyIiIiJyBIdN3Rk3bhwSExPxyiuvQKfTITQ0FBkZGSgvL8eGDRvM/dasWYNDhw5ZLIh14cIFZGZmAgB++uknANeS8piYGMyePRsAsG3bNuzYsQPx8fFQKpXmbTosWrQIgGm60OLFi5GUlIQRI0aYq+4cOHAACxYswKRJk/ouEEREREREfcBhiT4AvPTSS9i4cSMyMzOh1+sRHR2NzZs3IyEhocvtysrKsGnTJou2jp8XL15sTvR//tm0BPGxY8dw7Ngxq/10JPpqtRozZ85ETk4OMjIyYDQaER4ejrVr1yI5ObnHr5OIiIiIqL8Joij2b/3HIYTlNZ0f42U/xsw+jJd9GC/7MF72Ybzsw3jZx1Hx6qq8psPm6BMRERERUd9hok9ERERENAgx0SciIiIiGoQcejPuYCeRCEPquAMV42U/xsw+jJd9GC/7MF72Ybzsw3jZxxHx6uqYvBmXiIiIiGgQ4tQdIiIiIqJBiIk+EREREdEgxESfiIiIiGgQYqJPRERERDQIMdEnIiIiIhqEmOgTEREREQ1CTPSJiIiIiAYhJvpERERERIMQE30iIiIiokGIiT4RERER0SAkc/QAyJLBYMCmTZuQmZmJ2tpaxMTEYPXq1Zg2bdpNt62srMSLL76InJwcGI1GTJ06Fc888wxCQkKs+qalpWHr1q0oKytDUFAQkpOTsWzZsr54SX2qu/HKzs7Gnj17kJeXh+rqagQGBmLWrFn4/e9/D5VKZdE3Ojra5j6ee+45PPDAA732WvpDd+P1xhtv4J///KdVu6+vL3Jycqzah/r5NXv2bFy4cMHmc2FhYcjOzjb/PJjOr6qqKmzfvh3Hjx/HiRMn0NDQgO3bt2PKlCm3tH1RURFefPFFHD16FHK5HLNmzcKaNWvg7e1t0c9oNCIlJQUffPABdDodwsPD8dhjj2HBggV98bL6THfjZTQakZGRgS+//BL5+fnQ6/UIDg5GUlISVqxYARcXF3PfsrIyzJkzx+Z+tmzZgjvuuKNXX1Nf6sn5tXbtWmRkZFi1jxs3Drt27bJoG+rnF9D5+xIA3HbbbXj33XcBDK7zKy8vDxkZGfjhhx9QXl4OjUaD+Ph4PPnkkwgLC7vp9s6agzHRdzJr165FdnY2kpOTERYWhoyMDKxatQqpqamIj4/vdLv6+nokJyejvr4ejz76KGQyGbZt24bk5GR8/PHH8PT0NPfduXMn1q1bh8TERDz88MPIzc3F888/j+bmZqxYsaI/Xmav6W68/uu//gt+fn5YtGgRgoKCUFBQgNTUVPz73//GRx99BIVCYdF/xowZuPvuuy3axo0b1yevqS91N14dnn/+eSiVSvPP1z/uwPML+Mtf/oL6+nqLtvLycmzcuBHTp0+36j9Yzq9z585hy5YtCAsLQ3R0NI4dO3bL21ZUVGDZsmVQq9VYvXo1GhoasHXrVhQWFmLXrl2Qy+Xmvq+//jo2b96M+++/H2PGjMFXX32F1atXQyKRIDExsS9eWp/obrwaGxvxl7/8BePHj8evf/1r+Pj44NixY9i0aRMOHjyIbdu2WW1z9913Y8aMGRZtMTExvfEy+k1Pzi8AcHV1xfr16y3abvwQCfD8AoCXXnrJqu3EiRPYvn27zfewwXB+vfPOOzh69CgSExMRHR0NnU6H999/H/fccw92796NkSNHdrqtU+dgIjmN48ePi1FRUeK7775rbmtqahLnzp0rPvjgg11uu3nzZjE6Olo8efKkue3MmTPiqFGjxI0bN5rbGhsbxcmTJ4uPPfaYxfZPPfWUGB8fL9bW1vbOi+kHPYnXwYMHrdoyMjLEqKgo8aOPPrJoj4qKEv/7v/+7V8bsSD2J1z/+8Q8xKipK1Ov1Xfbj+dW5//mf/xGjoqLEI0eOWLQPlvNLFEXx6tWr4uXLl0VRFMUvv/xSjIqKsvm7Zsu6devE8ePHixUVFea2nJwcMSoqSkxLSzO3VVRUiLGxsRYxMxqN4oMPPijOmjVLbGtr66VX0/e6G6/m5mar80gURfGNN96w2kdpaanVeTxQ9eT8WrNmjZiQkHDTfjy/OveXv/xFjI6OFi9evGhuG0zn15EjR8Tm5maLtnPnzoljxowR16xZ0+W2zpyDcY6+E9m7dy/kcjmWLl1qblMoFFiyZAmOHDmCqqqqTrf94osvMH78eIwePdrcNnLkSEybNg2ff/65ue2HH35ATU0NHnzwQYvtly1bhvr6enz77be9+Ir6Vk/iZeury7lz5wIwTR+wpampCc3NzT0cteP0JF4dRFFEXV0dRFG0+TzPr859+umnCA4OxoQJE2w+P9DPLwDw8PCAl5dXt7bNzs7G7Nmz4e/vb2677bbbEB4ebvEetm/fPrS0tFicY4Ig4IEHHsCFCxeQl5fX/RfQz7obLxcXF5vn0bx58wB0/h7W0NAAg8Fg9/GcRU/Orw5tbW2oq6vr9HmeX7YZDAZkZ2dj0qRJCAgIsNlnoJ9fEyZMsJj2BgDh4eGIjIzs9HeqgzPnYEz0nUh+fj6GDx8Od3d3i/a4uDiIooj8/Hyb2xmNRhQUFGDMmDFWz40dOxbFxcVobGwEAJw6dQoArPrGxsZCIpGYnx8Iuhuvzly6dAkAbL4x7t69G+PHj0dcXBzuuusufPnll90fuIP0RrxmzpyJhIQEJCQk4JlnnkFNTY3F8zy/bDt16hSKioqQlJRk8/nBcH71RGVlJaqrq22+h8XFxVnEOj8/Hx4eHhg+fLhVPwAD6hzrbV29h23atAnx8fGIi4vD/fffj8OHD/f38Byuvr7e/P41ZcoUbNiwwerDNc8v2/bv34/a2lqrKYYdBuv5JYoiLl261OUHJmfPwThH34nodDqLq1kdtFotAHR6BbGmpgYGg8Hc78ZtRVGETqdDaGgodDodXFxcoNFoLPp1tNl7ldKRuhuvzmzZsgVSqRTz58+3aI+Pj8eCBQsQHByMixcvYvv27Xj88cfx6quvdpq4OaOexEutVuO3v/0txo0bB6wKoBoAAA3pSURBVLlcjoMHD+LDDz/EqVOnkJaWZr4KwvPLtqysLACw+Z/kYDm/eqIjlp29h1VXV6OtrQ1SqRQ6nQ6+vr42+12/r6HonXfegUqlspgrLZFIMGPGDMybNw9+fn44f/48UlJS8PDDD2Pbtm2YOHGiA0fcf7RaLVauXIlRo0bBaDTim2++wbZt21BUVIR33nnH3I/nl21ZWVlwcXHBnXfeadE+2M+vTz75BJWVlVi9enWnfZw9B2Oi70Samposbjjr0HFjaGdf63e03/iV0/XbNjU1dXmMjr4DaepAd+NlS1ZWFnbv3o1HHnkEoaGhFs/t3LnT4ufFixcjKSkJL7/8MhYuXAhBELox+v7Xk3g99NBDFj8nJiYiMjISzz//PD7++GPcd999XR6j4zhD8fwyGo347LPPMHr0aJs3cw2W86snbvU9zN3dHU1NTV32G0jnWG9666238P333+P555+3qBwWFBSElJQUi74LFizAwoUL8corr1idf4PVU089ZfFzUlIS/P39kZKSgpycHPMNpjy/rNXV1eF///d/8Ytf/AJqtdriucF8fhUVFeH5559HQkICFi1a1Gk/Z8/BOHXHiSiVSrS0tFi1d/zD31gJpkNHu625cR3bdlRHUSqVnc6ha25u7vQYzqi78bpRbm4unn32WcycORNPPPHETfu7ubnh17/+NSoqKnD27Fn7Bu1AvRWvDg888ABcXV1x4MABi2Pw/LJ06NAhVFZW4q677rql/gP1/OqJ3ngP6+55PBjs2bMHGzduxP3334/777//pv39/f2xcOFCHD9+3DylYCjqqHByK+9hQ/n8+uKLL9Dc3HzL72GD4fzS6XR45JFH4OnpiU2bNkEi6TxddvYcjIm+E9FqtTa/ttHpdAAAPz8/m9tpNBq4uLiY+924rSAI5q+UtFotWlparOZWGwwG1NTUdHoMZ9TdeF3v559/xmOPPYbo6Gi8/vrrkEqlt3TswMBAAIBer7djxI7VG/G6nkQigb+/v0UMeH5Zy8rKgkQiwcKFC2/52APx/OqJjlh29h7m4+Nj/t3UarXmueg39rt+X0NFTk4O/uM//gOzZs3CunXrbnm7wMBAGI1G1NbW9uHonJuvry/kcrnVexjPL0tZWVlQqVSYNWvWLW8zkM+vq1evYtWqVbh69Sreeecdm1NyrufsORgTfScSExODc+fOWdXfPn78uPl5WyQSCaKionDixAmr5/Ly8hAWFgZXV1cAwKhRowDAqu+JEydgNBrNzw8E3Y1Xh5KSEqxcuRLe3t54++234ebmdsvHLi0tBWC7BrOz6mm8btTS0oKLFy9a3KTE88tSR6WKyZMn25zv35mBeH71hL+/P7y9vTt9D7v+vBk1ahTq6upw7tw5i34d/y4D6RzrqePHj+Pxxx/H2LFj7bpQAZjOMalUalHfe6ipqKhAS0uLxe8Zzy9LVVVV+OGHHzB//nybU1M6M1DPr+bmZjz66KMoLi7G22+/jREjRtx0G2fPwZjoO5HExES0tLQgLS3N3GYwGJCeno4JEyaYE4Xy8nKrUk933nknfvzxR4s7ts+ePYuDBw9aLPAxdepUaDQa7Nixw2L7Dz74AG5ubgNqFbuexEun02HFihUQBAEpKSmdJlSXL1+2arty5Qp27NiB4OBghIeH994L6mM9iZetOKSkpKC5uRm33367uY3nl6WOShWdfeU9mM4ve5SUlKCkpMSibf78+fj6669RWVlpbjtw4ACKi4st3sPmzJkDuVxucY6JooidO3ciKChoQC40djO24lVUVITf/e53GDZsGN566y2bi9cBts+x8+fP47PPPsPEiRM73W4guzFezc3NNktqvvnmmwBgcfMyzy9Le/bsgdFotOs9bKCeX21tbXjyySfx448/YtOmTRg/frzNfgMtB/t/7d1bSBTtHwfwr1oWFRKaUpmWvpWiLipiHkrzkGBimBBI6m7ZapZpBzEsrIuOChpFmxfmgQgi7Yi2F5anMN0wtJDIzEzJXSwVTS0UO6z/i3Bo/2u94qul0/dzN8/8dnaeh3H9MvPMDG/GnUGcnZ0RHByMrKws4Q7tu3fvorOzE+np6UJdamoqnjx5glevXgltkZGRuHnzJnbv3o2YmBgYGRnhypUrMDc3x86dO4W6+fPnY//+/Th58iQOHDiADRs2oL6+HiUlJUhJSdG70WYm+y/jFRsbC7VajdjYWDQ0NKChoUFYZ21tLbz19Nq1a6ioqICfnx+WL1+Orq4uFBUVoa+vD9nZ2b+vs1Pgv4yXv78/QkJCsHbtWhgbG6Ourg7379+Hm5ubzpNheHzp+tmTKsaI6fgaMxaexv4RFhcXo6GhASYmJoiOjgYA4TepsrJS+NyePXtQWloKmUyG6OhoDA0NIT8/H/b29jo3wi1duhQymQwFBQUYGRmBRCJBeXk56uvrcf78+V/OpZ2JJjNenz59glwux+DgIORyOR4+fKizTTs7O+GKU2ZmJtRqNTw9PWFhYYGOjg7hBsnU1NTp7t6Um8x49fT0CDe529raCk/defz4MUJCQuDu7i5sn8eXrpKSElhYWIz77hlAXMdXRkYGKisr4e/vj/7+fhQXFwvrFi5cKLxrZ7ZlMIPRn735hv6IkZERXLhwAffu3cPAwADs7OyQnJwMb29voUYqlY4bLN6/f4+zZ8+itrYWWq0WHh4eSEtLg5WVld733LhxAwUFBdBoNFi2bBmkUilkMtm092+qTXa87OzsfrrN8PBwZGRkAABqamqQn5+PlpYWDAwMYMGCBXBxcUF8fDzc3Nymr2PTZLLjdezYMTx9+hTv3r3Dly9fYGlpiZCQEMTHx497xuZvP76A72HM29sbGzduhEKhGHf7Yju+gJ//bVlaWgpBIiAgAIB+sHj9+jUyMjLQ0NCAuXPnws/PD0ePHtW74qbVapGbm4uioiJ0d3fDxsYG8fHxs/JxpJMZL41Gg8DAwJ9uMzExEUlJSQC+v6itsLAQra2t+PjxI0xMTLBu3TokJiZizZo1U9mV32Iy4zU4OIhTp06hsbER3d3d0Gq1WLVqFcLDwyGTyfSmPP3tx9eYtrY2bN68GTExMThy5Mi42xHT8TX2Wz6eH8drtmUwBn0iIiIiIhGaXdegiIiIiIhoQhj0iYiIiIhEiEGfiIiIiEiEGPSJiIiIiESIQZ+IiIiISIQY9ImIiIiIRIhBn4iIiIhIhBj0iYhIVKRSqfASICKiv9mcP70DREQ089XV1f3yzY1GRkZoamr6jXtERET/hkGfiIgmLDQ0FL6+vnrthoa8QExENNMw6BMR0YQ5ODggLCzsT+8GERFNAE/BEBHRlNFoNLCzs4NCoYBSqcSWLVsgkUjg5+cHhUKBr1+/6n2mubkZ+/btg4eHByQSCUJCQpCbm4tv377p1fb09OD06dMIDAyEk5MTvLy8EBMTg9raWr3arq4uJCcnw93dHc7OzpDL5Whvb5+WfhMRzUQ8o09ERBM2PDyMvr4+vXZjY2MsWrRIWK6srIRarUZUVBSWLFmCyspKXLp0CZ2dnUhPTxfqnj9/DqlUijlz5gi1VVVVyMrKQnNzM86dOyfUajQabN++Hb29vQgLC4OTkxOGh4fR2NgIlUqF9evXC7VDQ0OIjo6Gs7MzDh06BI1Gg6tXryIhIQFKpRJGRkbTNEJERDMHgz4REU2YQqGAQqHQa/fz80NOTo6w3NzcjFu3bsHR0REAEB0djcTERNy5cwcRERFwcXEBAJw5cwafP39GYWEh7O3thdqDBw9CqVRi27Zt8PLyAgCcOHEC3d3dyMvLg4+Pj873a7VaneUPHz5ALpcjLi5OaDM1NUVmZiZUKpXe54mIxIhBn4iIJiwiIgLBwcF67aampjrL3t7eQsgHAAMDA8TGxqK8vBxlZWVwcXFBb28vnj17hqCgICHkj9Xu3bsXpaWlKCsrg5eXF/r7+/Ho0SP4+PiMG9L//2ZgQ0NDvacEeXp6AgDevn3LoE9EfwUGfSIimrCVK1fC29v7X+v++ecfvbbVq1cDANRqNYDvU3F+bP+Rra0tDA0NhdqOjg6Mjo7CwcFhQvtpYWGBefPm6bQtXrwYANDf3z+hbRARzXa8GZeIiETnV3PwR0dHf+OeEBH9OQz6REQ05d68eaPX1traCgCwsrICAKxYsUKn/UdtbW3QarVCrbW1NQwMDPDy5cvp2mUiItFh0CcioimnUqnw4sULYXl0dBR5eXkAgE2bNgEAzMzM4OrqiqqqKrS0tOjUXr58GQAQFBQE4Pu0G19fX1RXV0OlUul9H8/SExHp4xx9IiKasKamJhQXF4+7bizAA4C9vT127NiBqKgomJubo6KiAiqVCmFhYXB1dRXq0tLSIJVKERUVhcjISJibm6Oqqgo1NTUIDQ0VnrgDAMePH0dTUxPi4uKwdetWODo6YmRkBI2NjbC0tMThw4enr+NERLMQgz4REU2YUqmEUqkcd92DBw+EufEBAQGwsbFBTk4O2tvbYWZmhoSEBCQkJOh8RiKRoLCwEBcvXsT169cxNDQEKysrpKSkYNeuXTq1VlZWuH37NrKzs1FdXY3i4mKYmJjA3t4eERER09NhIqJZzGCU1zuJiGiKaDQaBAYGIjExEUlJSX96d4iI/mqco09EREREJEIM+kREREREIsSgT0REREQkQpyjT0REREQkQjyjT0REREQkQgz6REREREQixKBPRERERCRCDPpERERERCLEoE9EREREJEIM+kREREREIvQ/Tys5srg3eQAAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "\n",
        "\n",
        "# Use plot styling from seaborn.\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
        "\n",
        "# Plot the learning curve.\n",
        "plt.plot(loss_values, 'b-o', label=\"training loss\")\n",
        "plt.plot(validation_loss_values, 'r-o', label=\"validation loss\")\n",
        "\n",
        "# Label the plot.\n",
        "plt.title(\"Learning curve\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2KwddVzd0Adz"
      },
      "source": [
        "# Apply the model to a new sentence"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sKI9Xsfcg5VI"
      },
      "outputs": [],
      "source": [
        "test_sentence = \"\"\"\n",
        "Kedir is a student at ForthBrain. \n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TOgesTqB08Ei"
      },
      "outputs": [],
      "source": [
        "# first toknize the sentences\n",
        "tokenized_sentence = tokenizer.encode(test_sentence)\n",
        "input_ids = torch.tensor([tokenized_sentence]).cuda()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RSaFUFWN1JX-"
      },
      "outputs": [],
      "source": [
        "# run the sentences through the model\n",
        "with torch.no_grad():\n",
        "    output = model(input_ids)\n",
        "label_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ehUN42F71bDi"
      },
      "outputs": [],
      "source": [
        "# join bpe split tokens\n",
        "tokens = tokenizer.convert_ids_to_tokens(input_ids.to('cpu').numpy()[0])\n",
        "new_tokens, new_labels = [], []\n",
        "for token, label_idx in zip(tokens, label_indices[0]):\n",
        "    if token.startswith(\"##\"):\n",
        "        new_tokens[-1] = new_tokens[-1] + token[2:]\n",
        "    else:\n",
        "        new_labels.append(tag_values[label_idx])\n",
        "        new_tokens.append(token)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oc1g90-N1h_7",
        "outputId": "8ed012e6-57af-4861-c469-ab9d9903592e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "O\t[CLS]\n",
            "B-per\tkedir\n",
            "O\tis\n",
            "O\ta\n",
            "O\tstudent\n",
            "O\tat\n",
            "B-tim\tforthbrain\n",
            "O\t.\n",
            "O\t[SEP]\n"
          ]
        }
      ],
      "source": [
        "for token, label in zip(new_tokens, new_labels):\n",
        "    print(\"{}\\t{}\".format(label, token))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PxzDozoymJz"
      },
      "source": [
        "# Save Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "btjo7EzVy1gH"
      },
      "outputs": [],
      "source": [
        "model_path = '/content/model.pt'"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "torch.save(model, model_path)"
      ],
      "metadata": {
        "id": "NjNpiPDiSWjw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "filepath = '/content/model_state/model'"
      ],
      "metadata": {
        "id": "6h8zg3IxgIkL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# torch.save(model.state_dict(), filepath)\n",
        "\n",
        "# # #Later to restore:\n",
        "# model.load_state_dict(torch.load(filepath))\n",
        "# model.eval()"
      ],
      "metadata": {
        "id": "aZR7KKiOgIn-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RaxOcaDay-gw"
      },
      "source": [
        "# Load Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GkPf2lRvzDYf"
      },
      "outputs": [],
      "source": [
        "model_new = torch.load(model_path)\n",
        "model_predict = model_new.eval()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "duKmf635UGfi"
      },
      "outputs": [],
      "source": [
        "test_sentence = \"\"\"\n",
        "Kedir Ahmed is a student at forthbrain. \n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "43Fk7VMJUGfk"
      },
      "outputs": [],
      "source": [
        "# first toknize the sentences\n",
        "tokenized_sentence = tokenizer.encode(test_sentence)\n",
        "input_ids = torch.tensor([tokenized_sentence]).cuda()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w0AetuW_P6Dp"
      },
      "outputs": [],
      "source": [
        "# run the sentences through the model\n",
        "with torch.no_grad():\n",
        "    output = model_predict(input_ids)\n",
        "label_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OVkugdlmP6Dr"
      },
      "outputs": [],
      "source": [
        "# join bpe split tokens\n",
        "tokens = tokenizer.convert_ids_to_tokens(input_ids.to('cpu').numpy()[0])\n",
        "new_tokens, new_labels = [], []\n",
        "for token, label_idx in zip(tokens, label_indices[0]):\n",
        "    if token.startswith(\"##\"):\n",
        "        new_tokens[-1] = new_tokens[-1] + token[2:]\n",
        "    else:\n",
        "        new_labels.append(tag_values[label_idx])\n",
        "        new_tokens.append(token)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "14193b00-e606-461a-abf1-87b3de54e990",
        "id": "23KV6lBvP6Ds"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "O\t[CLS]\n",
            "B-per\tkedir\n",
            "I-per\tahmed\n",
            "O\tis\n",
            "O\ta\n",
            "O\tstudent\n",
            "O\tat\n",
            "B-org\tforthbrain\n",
            "O\t.\n",
            "O\t[SEP]\n"
          ]
        }
      ],
      "source": [
        "for token, label in zip(new_tokens, new_labels):\n",
        "    print(\"{}\\t{}\".format(label, token))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lw1wqV8C95xi"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "## Acknowledgements\n",
        "\n",
        "- The code for NER model is adapted from this online tutorial [Named Entity Recognation with bert ](https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "machine_shape": "hm",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "37f8492753364534bdfd6cce7d3b9879": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2a9a4eec9e894e2c820606c2f19ab7fc",
              "IPY_MODEL_52f4974caffb47e8bf2f1d7a2531ed75",
              "IPY_MODEL_040f040bb3204454ab2e50765f76addf"
            ],
            "layout": "IPY_MODEL_41a19caf7f834097b081f42eb03a9b6d"
          }
        },
        "2a9a4eec9e894e2c820606c2f19ab7fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_076ec77a50cb4495a22b678fa87578cf",
            "placeholder": "​",
            "style": "IPY_MODEL_be47d8bf7c354111a852e2a1aeea2bb3",
            "value": "Downloading: 100%"
          }
        },
        "52f4974caffb47e8bf2f1d7a2531ed75": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d84dce6797924e648eb15756abb32972",
            "max": 213450,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0625d3b1a61746f68a6784e4760a1369",
            "value": 213450
          }
        },
        "040f040bb3204454ab2e50765f76addf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8c37d9b2f9684048890691d38597ed98",
            "placeholder": "​",
            "style": "IPY_MODEL_1106a61ccbb341349f64aaf5b7993edd",
            "value": " 213k/213k [00:00&lt;00:00, 255kB/s]"
          }
        },
        "41a19caf7f834097b081f42eb03a9b6d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "076ec77a50cb4495a22b678fa87578cf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "be47d8bf7c354111a852e2a1aeea2bb3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d84dce6797924e648eb15756abb32972": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0625d3b1a61746f68a6784e4760a1369": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8c37d9b2f9684048890691d38597ed98": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1106a61ccbb341349f64aaf5b7993edd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ddb2c5db384b4d2e93a11e97059e162b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_2f640d4012de4902b8f7f9f2784f1f25",
              "IPY_MODEL_4abff02aca52432b988e4ee00195903f",
              "IPY_MODEL_690cb7b841994afabfc1b6611af13884"
            ],
            "layout": "IPY_MODEL_c37df6ff54894fe68d233b1bc549c16f"
          }
        },
        "2f640d4012de4902b8f7f9f2784f1f25": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b08b0d7849a47c082cb5fdef9572cef",
            "placeholder": "​",
            "style": "IPY_MODEL_476328dc592344cea67a9f3896aa5846",
            "value": "Downloading: 100%"
          }
        },
        "4abff02aca52432b988e4ee00195903f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a4d64fddc53b4c878626a5c0e61a74b5",
            "max": 433,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c21323df97e44d0489dde003a93df4be",
            "value": 433
          }
        },
        "690cb7b841994afabfc1b6611af13884": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b99ccd1a1b904fe7a92e41e6eb42ca2f",
            "placeholder": "​",
            "style": "IPY_MODEL_6dacafdcd3444641be139401dd4d308f",
            "value": " 433/433 [00:00&lt;00:00, 16.4kB/s]"
          }
        },
        "c37df6ff54894fe68d233b1bc549c16f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b08b0d7849a47c082cb5fdef9572cef": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "476328dc592344cea67a9f3896aa5846": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a4d64fddc53b4c878626a5c0e61a74b5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c21323df97e44d0489dde003a93df4be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b99ccd1a1b904fe7a92e41e6eb42ca2f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6dacafdcd3444641be139401dd4d308f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "71c9b23be1d64c5dbe45805009ef4c9b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bd18553e30ba43319f383bfbfabedd34",
              "IPY_MODEL_e02267a4c81a43ef9e32ade86ce9edfb",
              "IPY_MODEL_0e8ccbc5076942beb9ca9c007df141a2"
            ],
            "layout": "IPY_MODEL_ddb9642fe2d74c2eb1605d31aa480910"
          }
        },
        "bd18553e30ba43319f383bfbfabedd34": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6b93d34ecefd45688797b366f3ed1ebb",
            "placeholder": "​",
            "style": "IPY_MODEL_88ca5ad0d4384d3b9eead27b4045ad93",
            "value": "Downloading: 100%"
          }
        },
        "e02267a4c81a43ef9e32ade86ce9edfb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e439eb19190e41da87abc4daaae6797c",
            "max": 435779157,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d9ac365ce8964a1f95baa28716aee951",
            "value": 435779157
          }
        },
        "0e8ccbc5076942beb9ca9c007df141a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b0f17c04b2e0458e911f6d6dfb1dec6d",
            "placeholder": "​",
            "style": "IPY_MODEL_e712b244d60743ffa67df4e7e113de78",
            "value": " 436M/436M [00:29&lt;00:00, 15.9MB/s]"
          }
        },
        "ddb9642fe2d74c2eb1605d31aa480910": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6b93d34ecefd45688797b366f3ed1ebb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "88ca5ad0d4384d3b9eead27b4045ad93": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e439eb19190e41da87abc4daaae6797c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d9ac365ce8964a1f95baa28716aee951": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b0f17c04b2e0458e911f6d6dfb1dec6d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e712b244d60743ffa67df4e7e113de78": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}