en-nya/jw-300-baseline/en_nya_starter_notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Copy of efi_en_starter_notebook.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "61acdb19-5c3b-4937-beb9-2f5f6ebed4c1"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"nya\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead! \n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "g_drive_path = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)\n",
        "os.environ[\"gdrive_path\"] = g_drive_path\n",
        "models_path = '%s/models/%s%s_transformer'% (g_drive_path, source_language, target_language)\n",
        "# model temporary directory for training\n",
        "model_temp_dir = \"/content/drive/My Drive/masakhane/model-temp\"\n",
        "# model permanent storage on the drive\n",
        "!mkdir -p \"$gdrive_path/models/${src}${tgt}_transformer/\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "61688e5e-5c17-4baa-8854-958ae4f04c71"
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-nya-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "925edfb6-3c75-4601-ae6c-b38e1c50941e"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# Install opus-tools\n",
        "! pip install opustools-pkg "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\r\u001b[K     |████                            | 10kB 29.9MB/s eta 0:00:01\r\u001b[K     |████████                        | 20kB 6.3MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 30kB 7.6MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 40kB 8.1MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 51kB 7.3MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 61kB 8.3MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 71kB 8.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 81kB 5.6MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "5dbe00c9-1177-44c9-9fbb-fc282c40a960"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-nya.xml.gz not found. The following files are available for downloading:\n",
            "\n",
            "        ./JW300_latest_xml_en.zip already exists\n",
            "        ./JW300_latest_xml_nya.zip already exists\n",
            " 572 KB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-nya.xml.gz\n",
            "\n",
            " 572 KB Total size\n",
            "./JW300_latest_xml_en-nya.xml.gz ... 100% of 572 KB\n",
            "gzip: JW300_latest_xml_en-nya.xml already exists; do you wish to overwrite (y or n)? n\n",
            "\tnot overwritten\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j2K6QK2NOaUX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f10dd54e-7fb9-44a7-8c36-38e2a1db1555"
      },
      "source": [
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$tgt-$src.xml.gz"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "gzip: JW300_latest_xml_nya-en.xml.gz: No such file or directory\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n48GDRnP8y2G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        },
        "outputId": "32880d12-76cb-446d-8b95-112a5877508c"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-07-12 20:08:28--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en.1’\n",
            "\n",
            "\rtest.en-any.en.1      0%[                    ]       0  --.-KB/s               \rtest.en-any.en.1    100%[===================>] 271.28K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2020-07-12 20:08:28 (11.8 MB/s) - ‘test.en-any.en.1’ saved [277791/277791]\n",
            "\n",
            "--2020-07-12 20:08:30--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-nya.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 203330 (199K) [text/plain]\n",
            "Saving to: ‘test.en-nya.en’\n",
            "\n",
            "test.en-nya.en      100%[===================>] 198.56K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2020-07-12 20:08:30 (13.0 MB/s) - ‘test.en-nya.en’ saved [203330/203330]\n",
            "\n",
            "--2020-07-12 20:08:32--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-nya.nya\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 226404 (221K) [text/plain]\n",
            "Saving to: ‘test.en-nya.nya’\n",
            "\n",
            "test.en-nya.nya     100%[===================>] 221.10K  --.-KB/s    in 0.02s   \n",
            "\n",
            "2020-07-12 20:08:33 (11.1 MB/s) - ‘test.en-nya.nya’ saved [226404/226404]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqDG-CI28y2L",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5265c307-2d30-4133-efb9-e968d324db2d"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 376
        },
        "outputId": "4df3192b-b6be-43a8-af23-21e65a3e3b10"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 4429/60566 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>This publication is not for sale .</td>\n",
              "      <td>Magazini ino si yogulitsa .</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td></td>\n",
              "      <td>Colinga cake ni kuthandiza pa nchito yophunzit...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3 Finding the Way</td>\n",
              "      <td>3 Mmene Mungaipezele</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4 Contentment and Generosity</td>\n",
              "      <td>4 Kukhutila Komanso Kupatsa</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>6 Physical Health and Resilience</td>\n",
              "      <td>6 Thanzi Labwino na Kupilila</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>8 Love</td>\n",
              "      <td>8 Cikondi</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>10 Forgiveness</td>\n",
              "      <td>10 Kukhululuka</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>12 Purpose in Life</td>\n",
              "      <td>12 Colinga ca Moyo</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>14 Hope</td>\n",
              "      <td>14 Ciyembekezo</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>16 Learn More</td>\n",
              "      <td>16 Dziŵani Zambili</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                      source_sentence                                    target_sentence\n",
              "0  This publication is not for sale .                        Magazini ino si yogulitsa .\n",
              "1                                      Colinga cake ni kuthandiza pa nchito yophunzit...\n",
              "2                   3 Finding the Way                               3 Mmene Mungaipezele\n",
              "3        4 Contentment and Generosity                        4 Kukhutila Komanso Kupatsa\n",
              "4    6 Physical Health and Resilience                       6 Thanzi Labwino na Kupilila\n",
              "5                              8 Love                                          8 Cikondi\n",
              "6                      10 Forgiveness                                     10 Kukhululuka\n",
              "7                  12 Purpose in Life                                 12 Colinga ca Moyo\n",
              "8                             14 Hope                                     14 Ciyembekezo\n",
              "9                       16 Learn More                                 16 Dziŵani Zambili"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "2d141bdf-56a8-4f62-d44e-4fc4371f7064"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  if __name__ == '__main__':\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_1BwAApEtMk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d7ad4abd-1d0e-4591-c155-0879765269a7"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan\n",
        "\n",
        "# NOTE - This might run slow depending on the size of your training set. We are\n",
        "# printing some information to help you track how long it would take. \n",
        "scores = []\n",
        "start_time = time.time()\n",
        "for idx, row in df_pp.iterrows():\n",
        "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
        "  if idx % 1000 == 0:\n",
        "    hours, rem = divmod(time.time() - start_time, 3600)\n",
        "    minutes, seconds = divmod(rem, 60)\n",
        "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp['scores'] = scores\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading https://files.pythonhosted.org/packages/43/ff/74f23998ad2f93b945c0309f825be92e04e0348e062026998b5eefef4c33/fuzzywuzzy-0.18.0-py2.py3-none-any.whl\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.18.0\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 4.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (47.3.1)\n",
            "Building wheels for collected packages: python-Levenshtein\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144805 sha256=e7a3cd1d3b3a56457aa3239d36cb86516c41ead306b3503e9238370e3113acb6\n",
            "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
            "Successfully built python-Levenshtein\n",
            "Installing collected packages: python-Levenshtein\n",
            "Successfully installed python-Levenshtein-0.12.0\n",
            "00:00:00.02 0.00 percent complete\n",
            "00:00:19.79 1.96 percent complete\n",
            "00:00:39.92 3.92 percent complete\n",
            "00:00:59.16 5.88 percent complete\n",
            "00:01:18.63 7.84 percent complete\n",
            "00:01:38.33 9.79 percent complete\n",
            "00:01:57.66 11.75 percent complete\n",
            "00:02:17.39 13.71 percent complete\n",
            "00:02:36.66 15.67 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:02:56.46 17.63 percent complete\n",
            "00:03:16.09 19.59 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:03:35.71 21.55 percent complete\n",
            "00:03:54.99 23.51 percent complete\n",
            "00:04:14.15 25.47 percent complete\n",
            "00:04:33.53 27.42 percent complete\n",
            "00:04:52.75 29.38 percent complete\n",
            "00:05:11.77 31.34 percent complete\n",
            "00:05:30.92 33.30 percent complete\n",
            "00:05:50.44 35.26 percent complete\n",
            "00:06:09.47 37.22 percent complete\n",
            "00:06:29.46 39.18 percent complete\n",
            "00:06:48.79 41.14 percent complete\n",
            "00:07:08.00 43.10 percent complete\n",
            "00:07:27.52 45.05 percent complete\n",
            "00:07:46.96 47.01 percent complete\n",
            "00:08:06.48 48.97 percent complete\n",
            "00:08:26.28 50.93 percent complete\n",
            "00:08:45.30 52.89 percent complete\n",
            "00:09:04.50 54.85 percent complete\n",
            "00:09:23.50 56.81 percent complete\n",
            "00:09:42.81 58.77 percent complete\n",
            "00:10:02.45 60.73 percent complete\n",
            "00:10:21.29 62.68 percent complete\n",
            "00:10:40.70 64.64 percent complete\n",
            "00:10:59.94 66.60 percent complete\n",
            "00:11:19.03 68.56 percent complete\n",
            "00:11:38.52 70.52 percent complete\n",
            "00:11:57.42 72.48 percent complete\n",
            "00:12:16.58 74.44 percent complete\n",
            "00:12:35.66 76.40 percent complete\n",
            "00:12:55.66 78.36 percent complete\n",
            "00:13:14.89 80.31 percent complete\n",
            "00:13:34.63 82.27 percent complete\n",
            "00:13:53.54 84.23 percent complete\n",
            "00:14:12.79 86.19 percent complete\n",
            "00:14:31.36 88.15 percent complete\n",
            "00:14:50.79 90.11 percent complete\n",
            "00:15:10.14 92.07 percent complete\n",
            "00:15:29.67 94.03 percent complete\n",
            "00:15:49.44 95.99 percent complete\n",
            "00:16:09.30 97.95 percent complete\n",
            "00:16:28.20 99.90 percent complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 799
        },
        "outputId": "2fdcb050-df32-4e1e-884f-ae110a3085b7"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.en <==\n",
            "How Job came to know Jehovah .\n",
            "Imagine Jacob gently wiping the tears from Joseph’s eyes , comforting him with the same hope that had once comforted Jacob’s grandfather Abraham .\n",
            "What gathering work is referred to at Matthew 24 : 31 ?\n",
            "( b ) Why is that ancient event significant for us ?\n",
            "Consider what is involved in making that amazing feat possible .\n",
            "Neither are we required to adopt a special posture .\n",
            "Or what if you lose your job , and you are having difficulty finding another ?\n",
            "The Bible explains : “ By the trespass of the one man [ Adam ] death ruled as king ” over Adam’s descendants .\n",
            "The next article will address this concern .\n",
            "However , more is involved than simply telling your children what is right and what is wrong .\n",
            "\n",
            "==> train.nya <==\n",
            "Mmene Yobu anadziŵila Yehova .\n",
            "Yelekezani kuti mukuona Yakobo atate ake a Yosefe akumupukuta misozi mokoma mtima , kenako akumutonthoza pomulimbikitsa kukhala ndi ciyembekezo cimene cinatonthoza agogo ake a Abulahamu .\n",
            "Ndi nchito yosonkhanitsa iti imene ikuchulidwa pa Mateyu 24 : 31 ?\n",
            "( b ) N’cifukwa ciani cocitika cakale cimeneci ndi nkhani yaikulu kwa ife ?\n",
            "Onani nchito imene imakhalapo kuti Baibulo lizipezeka m’zinenelo zambili .\n",
            "Safunanso kuti popemphela tizitsatila kakhalidwe ka thupi kapadela .\n",
            "Nanga bwanji ngati nchito imene munali kugwila inatha ndipo simukupezanso ina ?\n",
            "Baibulo imati : “ Cifukwa ca ucimo wa munthu mmodziyo [ Adamu ] imfa inalamulila monga mfumu ” kwa mbadwa za Adamu .\n",
            "Nkhani yotsatila idzafotokoza mmene tingacitile zimenezi .\n",
            "Komabe , kuuza ana anu kuti ici n’cabwino ici n’coipa , pakokha si kokwanila .\n",
            "==> dev.en <==\n",
            "Our professional ballet careers took us around the world to dance\n",
            "He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "Spread the Good News of Undeserved Kindness , July\n",
            "Now , do you think that it bothers Satan that he and his invisible cohorts have been relegated to the world of folklore ?\n",
            "To some people , it is nothing less than divine approval of revenge .\n",
            "You could add , “ Actually , that passage says much more . ”\n",
            "When we first arrived in Pine Bluff , we moved in with the brother who was the congregation servant at the time .\n",
            "Without their loving support , I could never have served where the need is greater . ” Simon\n",
            "But , instead , my employer thanked me for my good work .\n",
            "\n",
            "==> dev.nya <==\n",
            "Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "Koma kodi muganiza kuti Satana ndi ziwanda zake amakhumudwa ndi zimene anthu amanena zoti iwo ndi ongopeka ?\n",
            "( Ekisodo 21 : 24 ) Anthu ena amakhulupilila kuti mau amenewo akuonetsa kuti Mulungu amavomeleza kubwezela .\n",
            "Mwina mungakambenso kuti , “ Lembali lili ndi zambili . ”\n",
            "Pamene tinafika ku Pine Bluff , tinayamba kukhala ku nyumba kwa m’bale amene anali mtumiki wa mpingo pa nthawiyo .\n",
            "Popanda thandizo lao la cikondi , sindikanakwanitsa kutumikila kumalo osoŵa . ”\n",
            "Koma iwo ananiyamikila cifukwa cogwila bwino nchito .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "4c788f60-64f6-4c7e-a012-51ec529fac42"
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 2467, done.\u001b[K\n",
            "remote: Total 2467 (delta 0), reused 0 (delta 0), pack-reused 2467\u001b[K\n",
            "Receiving objects: 100% (2467/2467), 2.64 MiB | 4.34 MiB/s, done.\n",
            "Resolving deltas: 100% (1725/1725), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (7.0.0)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.18.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (47.3.1)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.5.1+cu101)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (2.2.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/66/5b/cf661da8e9b0229f5d98c2961b072a5728fd11a0758957f8c0fd36081c06/sacrebleu-1.4.12-py3-none-any.whl (54kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 4.5MB/s \n",
            "\u001b[?25hCollecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.2.2)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.10.1)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)\n",
            "\u001b[K     |████████████████████████████████| 276kB 9.7MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e8/fb/734960c55474c8f74e6ad4c8588fc44073fb9d69e223269d26a3c2435d16/pylint-2.5.3-py3-none-any.whl (324kB)\n",
            "\u001b[K     |████████████████████████████████| 327kB 19.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Collecting wrapt==1.11.1\n",
            "  Downloading https://files.pythonhosted.org/packages/67/b2/0f71ca90b0ade7fad27e3d20327c996c6252a2ffe88f50a95bba7434eda9/wrapt-1.11.1.tar.gz\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.30.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.0)\n",
            "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.4.1)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.2)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.2.1)\n",
            "Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.2.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.34.2)\n",
            "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.6.3)\n",
            "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.3.3)\n",
            "Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.2.2)\n",
            "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (2.10.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.23.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.41.1)\n",
            "Collecting mecab-python3==0.996.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/18/49/b55a839a77189042960bf96490640c44816073f917d489acbc5d79fa5cc3/mecab_python3-0.996.5-cp36-cp36m-manylinux2010_x86_64.whl (17.1MB)\n",
            "\u001b[K     |████████████████████████████████| 17.1MB 205kB/s \n",
            "\u001b[?25hCollecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/53/84/7b3146ec6378d28abc73ab484f09f47dfa008ad6f03f33d90a369f880e25/portalocker-1.7.0-py2.py3-none-any.whl\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.2.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.8.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.7)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: pandas>=0.22.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.0.5)\n",
            "Collecting astroid<=2.5,>=2.4.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/24/a8/5133f51967fb21e46ee50831c3f5dda49e976b7f915408d670b1603d41d6/astroid-2.4.2-py3-none-any.whl (213kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 57.1MB/s \n",
            "\u001b[?25hCollecting toml>=0.7.1\n",
            "  Downloading https://files.pythonhosted.org/packages/9f/e1/1b40b80f2e1663a6b9f497123c11d7d988c0919abbf3c3f2688e448c5363/toml-0.10.1-py2.py3-none-any.whl\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 8.6MB/s \n",
            "\u001b[?25hCollecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.17.2)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.2.2)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.6.0.post3)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.1)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.9)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2020.6.20)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.22.0->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 10.3MB/s \n",
            "\u001b[?25hCollecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 57.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (4.6)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.2.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (4.1.0)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.6.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (1.3.0)\n",
            "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4; python_version >= \"3\"->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (0.4.8)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Building wheels for collected packages: joeynmt, pyyaml, wrapt\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=77293 sha256=3c8fae0f346f99b8904b109c79858a5b46d70c5f4a22b0f98cafbd4f266a220e\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-tty7q62f/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44621 sha256=dbad71b916be69235a8fee7be2bbb6b3bd66580496180dc64d349a9d2a140ca7\n",
            "  Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd\n",
            "  Building wheel for wrapt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wrapt: filename=wrapt-1.11.1-cp36-cp36m-linux_x86_64.whl size=67430 sha256=f843d38d6a7a3933324cc1b0272e730a5b15ac748347d4cf7fc5eb5660fb1139\n",
            "  Stored in directory: /root/.cache/pip/wheels/89/67/41/63cbf0f6ac0a6156588b9587be4db5565f8c6d8ccef98202fc\n",
            "Successfully built joeynmt pyyaml wrapt\n",
            "Installing collected packages: mecab-python3, portalocker, sacrebleu, subword-nmt, pyyaml, lazy-object-proxy, wrapt, typed-ast, astroid, toml, isort, mccabe, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Found existing installation: wrapt 1.12.1\n",
            "    Uninstalling wrapt-1.12.1:\n",
            "      Successfully uninstalled wrapt-1.12.1\n",
            "Successfully installed astroid-2.4.2 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 mecab-python3-0.996.5 portalocker-1.7.0 pylint-2.5.3 pyyaml-5.3.1 sacrebleu-1.4.12 subword-nmt-0.3.7 toml-0.10.1 typed-ast-1.4.1 wrapt-1.11.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 425
        },
        "outputId": "2985d384-7d53-4026-8d34-9a8d5de1f0df"
      },
      "source": [
        "#TODO: Skip for retrain\n",
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "from os import path\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# Learn BPEs on the training data.\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# Apply BPE splits to the development and test data.\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! cp train.* $data_path\n",
        "! cp test.* $data_path\n",
        "! cp dev.* $data_path\n",
        "! cp bpe.codes.4000 $data_path\n",
        "! ls $data_path\n",
        "\n",
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\"\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path \"$gdrive_path/vocab.txt\"\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE Nyanja Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 \"$gdrive_path/vocab.txt\"  # Herman"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.en\t     test.bpe.nya    test.en-any.en.1  train.bpe.nya\n",
            "dev.bpe.en\tdev.nya      test.en\t     test.nya\t       train.en\n",
            "dev.bpe.nya\ttest.bpe.en  test.en-any.en  train.bpe.en      train.nya\n",
            "bpe.codes.4000\tdev.nya       test.en\t\ttrain.bpe.en\n",
            "dev.bpe.en\tmodels\t      test.en-any.en\ttrain.bpe.nya\n",
            "dev.bpe.nya\ttest.bpe.en   test.en-any.en.1\ttrain.en\n",
            "dev.en\t\ttest.bpe.nya  test.nya\t\ttrain.nya\n",
            "BPE Xhosa Sentences\n",
            "Izi zin@@ acititsa kuti nd@@ iz@@ idz@@ iŵika monga munthu wos@@ aona mtima .\n",
            "N’@@ t@@ aphunzila coonadi , ndin@@ al@@ eka n@@ chit@@ oyo ngakhale kuti n’nali kulandila ndalama zambili .\n",
            "N@@ apeleka citsanzo cabwino kwa ana anga aŵili a@@ amuna , ndipo tsopano n’n@@ a@@ ikidwa pau@@ d@@ indo mumpingo .\n",
            "Cifukwa co@@ khala wo@@ ona mtima , n@@ ili ndi mbili yabwino kwa anthu amene amat@@ enga mis@@ onkh@@ o ndi ena amene nim@@ acita nawo b@@ iz@@ in@@ esi . ”\n",
            "Kum@@ en@@ eko anayamba kulambila Mulungu wo@@ ona .\n",
            "Combined BPE Vocab\n",
            "ʺ\n",
            "righte@@\n",
            "ambuyo\n",
            "amvu\n",
            "dif@@\n",
            "SONG@@\n",
            "cogn@@\n",
            "ʼ@@\n",
            "Ó@@\n",
            "Yosef@@\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wc47fvWqyxbd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_last_checkpoint(directory):\n",
        "  last_checkpoint = ''\n",
        "  try:\n",
        "    for filename in os.listdir(directory):\n",
        "      if 'best' in filename and filename.endswith(\".ckpt\"):\n",
        "        return filename\n",
        "      if not 'best' in filename and filename.endswith(\".ckpt\"):\n",
        "          if not last_checkpoint or int(filename.split('.')[0]) > int(last_checkpoint.split('.')[0]):\n",
        "            last_checkpoint = filename\n",
        "  except FileNotFoundError as e:\n",
        "    print('Error Occur ', e)\n",
        "  return last_checkpoint"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x_ffEoFdy1Qo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "9a8b9540-cf44-4790-9fb8-3e38c4578494"
      },
      "source": [
        "# Copy the created models from the temporary storage to main storage on google drive for persistant storage \n",
        "# the content of te folder will be overwrite when you start trainin\n",
        "!cp -r \"/content/drive/My Drive/masakhane/model-temp/\"* \"$gdrive_path/models/${src}${tgt}_transformer/\"\n",
        "last_checkpoint = get_last_checkpoint(models_path)\n",
        "print('Last checkpoint :',last_checkpoint)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cp: cannot stat '/content/drive/My Drive/masakhane/model-temp/*': No such file or directory\n",
            "Last checkpoint : \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"{gdrive_path}/train.bpe\"\n",
        "    dev:   \"{gdrive_path}/dev.bpe\"\n",
        "    test:  \"{gdrive_path}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"{gdrive_path}/vocab.txt\"\n",
        "    trg_vocab: \"{gdrive_path}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    #load_model: \"{gdrive_path}/models/{name}_transformer/{last_checkpoint}\" # TODO: uncommented to load a pre-trained model from last checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 50                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 1000          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"{model_temp_dir}\"\n",
        "    overwrite: True               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language, model_temp_dir=model_temp_dir, last_checkpoint=last_checkpoint)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "70255270-149d-47a6-c71e-08d2ac4d67a4"
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-07-12 20:38:52,275 Hello! This is Joey-NMT.\n",
            "2020-07-12 20:38:52.392375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
            "2020-07-12 20:38:53,554 Total params: 12128768\n",
            "2020-07-12 20:38:53,556 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2020-07-12 20:39:09,397 cfg.name                           : ennya_transformer\n",
            "2020-07-12 20:39:09,398 cfg.data.src                       : en\n",
            "2020-07-12 20:39:09,398 cfg.data.trg                       : nya\n",
            "2020-07-12 20:39:09,398 cfg.data.train                     : /content/drive/My Drive/masakhane/en-nya-baseline/train.bpe\n",
            "2020-07-12 20:39:09,398 cfg.data.dev                       : /content/drive/My Drive/masakhane/en-nya-baseline/dev.bpe\n",
            "2020-07-12 20:39:09,398 cfg.data.test                      : /content/drive/My Drive/masakhane/en-nya-baseline/test.bpe\n",
            "2020-07-12 20:39:09,398 cfg.data.level                     : bpe\n",
            "2020-07-12 20:39:09,399 cfg.data.lowercase                 : False\n",
            "2020-07-12 20:39:09,399 cfg.data.max_sent_length           : 100\n",
            "2020-07-12 20:39:09,399 cfg.data.src_vocab                 : /content/drive/My Drive/masakhane/en-nya-baseline/vocab.txt\n",
            "2020-07-12 20:39:09,399 cfg.data.trg_vocab                 : /content/drive/My Drive/masakhane/en-nya-baseline/vocab.txt\n",
            "2020-07-12 20:39:09,399 cfg.testing.beam_size              : 5\n",
            "2020-07-12 20:39:09,399 cfg.testing.alpha                  : 1.0\n",
            "2020-07-12 20:39:09,399 cfg.training.random_seed           : 42\n",
            "2020-07-12 20:39:09,400 cfg.training.optimizer             : adam\n",
            "2020-07-12 20:39:09,400 cfg.training.normalization         : tokens\n",
            "2020-07-12 20:39:09,400 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2020-07-12 20:39:09,400 cfg.training.scheduling            : plateau\n",
            "2020-07-12 20:39:09,400 cfg.training.patience              : 5\n",
            "2020-07-12 20:39:09,400 cfg.training.learning_rate_factor  : 0.5\n",
            "2020-07-12 20:39:09,400 cfg.training.learning_rate_warmup  : 1000\n",
            "2020-07-12 20:39:09,401 cfg.training.decrease_factor       : 0.7\n",
            "2020-07-12 20:39:09,401 cfg.training.loss                  : crossentropy\n",
            "2020-07-12 20:39:09,401 cfg.training.learning_rate         : 0.0003\n",
            "2020-07-12 20:39:09,401 cfg.training.learning_rate_min     : 1e-08\n",
            "2020-07-12 20:39:09,401 cfg.training.weight_decay          : 0.0\n",
            "2020-07-12 20:39:09,401 cfg.training.label_smoothing       : 0.1\n",
            "2020-07-12 20:39:09,401 cfg.training.batch_size            : 4096\n",
            "2020-07-12 20:39:09,401 cfg.training.batch_type            : token\n",
            "2020-07-12 20:39:09,402 cfg.training.eval_batch_size       : 3600\n",
            "2020-07-12 20:39:09,402 cfg.training.eval_batch_type       : token\n",
            "2020-07-12 20:39:09,402 cfg.training.batch_multiplier      : 1\n",
            "2020-07-12 20:39:09,402 cfg.training.early_stopping_metric : ppl\n",
            "2020-07-12 20:39:09,402 cfg.training.epochs                : 50\n",
            "2020-07-12 20:39:09,402 cfg.training.validation_freq       : 1000\n",
            "2020-07-12 20:39:09,402 cfg.training.logging_freq          : 100\n",
            "2020-07-12 20:39:09,403 cfg.training.eval_metric           : bleu\n",
            "2020-07-12 20:39:09,403 cfg.training.model_dir             : /content/drive/My Drive/masakhane/model-temp\n",
            "2020-07-12 20:39:09,403 cfg.training.overwrite             : True\n",
            "2020-07-12 20:39:09,403 cfg.training.shuffle               : True\n",
            "2020-07-12 20:39:09,403 cfg.training.use_cuda              : True\n",
            "2020-07-12 20:39:09,403 cfg.training.max_output_length     : 100\n",
            "2020-07-12 20:39:09,403 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2020-07-12 20:39:09,404 cfg.training.keep_last_ckpts       : 3\n",
            "2020-07-12 20:39:09,404 cfg.model.initializer              : xavier\n",
            "2020-07-12 20:39:09,404 cfg.model.bias_initializer         : zeros\n",
            "2020-07-12 20:39:09,404 cfg.model.init_gain                : 1.0\n",
            "2020-07-12 20:39:09,404 cfg.model.embed_initializer        : xavier\n",
            "2020-07-12 20:39:09,404 cfg.model.embed_init_gain          : 1.0\n",
            "2020-07-12 20:39:09,404 cfg.model.tied_embeddings          : True\n",
            "2020-07-12 20:39:09,405 cfg.model.tied_softmax             : True\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.type             : transformer\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.num_layers       : 6\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.num_heads        : 4\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.embeddings.scale : True\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2020-07-12 20:39:09,405 cfg.model.encoder.hidden_size      : 256\n",
            "2020-07-12 20:39:09,406 cfg.model.encoder.ff_size          : 1024\n",
            "2020-07-12 20:39:09,406 cfg.model.encoder.dropout          : 0.3\n",
            "2020-07-12 20:39:09,406 cfg.model.decoder.type             : transformer\n",
            "2020-07-12 20:39:09,406 cfg.model.decoder.num_layers       : 6\n",
            "2020-07-12 20:39:09,406 cfg.model.decoder.num_heads        : 4\n",
            "2020-07-12 20:39:09,406 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2020-07-12 20:39:09,406 cfg.model.decoder.embeddings.scale : True\n",
            "2020-07-12 20:39:09,407 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2020-07-12 20:39:09,407 cfg.model.decoder.hidden_size      : 256\n",
            "2020-07-12 20:39:09,407 cfg.model.decoder.ff_size          : 1024\n",
            "2020-07-12 20:39:09,407 cfg.model.decoder.dropout          : 0.3\n",
            "2020-07-12 20:39:09,407 Data set sizes: \n",
            "\ttrain 49851,\n",
            "\tvalid 1000,\n",
            "\ttest 2672\n",
            "2020-07-12 20:39:09,407 First training example:\n",
            "\t[SRC] How Job came to know Jehovah .\n",
            "\t[TRG] M@@ mene Yobu anadz@@ iŵ@@ ila Yehova .\n",
            "2020-07-12 20:39:09,408 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) the (7) to (8) a (9) :\n",
            "2020-07-12 20:39:09,408 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) the (7) to (8) a (9) :\n",
            "2020-07-12 20:39:09,408 Number of Src words (types): 4174\n",
            "2020-07-12 20:39:09,409 Number of Trg words (types): 4174\n",
            "2020-07-12 20:39:09,409 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4174),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4174))\n",
            "2020-07-12 20:39:09,418 EPOCH 1\n",
            "/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of nonzero is deprecated:\n",
            "\tnonzero(Tensor input, *, Tensor out)\n",
            "Consider using one of the following signatures instead:\n",
            "\tnonzero(Tensor input, *, bool as_tuple)\n",
            "2020-07-12 20:39:28,225 Epoch   1 Step:      100 Batch Loss:     5.851800 Tokens per Sec:    11050, Lr: 0.000300\n",
            "2020-07-12 20:39:46,905 Epoch   1 Step:      200 Batch Loss:     5.502884 Tokens per Sec:    11317, Lr: 0.000300\n",
            "2020-07-12 20:40:05,233 Epoch   1 Step:      300 Batch Loss:     5.292922 Tokens per Sec:    11160, Lr: 0.000300\n",
            "2020-07-12 20:40:23,613 Epoch   1 Step:      400 Batch Loss:     5.175645 Tokens per Sec:    11256, Lr: 0.000300\n",
            "2020-07-12 20:40:41,982 Epoch   1 Step:      500 Batch Loss:     4.725220 Tokens per Sec:    11361, Lr: 0.000300\n",
            "2020-07-12 20:40:56,727 Epoch   1: total training loss 3090.93\n",
            "2020-07-12 20:40:56,728 EPOCH 2\n",
            "2020-07-12 20:41:00,417 Epoch   2 Step:      600 Batch Loss:     4.830907 Tokens per Sec:    11316, Lr: 0.000300\n",
            "2020-07-12 20:41:18,572 Epoch   2 Step:      700 Batch Loss:     4.736766 Tokens per Sec:    11298, Lr: 0.000300\n",
            "2020-07-12 20:41:36,802 Epoch   2 Step:      800 Batch Loss:     4.635335 Tokens per Sec:    11134, Lr: 0.000300\n",
            "2020-07-12 20:41:55,103 Epoch   2 Step:      900 Batch Loss:     4.045403 Tokens per Sec:    11358, Lr: 0.000300\n",
            "2020-07-12 20:42:13,600 Epoch   2 Step:     1000 Batch Loss:     4.308612 Tokens per Sec:    11398, Lr: 0.000300\n",
            "2020-07-12 20:42:55,456 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 20:42:55,456 Saving new checkpoint.\n",
            "2020-07-12 20:42:56,628 Example #0\n",
            "2020-07-12 20:42:56,628 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 20:42:56,629 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 20:42:56,629 \tHypothesis: ( 1 : 3 ) Kodi anthu amene anali kuthandiza kuti anali kuthandiza kuti anali kuthandiza .\n",
            "2020-07-12 20:42:56,629 Example #1\n",
            "2020-07-12 20:42:56,629 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 20:42:56,630 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 20:42:56,630 \tHypothesis: ( 1 : 1 ) Koma anthu amene anali kuthandiza kuti anali kukuthandiza .\n",
            "2020-07-12 20:42:56,630 Example #2\n",
            "2020-07-12 20:42:56,630 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 20:42:56,630 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 20:42:56,630 \tHypothesis: ( b ) Kodi anthu a Mulungu amene anali kuthandiza bwanji kuti : “ Kodi Yehova ?\n",
            "2020-07-12 20:42:56,631 Example #3\n",
            "2020-07-12 20:42:56,631 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 20:42:56,631 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 20:42:56,631 \tHypothesis: Kodi Malile\n",
            "2020-07-12 20:42:56,631 Validation result (greedy) at epoch   2, step     1000: bleu:   0.73, loss: 104985.1797, ppl:  69.0250, duration: 43.0307s\n",
            "2020-07-12 20:43:15,235 Epoch   2 Step:     1100 Batch Loss:     4.252433 Tokens per Sec:    11137, Lr: 0.000300\n",
            "2020-07-12 20:43:27,269 Epoch   2: total training loss 2596.20\n",
            "2020-07-12 20:43:27,270 EPOCH 3\n",
            "2020-07-12 20:43:33,992 Epoch   3 Step:     1200 Batch Loss:     3.654455 Tokens per Sec:    11098, Lr: 0.000300\n",
            "2020-07-12 20:43:52,339 Epoch   3 Step:     1300 Batch Loss:     3.389505 Tokens per Sec:    11076, Lr: 0.000300\n",
            "2020-07-12 20:44:11,141 Epoch   3 Step:     1400 Batch Loss:     3.965687 Tokens per Sec:    11462, Lr: 0.000300\n",
            "2020-07-12 20:44:29,325 Epoch   3 Step:     1500 Batch Loss:     3.584218 Tokens per Sec:    11146, Lr: 0.000300\n",
            "2020-07-12 20:44:47,940 Epoch   3 Step:     1600 Batch Loss:     3.427156 Tokens per Sec:    11108, Lr: 0.000300\n",
            "2020-07-12 20:45:06,608 Epoch   3 Step:     1700 Batch Loss:     4.065697 Tokens per Sec:    11517, Lr: 0.000300\n",
            "2020-07-12 20:45:14,535 Epoch   3: total training loss 2279.14\n",
            "2020-07-12 20:45:14,536 EPOCH 4\n",
            "2020-07-12 20:45:25,196 Epoch   4 Step:     1800 Batch Loss:     3.446399 Tokens per Sec:    11137, Lr: 0.000300\n",
            "2020-07-12 20:45:43,520 Epoch   4 Step:     1900 Batch Loss:     3.538134 Tokens per Sec:    11206, Lr: 0.000300\n",
            "2020-07-12 20:46:02,213 Epoch   4 Step:     2000 Batch Loss:     3.472388 Tokens per Sec:    11551, Lr: 0.000300\n",
            "2020-07-12 20:46:31,222 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 20:46:31,223 Saving new checkpoint.\n",
            "2020-07-12 20:46:32,434 Example #0\n",
            "2020-07-12 20:46:32,434 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 20:46:32,434 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 20:46:32,434 \tHypothesis: Kodi anthu ambili ambili ambili anali kudya ku dziko lapansi ?\n",
            "2020-07-12 20:46:32,435 Example #1\n",
            "2020-07-12 20:46:32,435 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 20:46:32,435 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 20:46:32,435 \tHypothesis: Iye anali kudya , ndipo anali kudya ku Yerusalemu .\n",
            "2020-07-12 20:46:32,436 Example #2\n",
            "2020-07-12 20:46:32,436 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 20:46:32,436 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 20:46:32,436 \tHypothesis: Kodi Yesu anakamba kuti anthu a Yesu anali ndi mau a Yesu ?\n",
            "2020-07-12 20:46:32,437 Example #3\n",
            "2020-07-12 20:46:32,437 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 20:46:32,437 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 20:46:32,437 \tHypothesis: Kodi Mfundo ya Mudzi Wosiyana ndi Mtima\n",
            "2020-07-12 20:46:32,437 Validation result (greedy) at epoch   4, step     2000: bleu:   2.10, loss: 87342.0547, ppl:  33.8809, duration: 30.2243s\n",
            "2020-07-12 20:46:50,898 Epoch   4 Step:     2100 Batch Loss:     3.669708 Tokens per Sec:    11043, Lr: 0.000300\n",
            "2020-07-12 20:47:09,397 Epoch   4 Step:     2200 Batch Loss:     3.272874 Tokens per Sec:    11294, Lr: 0.000300\n",
            "2020-07-12 20:47:27,868 Epoch   4 Step:     2300 Batch Loss:     3.434987 Tokens per Sec:    11149, Lr: 0.000300\n",
            "2020-07-12 20:47:32,157 Epoch   4: total training loss 2088.17\n",
            "2020-07-12 20:47:32,158 EPOCH 5\n",
            "2020-07-12 20:47:46,655 Epoch   5 Step:     2400 Batch Loss:     2.883982 Tokens per Sec:    11188, Lr: 0.000300\n",
            "2020-07-12 20:48:05,133 Epoch   5 Step:     2500 Batch Loss:     3.808741 Tokens per Sec:    11522, Lr: 0.000300\n",
            "2020-07-12 20:48:23,395 Epoch   5 Step:     2600 Batch Loss:     3.231003 Tokens per Sec:    11131, Lr: 0.000300\n",
            "2020-07-12 20:48:41,768 Epoch   5 Step:     2700 Batch Loss:     3.547242 Tokens per Sec:    11405, Lr: 0.000300\n",
            "2020-07-12 20:49:00,056 Epoch   5 Step:     2800 Batch Loss:     3.122620 Tokens per Sec:    11173, Lr: 0.000300\n",
            "2020-07-12 20:49:18,411 Epoch   5 Step:     2900 Batch Loss:     3.458104 Tokens per Sec:    11189, Lr: 0.000300\n",
            "2020-07-12 20:49:19,182 Epoch   5: total training loss 1954.32\n",
            "2020-07-12 20:49:19,182 EPOCH 6\n",
            "2020-07-12 20:49:37,162 Epoch   6 Step:     3000 Batch Loss:     3.304617 Tokens per Sec:    11262, Lr: 0.000300\n",
            "2020-07-12 20:49:56,697 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 20:49:56,697 Saving new checkpoint.\n",
            "2020-07-12 20:49:57,944 Example #0\n",
            "2020-07-12 20:49:57,945 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 20:49:57,945 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 20:49:57,945 \tHypothesis: Tinali kufunitsitsa kukonzekela nchito yolalikila yolalikila yolalikila yolalikila .\n",
            "2020-07-12 20:49:57,946 Example #1\n",
            "2020-07-12 20:49:57,946 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 20:49:57,947 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 20:49:57,947 \tHypothesis: Iye anali kufunitsitsa kucita zimenezi , ndipo anali kudya zaka zambili .\n",
            "2020-07-12 20:49:57,947 Example #2\n",
            "2020-07-12 20:49:57,948 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 20:49:57,948 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 20:49:57,948 \tHypothesis: Kodi Mose anacita ciani kuti ophunzila ake adzagwilitsila nchito mau a Paulo ?\n",
            "2020-07-12 20:49:57,948 Example #3\n",
            "2020-07-12 20:49:57,949 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 20:49:57,949 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 20:49:57,949 \tHypothesis: Kudziŵa kuti Mmene Mmene Mmene Mmene Mmene Mmene Mudziŵa\n",
            "2020-07-12 20:49:57,949 Validation result (greedy) at epoch   6, step     3000: bleu:   3.32, loss: 78307.3125, ppl:  23.5340, duration: 20.7865s\n",
            "2020-07-12 20:50:16,871 Epoch   6 Step:     3100 Batch Loss:     3.466974 Tokens per Sec:    11099, Lr: 0.000300\n",
            "2020-07-12 20:50:35,318 Epoch   6 Step:     3200 Batch Loss:     2.713333 Tokens per Sec:    10942, Lr: 0.000300\n",
            "2020-07-12 20:50:53,654 Epoch   6 Step:     3300 Batch Loss:     3.480570 Tokens per Sec:    11215, Lr: 0.000300\n",
            "2020-07-12 20:51:12,139 Epoch   6 Step:     3400 Batch Loss:     2.565523 Tokens per Sec:    11391, Lr: 0.000300\n",
            "2020-07-12 20:51:28,070 Epoch   6: total training loss 1860.15\n",
            "2020-07-12 20:51:28,070 EPOCH 7\n",
            "2020-07-12 20:51:30,605 Epoch   7 Step:     3500 Batch Loss:     3.121952 Tokens per Sec:    10914, Lr: 0.000300\n",
            "2020-07-12 20:51:49,109 Epoch   7 Step:     3600 Batch Loss:     2.306157 Tokens per Sec:    11148, Lr: 0.000300\n",
            "2020-07-12 20:52:07,713 Epoch   7 Step:     3700 Batch Loss:     2.988599 Tokens per Sec:    11346, Lr: 0.000300\n",
            "2020-07-12 20:52:26,172 Epoch   7 Step:     3800 Batch Loss:     3.353390 Tokens per Sec:    11405, Lr: 0.000300\n",
            "2020-07-12 20:52:44,412 Epoch   7 Step:     3900 Batch Loss:     2.968240 Tokens per Sec:    11083, Lr: 0.000300\n",
            "2020-07-12 20:53:02,675 Epoch   7 Step:     4000 Batch Loss:     2.396898 Tokens per Sec:    11183, Lr: 0.000300\n",
            "2020-07-12 20:53:26,332 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 20:53:26,332 Saving new checkpoint.\n",
            "2020-07-12 20:53:27,500 Example #0\n",
            "2020-07-12 20:53:27,500 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 20:53:27,501 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 20:53:27,501 \tHypothesis: Tidzakhala ndi mavuto amene tinali kukumana nao padziko lonse .\n",
            "2020-07-12 20:53:27,501 Example #1\n",
            "2020-07-12 20:53:27,502 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 20:53:27,502 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 20:53:27,502 \tHypothesis: Iye angacite zimenezi , ndipo anali kudya zonsezi .\n",
            "2020-07-12 20:53:27,502 Example #2\n",
            "2020-07-12 20:53:27,502 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 20:53:27,502 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 20:53:27,503 \tHypothesis: Kodi Mose anakamba ciani ponena za anthu amene Paulo analemba kuti : “ Kodi mau a Paulo a Paulo ali ndi mau a Paulo ?\n",
            "2020-07-12 20:53:27,503 Example #3\n",
            "2020-07-12 20:53:27,503 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 20:53:27,503 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 20:53:27,503 \tHypothesis: Kuwonjezela apo , Nchito Yabwino Amene Ahabu\n",
            "2020-07-12 20:53:27,504 Validation result (greedy) at epoch   7, step     4000: bleu:   5.24, loss: 72736.8750, ppl:  18.7983, duration: 24.8278s\n",
            "2020-07-12 20:53:40,429 Epoch   7: total training loss 1776.31\n",
            "2020-07-12 20:53:40,429 EPOCH 8\n",
            "2020-07-12 20:53:46,268 Epoch   8 Step:     4100 Batch Loss:     2.750468 Tokens per Sec:    11046, Lr: 0.000300\n",
            "2020-07-12 20:54:04,739 Epoch   8 Step:     4200 Batch Loss:     3.271371 Tokens per Sec:    11177, Lr: 0.000300\n",
            "2020-07-12 20:54:23,319 Epoch   8 Step:     4300 Batch Loss:     3.047179 Tokens per Sec:    11402, Lr: 0.000300\n",
            "2020-07-12 20:54:41,779 Epoch   8 Step:     4400 Batch Loss:     2.842948 Tokens per Sec:    11260, Lr: 0.000300\n",
            "2020-07-12 20:55:00,279 Epoch   8 Step:     4500 Batch Loss:     2.115280 Tokens per Sec:    11292, Lr: 0.000300\n",
            "2020-07-12 20:55:18,684 Epoch   8 Step:     4600 Batch Loss:     1.979993 Tokens per Sec:    11151, Lr: 0.000300\n",
            "2020-07-12 20:55:27,832 Epoch   8: total training loss 1702.89\n",
            "2020-07-12 20:55:27,833 EPOCH 9\n",
            "2020-07-12 20:55:37,475 Epoch   9 Step:     4700 Batch Loss:     1.902806 Tokens per Sec:    11243, Lr: 0.000300\n",
            "2020-07-12 20:55:56,021 Epoch   9 Step:     4800 Batch Loss:     3.366707 Tokens per Sec:    11431, Lr: 0.000300\n",
            "2020-07-12 20:56:14,599 Epoch   9 Step:     4900 Batch Loss:     2.641436 Tokens per Sec:    11356, Lr: 0.000300\n",
            "2020-07-12 20:56:33,118 Epoch   9 Step:     5000 Batch Loss:     1.813956 Tokens per Sec:    11200, Lr: 0.000300\n",
            "2020-07-12 20:56:59,654 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 20:56:59,654 Saving new checkpoint.\n",
            "2020-07-12 20:57:00,799 Example #0\n",
            "2020-07-12 20:57:00,800 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 20:57:00,800 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 20:57:00,800 \tHypothesis: Tinali kufufuza za dziko lathu limene tinalamula anthu ambili .\n",
            "2020-07-12 20:57:00,800 Example #1\n",
            "2020-07-12 20:57:00,801 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 20:57:00,801 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 20:57:00,801 \tHypothesis: Iye angacite zimenezo kuti akakhale ndi zaka zambili .\n",
            "2020-07-12 20:57:00,801 Example #2\n",
            "2020-07-12 20:57:00,802 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 20:57:00,802 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 20:57:00,802 \tHypothesis: Kodi Mose anaonetsa bwanji kuti anali ndi mau a Paulo ?\n",
            "2020-07-12 20:57:00,802 Example #3\n",
            "2020-07-12 20:57:00,802 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 20:57:00,803 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 20:57:00,803 \tHypothesis: Cikondi Cikondi Cikondi Cikondi Cikondi Cake Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cikondi Cake\n",
            "2020-07-12 20:57:00,803 Validation result (greedy) at epoch   9, step     5000: bleu:   6.91, loss: 68773.7266, ppl:  16.0213, duration: 27.6843s\n",
            "2020-07-12 20:57:19,792 Epoch   9 Step:     5100 Batch Loss:     3.378844 Tokens per Sec:    10709, Lr: 0.000300\n",
            "2020-07-12 20:57:38,438 Epoch   9 Step:     5200 Batch Loss:     3.053119 Tokens per Sec:    11093, Lr: 0.000300\n",
            "2020-07-12 20:57:43,714 Epoch   9: total training loss 1654.79\n",
            "2020-07-12 20:57:43,715 EPOCH 10\n",
            "2020-07-12 20:57:56,691 Epoch  10 Step:     5300 Batch Loss:     2.894244 Tokens per Sec:    11050, Lr: 0.000300\n",
            "2020-07-12 20:58:14,962 Epoch  10 Step:     5400 Batch Loss:     3.201298 Tokens per Sec:    11192, Lr: 0.000300\n",
            "2020-07-12 20:58:33,363 Epoch  10 Step:     5500 Batch Loss:     3.223227 Tokens per Sec:    11257, Lr: 0.000300\n",
            "2020-07-12 20:58:51,574 Epoch  10 Step:     5600 Batch Loss:     2.737585 Tokens per Sec:    11035, Lr: 0.000300\n",
            "2020-07-12 20:59:09,738 Epoch  10 Step:     5700 Batch Loss:     2.743144 Tokens per Sec:    11207, Lr: 0.000300\n",
            "2020-07-12 20:59:28,488 Epoch  10 Step:     5800 Batch Loss:     2.704981 Tokens per Sec:    11613, Lr: 0.000300\n",
            "2020-07-12 20:59:31,146 Epoch  10: total training loss 1624.04\n",
            "2020-07-12 20:59:31,146 EPOCH 11\n",
            "2020-07-12 20:59:47,229 Epoch  11 Step:     5900 Batch Loss:     2.886471 Tokens per Sec:    11408, Lr: 0.000300\n",
            "2020-07-12 21:00:05,688 Epoch  11 Step:     6000 Batch Loss:     2.807448 Tokens per Sec:    11172, Lr: 0.000300\n",
            "2020-07-12 21:00:38,757 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:00:38,758 Saving new checkpoint.\n",
            "2020-07-12 21:00:39,876 Example #0\n",
            "2020-07-12 21:00:39,876 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:00:39,877 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:00:39,877 \tHypothesis: Mogwilizana ndi anthu amene tinali kukumana nao padziko lapansi , tinayamba kukumana ndi mavuto ambili\n",
            "2020-07-12 21:00:39,877 Example #1\n",
            "2020-07-12 21:00:39,877 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:00:39,877 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:00:39,878 \tHypothesis: Iye angacite bwino kuona mmene anali kucitila zinthu zambili , ngakhale kuti anali ndi zaka zambili .\n",
            "2020-07-12 21:00:39,878 Example #2\n",
            "2020-07-12 21:00:39,878 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:00:39,878 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:00:39,878 \tHypothesis: Kodi Mose anacita ciani pa tsiku la Paulo la 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:00:39,878 Example #3\n",
            "2020-07-12 21:00:39,879 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:00:39,879 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:00:39,879 \tHypothesis: Mmene Mumacita Mkulu Wabwino , July\n",
            "2020-07-12 21:00:39,879 Validation result (greedy) at epoch  11, step     6000: bleu:   8.22, loss: 66082.8594, ppl:  14.3735, duration: 34.1909s\n",
            "2020-07-12 21:00:58,498 Epoch  11 Step:     6100 Batch Loss:     2.824265 Tokens per Sec:    11075, Lr: 0.000300\n",
            "2020-07-12 21:01:17,161 Epoch  11 Step:     6200 Batch Loss:     2.993033 Tokens per Sec:    11128, Lr: 0.000300\n",
            "2020-07-12 21:01:35,768 Epoch  11 Step:     6300 Batch Loss:     2.654554 Tokens per Sec:    11077, Lr: 0.000300\n",
            "2020-07-12 21:01:53,341 Epoch  11: total training loss 1568.96\n",
            "2020-07-12 21:01:53,342 EPOCH 12\n",
            "2020-07-12 21:01:54,486 Epoch  12 Step:     6400 Batch Loss:     2.727532 Tokens per Sec:     9916, Lr: 0.000300\n",
            "2020-07-12 21:02:13,003 Epoch  12 Step:     6500 Batch Loss:     2.721638 Tokens per Sec:    11120, Lr: 0.000300\n",
            "2020-07-12 21:02:31,610 Epoch  12 Step:     6600 Batch Loss:     2.611042 Tokens per Sec:    11272, Lr: 0.000300\n",
            "2020-07-12 21:02:50,016 Epoch  12 Step:     6700 Batch Loss:     3.039977 Tokens per Sec:    11127, Lr: 0.000300\n",
            "2020-07-12 21:03:08,468 Epoch  12 Step:     6800 Batch Loss:     2.798577 Tokens per Sec:    11329, Lr: 0.000300\n",
            "2020-07-12 21:03:27,082 Epoch  12 Step:     6900 Batch Loss:     2.618927 Tokens per Sec:    11387, Lr: 0.000300\n",
            "2020-07-12 21:03:40,650 Epoch  12: total training loss 1529.47\n",
            "2020-07-12 21:03:40,651 EPOCH 13\n",
            "2020-07-12 21:03:45,869 Epoch  13 Step:     7000 Batch Loss:     2.779525 Tokens per Sec:    11528, Lr: 0.000300\n",
            "2020-07-12 21:04:08,411 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:04:08,412 Saving new checkpoint.\n",
            "2020-07-12 21:04:09,588 Example #0\n",
            "2020-07-12 21:04:09,588 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:04:09,589 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:04:09,589 \tHypothesis: Tinali kudya zakudya zocepa kuti tikwanitse anthu ambili m’dzikoli\n",
            "2020-07-12 21:04:09,589 Example #1\n",
            "2020-07-12 21:04:09,589 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:04:09,590 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:04:09,590 \tHypothesis: Iye angacite zimenezi mwa kucita zinthu zimene zinacitika m’nthawi ya atumwi .\n",
            "2020-07-12 21:04:09,590 Example #2\n",
            "2020-07-12 21:04:09,591 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:04:09,591 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:04:09,591 \tHypothesis: Kodi Mose anacita ciani pa nthawi ya atumwi ?\n",
            "2020-07-12 21:04:09,591 Example #3\n",
            "2020-07-12 21:04:09,591 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:04:09,592 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:04:09,592 \tHypothesis: Kudziŵa Coonadi Conse Conse Conse Coonadi Conse , July\n",
            "2020-07-12 21:04:09,592 Validation result (greedy) at epoch  13, step     7000: bleu:   9.08, loss: 63526.4961, ppl:  12.9653, duration: 23.7224s\n",
            "2020-07-12 21:04:28,101 Epoch  13 Step:     7100 Batch Loss:     2.530882 Tokens per Sec:    11105, Lr: 0.000300\n",
            "2020-07-12 21:04:46,627 Epoch  13 Step:     7200 Batch Loss:     2.307031 Tokens per Sec:    11074, Lr: 0.000300\n",
            "2020-07-12 21:05:04,913 Epoch  13 Step:     7300 Batch Loss:     2.874648 Tokens per Sec:    11324, Lr: 0.000300\n",
            "2020-07-12 21:05:23,442 Epoch  13 Step:     7400 Batch Loss:     2.225077 Tokens per Sec:    10968, Lr: 0.000300\n",
            "2020-07-12 21:05:41,946 Epoch  13 Step:     7500 Batch Loss:     2.328564 Tokens per Sec:    11249, Lr: 0.000300\n",
            "2020-07-12 21:05:52,458 Epoch  13: total training loss 1513.63\n",
            "2020-07-12 21:05:52,458 EPOCH 14\n",
            "2020-07-12 21:06:00,668 Epoch  14 Step:     7600 Batch Loss:     2.770868 Tokens per Sec:    11209, Lr: 0.000300\n",
            "2020-07-12 21:06:19,075 Epoch  14 Step:     7700 Batch Loss:     2.845640 Tokens per Sec:    11402, Lr: 0.000300\n",
            "2020-07-12 21:06:37,491 Epoch  14 Step:     7800 Batch Loss:     2.488195 Tokens per Sec:    11250, Lr: 0.000300\n",
            "2020-07-12 21:06:55,905 Epoch  14 Step:     7900 Batch Loss:     2.452046 Tokens per Sec:    11275, Lr: 0.000300\n",
            "2020-07-12 21:07:14,267 Epoch  14 Step:     8000 Batch Loss:     2.044750 Tokens per Sec:    11336, Lr: 0.000300\n",
            "2020-07-12 21:07:32,905 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:07:32,905 Saving new checkpoint.\n",
            "2020-07-12 21:07:34,103 Example #0\n",
            "2020-07-12 21:07:34,104 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:07:34,104 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:07:34,104 \tHypothesis: Anthu athu amene anatilamula kuti tikasokoneze dziko lonse\n",
            "2020-07-12 21:07:34,104 Example #1\n",
            "2020-07-12 21:07:34,105 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:07:34,105 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:07:34,105 \tHypothesis: Iye angacite zimenezo mwa kucita zinthu zimene zinacitika kwa zaka zambili .\n",
            "2020-07-12 21:07:34,105 Example #2\n",
            "2020-07-12 21:07:34,105 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:07:34,106 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:07:34,106 \tHypothesis: Kodi Mose anacita ciani pa lemba la 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:07:34,106 Example #3\n",
            "2020-07-12 21:07:34,106 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:07:34,107 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:07:34,107 \tHypothesis: Kuphunzila Cikondi Catsopano Catsopano Conse Cimalimba , July\n",
            "2020-07-12 21:07:34,107 Validation result (greedy) at epoch  14, step     8000: bleu:   9.82, loss: 61911.1172, ppl:  12.1475, duration: 19.8398s\n",
            "2020-07-12 21:07:52,888 Epoch  14 Step:     8100 Batch Loss:     1.988333 Tokens per Sec:    10812, Lr: 0.000300\n",
            "2020-07-12 21:07:59,991 Epoch  14: total training loss 1482.87\n",
            "2020-07-12 21:07:59,991 EPOCH 15\n",
            "2020-07-12 21:08:11,604 Epoch  15 Step:     8200 Batch Loss:     2.349521 Tokens per Sec:    11173, Lr: 0.000300\n",
            "2020-07-12 21:08:30,140 Epoch  15 Step:     8300 Batch Loss:     2.845519 Tokens per Sec:    11171, Lr: 0.000300\n",
            "2020-07-12 21:08:48,724 Epoch  15 Step:     8400 Batch Loss:     2.336199 Tokens per Sec:    11297, Lr: 0.000300\n",
            "2020-07-12 21:09:07,205 Epoch  15 Step:     8500 Batch Loss:     2.768113 Tokens per Sec:    11375, Lr: 0.000300\n",
            "2020-07-12 21:09:25,565 Epoch  15 Step:     8600 Batch Loss:     2.474854 Tokens per Sec:    10940, Lr: 0.000300\n",
            "2020-07-12 21:09:43,770 Epoch  15 Step:     8700 Batch Loss:     2.519118 Tokens per Sec:    11164, Lr: 0.000300\n",
            "2020-07-12 21:09:47,738 Epoch  15: total training loss 1457.46\n",
            "2020-07-12 21:09:47,739 EPOCH 16\n",
            "2020-07-12 21:10:02,372 Epoch  16 Step:     8800 Batch Loss:     1.913652 Tokens per Sec:    11166, Lr: 0.000300\n",
            "2020-07-12 21:10:20,707 Epoch  16 Step:     8900 Batch Loss:     2.514320 Tokens per Sec:    11140, Lr: 0.000300\n",
            "2020-07-12 21:10:39,333 Epoch  16 Step:     9000 Batch Loss:     1.852291 Tokens per Sec:    11105, Lr: 0.000300\n",
            "2020-07-12 21:11:06,572 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:11:06,573 Saving new checkpoint.\n",
            "2020-07-12 21:11:08,349 Example #0\n",
            "2020-07-12 21:11:08,349 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:11:08,350 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:11:08,350 \tHypothesis: Tinali kugaŵila ku mbali ya anthu amene anatilimbikitsa kuti tikapulumuke m’dziko lathu\n",
            "2020-07-12 21:11:08,350 Example #1\n",
            "2020-07-12 21:11:08,350 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:11:08,350 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:11:08,350 \tHypothesis: Iye akanakhala ndi maganizo ake pa zinthu zimene anacita , ngakhale kuti anali kucita zaka zambili .\n",
            "2020-07-12 21:11:08,351 Example #2\n",
            "2020-07-12 21:11:08,351 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:11:08,351 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:11:08,351 \tHypothesis: Kodi Mose anali kuganizila ciani ponena za mau a Paulo a m’mau a Paulo a pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:11:08,351 Example #3\n",
            "2020-07-12 21:11:08,352 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:11:08,352 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:11:08,352 \tHypothesis: Kukhulupilila Moyo Wosaoneka , July\n",
            "2020-07-12 21:11:08,352 Validation result (greedy) at epoch  16, step     9000: bleu:  10.86, loss: 60281.3047, ppl:  11.3746, duration: 29.0181s\n",
            "2020-07-12 21:11:26,775 Epoch  16 Step:     9100 Batch Loss:     2.761579 Tokens per Sec:    11098, Lr: 0.000300\n",
            "2020-07-12 21:11:45,171 Epoch  16 Step:     9200 Batch Loss:     2.247161 Tokens per Sec:    11195, Lr: 0.000300\n",
            "2020-07-12 21:12:03,743 Epoch  16 Step:     9300 Batch Loss:     2.698262 Tokens per Sec:    11143, Lr: 0.000300\n",
            "2020-07-12 21:12:05,049 Epoch  16: total training loss 1439.43\n",
            "2020-07-12 21:12:05,049 EPOCH 17\n",
            "2020-07-12 21:12:22,048 Epoch  17 Step:     9400 Batch Loss:     2.614497 Tokens per Sec:    10893, Lr: 0.000300\n",
            "2020-07-12 21:12:40,624 Epoch  17 Step:     9500 Batch Loss:     2.584436 Tokens per Sec:    11356, Lr: 0.000300\n",
            "2020-07-12 21:12:59,127 Epoch  17 Step:     9600 Batch Loss:     2.364741 Tokens per Sec:    11265, Lr: 0.000300\n",
            "2020-07-12 21:13:17,584 Epoch  17 Step:     9700 Batch Loss:     2.226366 Tokens per Sec:    11219, Lr: 0.000300\n",
            "2020-07-12 21:13:36,308 Epoch  17 Step:     9800 Batch Loss:     2.059721 Tokens per Sec:    11340, Lr: 0.000300\n",
            "2020-07-12 21:13:52,792 Epoch  17: total training loss 1409.13\n",
            "2020-07-12 21:13:52,793 EPOCH 18\n",
            "2020-07-12 21:13:54,731 Epoch  18 Step:     9900 Batch Loss:     2.092695 Tokens per Sec:    11694, Lr: 0.000300\n",
            "2020-07-12 21:14:13,348 Epoch  18 Step:    10000 Batch Loss:     2.355742 Tokens per Sec:    11207, Lr: 0.000300\n",
            "2020-07-12 21:14:46,358 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:14:46,358 Saving new checkpoint.\n",
            "2020-07-12 21:14:47,559 Example #0\n",
            "2020-07-12 21:14:47,560 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:14:47,560 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:14:47,560 \tHypothesis: Tinali kufunitsitsa kutengako mbali m’madela ambili m’dziko lathu\n",
            "2020-07-12 21:14:47,560 Example #1\n",
            "2020-07-12 21:14:47,560 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:14:47,561 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:14:47,561 \tHypothesis: Iye akanaika maganizo ake pa zinthu zina zimene anali kucita , ngakhale kuti anali kucita zinthu zambili .\n",
            "2020-07-12 21:14:47,561 Example #2\n",
            "2020-07-12 21:14:47,561 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:14:47,561 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:14:47,561 \tHypothesis: Kodi zocitika za Mose zinathandiza bwanji Timoteyo kukhala na mau a Paulo pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:14:47,562 Example #3\n",
            "2020-07-12 21:14:47,562 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:14:47,562 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:14:47,562 \tHypothesis: Kugwila Nchito Yopambana Kupilila Kupilila , July\n",
            "2020-07-12 21:14:47,562 Validation result (greedy) at epoch  18, step    10000: bleu:  11.17, loss: 58961.6133, ppl:  10.7850, duration: 34.2140s\n",
            "2020-07-12 21:15:06,506 Epoch  18 Step:    10100 Batch Loss:     2.428898 Tokens per Sec:    11052, Lr: 0.000300\n",
            "2020-07-12 21:15:25,191 Epoch  18 Step:    10200 Batch Loss:     2.221598 Tokens per Sec:    11326, Lr: 0.000300\n",
            "2020-07-12 21:15:43,883 Epoch  18 Step:    10300 Batch Loss:     2.580553 Tokens per Sec:    11107, Lr: 0.000300\n",
            "2020-07-12 21:16:02,403 Epoch  18 Step:    10400 Batch Loss:     2.540907 Tokens per Sec:    11155, Lr: 0.000300\n",
            "2020-07-12 21:16:15,084 Epoch  18: total training loss 1378.97\n",
            "2020-07-12 21:16:15,084 EPOCH 19\n",
            "2020-07-12 21:16:20,903 Epoch  19 Step:    10500 Batch Loss:     1.792645 Tokens per Sec:    10919, Lr: 0.000300\n",
            "2020-07-12 21:16:39,369 Epoch  19 Step:    10600 Batch Loss:     2.224898 Tokens per Sec:    11440, Lr: 0.000300\n",
            "2020-07-12 21:16:57,628 Epoch  19 Step:    10700 Batch Loss:     1.972942 Tokens per Sec:    11074, Lr: 0.000300\n",
            "2020-07-12 21:17:16,274 Epoch  19 Step:    10800 Batch Loss:     2.240098 Tokens per Sec:    11328, Lr: 0.000300\n",
            "2020-07-12 21:17:34,932 Epoch  19 Step:    10900 Batch Loss:     1.967205 Tokens per Sec:    11344, Lr: 0.000300\n",
            "2020-07-12 21:17:53,616 Epoch  19 Step:    11000 Batch Loss:     2.066128 Tokens per Sec:    11285, Lr: 0.000300\n",
            "2020-07-12 21:18:15,331 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:18:15,331 Saving new checkpoint.\n",
            "2020-07-12 21:18:16,431 Example #0\n",
            "2020-07-12 21:18:16,432 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:18:16,432 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:18:16,432 \tHypothesis: Pali pano , tinayamba kusamalila dziko lonse\n",
            "2020-07-12 21:18:16,432 Example #1\n",
            "2020-07-12 21:18:16,433 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:18:16,433 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:18:16,433 \tHypothesis: Iye angafune kuti akambilane za zinthu zimene zinacitika m’zaka zambili , ngakhale kuti anali ndi zaka zambili .\n",
            "2020-07-12 21:18:16,433 Example #2\n",
            "2020-07-12 21:18:16,434 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:18:16,434 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:18:16,434 \tHypothesis: Kodi Mose anali kuganizila ciani pa lemba la 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:18:16,434 Example #3\n",
            "2020-07-12 21:18:16,435 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:18:16,435 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:18:16,435 \tHypothesis: Kucokela kwa Pabanja , July\n",
            "2020-07-12 21:18:16,435 Validation result (greedy) at epoch  19, step    11000: bleu:  11.81, loss: 57576.3789, ppl:  10.1989, duration: 22.8182s\n",
            "2020-07-12 21:18:25,562 Epoch  19: total training loss 1364.70\n",
            "2020-07-12 21:18:25,562 EPOCH 20\n",
            "2020-07-12 21:18:34,940 Epoch  20 Step:    11100 Batch Loss:     2.185124 Tokens per Sec:    11070, Lr: 0.000300\n",
            "2020-07-12 21:18:53,094 Epoch  20 Step:    11200 Batch Loss:     2.498385 Tokens per Sec:    11016, Lr: 0.000300\n",
            "2020-07-12 21:19:11,695 Epoch  20 Step:    11300 Batch Loss:     2.564159 Tokens per Sec:    11289, Lr: 0.000300\n",
            "2020-07-12 21:19:30,054 Epoch  20 Step:    11400 Batch Loss:     2.303161 Tokens per Sec:    11185, Lr: 0.000300\n",
            "2020-07-12 21:19:48,614 Epoch  20 Step:    11500 Batch Loss:     2.259850 Tokens per Sec:    11408, Lr: 0.000300\n",
            "2020-07-12 21:20:06,981 Epoch  20 Step:    11600 Batch Loss:     2.186822 Tokens per Sec:    11156, Lr: 0.000300\n",
            "2020-07-12 21:20:13,477 Epoch  20: total training loss 1357.13\n",
            "2020-07-12 21:20:13,478 EPOCH 21\n",
            "2020-07-12 21:20:25,606 Epoch  21 Step:    11700 Batch Loss:     2.087318 Tokens per Sec:    11080, Lr: 0.000300\n",
            "2020-07-12 21:20:44,269 Epoch  21 Step:    11800 Batch Loss:     1.906124 Tokens per Sec:    11319, Lr: 0.000300\n",
            "2020-07-12 21:21:02,649 Epoch  21 Step:    11900 Batch Loss:     2.099954 Tokens per Sec:    11268, Lr: 0.000300\n",
            "2020-07-12 21:21:21,091 Epoch  21 Step:    12000 Batch Loss:     2.294059 Tokens per Sec:    11470, Lr: 0.000300\n",
            "2020-07-12 21:21:52,392 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:21:52,393 Saving new checkpoint.\n",
            "2020-07-12 21:21:53,628 Example #0\n",
            "2020-07-12 21:21:53,629 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:21:53,629 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:21:53,629 \tHypothesis: Tinali kugona katundu wathu pa malo athu amene tinali kukumana nao m’dziko lonse\n",
            "2020-07-12 21:21:53,629 Example #1\n",
            "2020-07-12 21:21:53,630 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:21:53,630 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:21:53,630 \tHypothesis: Iye angafunike kucita zinthu zimene anacita kwa zaka zambili .\n",
            "2020-07-12 21:21:53,630 Example #2\n",
            "2020-07-12 21:21:53,630 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:21:53,630 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:21:53,631 \tHypothesis: Kodi zocitika za Mose zinasonkhezela mau a Paulo pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:21:53,631 Example #3\n",
            "2020-07-12 21:21:53,631 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:21:53,631 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:21:53,631 \tHypothesis: Tili na Ciyembekezo Copambana , July\n",
            "2020-07-12 21:21:53,632 Validation result (greedy) at epoch  21, step    12000: bleu:  12.65, loss: 56646.6211, ppl:   9.8236, duration: 32.5401s\n",
            "2020-07-12 21:22:12,422 Epoch  21 Step:    12100 Batch Loss:     2.145493 Tokens per Sec:    11082, Lr: 0.000300\n",
            "2020-07-12 21:22:30,791 Epoch  21 Step:    12200 Batch Loss:     2.475222 Tokens per Sec:    11296, Lr: 0.000300\n",
            "2020-07-12 21:22:33,189 Epoch  21: total training loss 1322.48\n",
            "2020-07-12 21:22:33,190 EPOCH 22\n",
            "2020-07-12 21:22:49,387 Epoch  22 Step:    12300 Batch Loss:     2.111602 Tokens per Sec:    11117, Lr: 0.000300\n",
            "2020-07-12 21:23:07,757 Epoch  22 Step:    12400 Batch Loss:     2.236841 Tokens per Sec:    11236, Lr: 0.000300\n",
            "2020-07-12 21:23:26,279 Epoch  22 Step:    12500 Batch Loss:     2.356989 Tokens per Sec:    11113, Lr: 0.000300\n",
            "2020-07-12 21:23:44,949 Epoch  22 Step:    12600 Batch Loss:     2.198585 Tokens per Sec:    11457, Lr: 0.000300\n",
            "2020-07-12 21:24:03,276 Epoch  22 Step:    12700 Batch Loss:     2.277119 Tokens per Sec:    11271, Lr: 0.000300\n",
            "2020-07-12 21:24:20,559 Epoch  22: total training loss 1309.67\n",
            "2020-07-12 21:24:20,559 EPOCH 23\n",
            "2020-07-12 21:24:21,723 Epoch  23 Step:    12800 Batch Loss:     2.211040 Tokens per Sec:    11522, Lr: 0.000300\n",
            "2020-07-12 21:24:39,921 Epoch  23 Step:    12900 Batch Loss:     2.081254 Tokens per Sec:    10983, Lr: 0.000300\n",
            "2020-07-12 21:24:58,378 Epoch  23 Step:    13000 Batch Loss:     2.620756 Tokens per Sec:    11204, Lr: 0.000300\n",
            "2020-07-12 21:25:24,313 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:25:24,314 Saving new checkpoint.\n",
            "2020-07-12 21:25:25,485 Example #0\n",
            "2020-07-12 21:25:25,486 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:25:25,486 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:25:25,486 \tHypothesis: Anthu athu anali kutilimbikitsa kwambili padziko lonse lapansi\n",
            "2020-07-12 21:25:25,487 Example #1\n",
            "2020-07-12 21:25:25,487 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:25:25,487 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:25:25,487 \tHypothesis: Iye angaganize kuti anali ndi vuto laciwelewele , ngakhale kuti zaka zambili zapitazo .\n",
            "2020-07-12 21:25:25,487 Example #2\n",
            "2020-07-12 21:25:25,488 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:25:25,488 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:25:25,488 \tHypothesis: Kodi zocitika za Mose zinathandiza bwanji masiku ano ?\n",
            "2020-07-12 21:25:25,488 Example #3\n",
            "2020-07-12 21:25:25,488 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:25:25,489 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:25:25,489 \tHypothesis: Ndani Kuti Kuti Kuti Kuti Mumulile Uthenga Wabwino , July\n",
            "2020-07-12 21:25:25,489 Validation result (greedy) at epoch  23, step    13000: bleu:  13.00, loss: 55712.6133, ppl:   9.4604, duration: 27.1106s\n",
            "2020-07-12 21:25:44,251 Epoch  23 Step:    13100 Batch Loss:     2.018826 Tokens per Sec:    11284, Lr: 0.000300\n",
            "2020-07-12 21:26:02,419 Epoch  23 Step:    13200 Batch Loss:     2.134311 Tokens per Sec:    11186, Lr: 0.000300\n",
            "2020-07-12 21:26:21,019 Epoch  23 Step:    13300 Batch Loss:     2.013390 Tokens per Sec:    11224, Lr: 0.000300\n",
            "2020-07-12 21:26:35,650 Epoch  23: total training loss 1304.75\n",
            "2020-07-12 21:26:35,651 EPOCH 24\n",
            "2020-07-12 21:26:39,568 Epoch  24 Step:    13400 Batch Loss:     1.963058 Tokens per Sec:    10727, Lr: 0.000300\n",
            "2020-07-12 21:26:58,041 Epoch  24 Step:    13500 Batch Loss:     2.269552 Tokens per Sec:    11266, Lr: 0.000300\n",
            "2020-07-12 21:27:16,574 Epoch  24 Step:    13600 Batch Loss:     1.387344 Tokens per Sec:    11366, Lr: 0.000300\n",
            "2020-07-12 21:27:35,089 Epoch  24 Step:    13700 Batch Loss:     1.838774 Tokens per Sec:    11199, Lr: 0.000300\n",
            "2020-07-12 21:27:53,492 Epoch  24 Step:    13800 Batch Loss:     2.099367 Tokens per Sec:    11160, Lr: 0.000300\n",
            "2020-07-12 21:28:11,929 Epoch  24 Step:    13900 Batch Loss:     2.407192 Tokens per Sec:    11325, Lr: 0.000300\n",
            "2020-07-12 21:28:23,198 Epoch  24: total training loss 1284.61\n",
            "2020-07-12 21:28:23,198 EPOCH 25\n",
            "2020-07-12 21:28:30,374 Epoch  25 Step:    14000 Batch Loss:     2.420966 Tokens per Sec:    11092, Lr: 0.000300\n",
            "2020-07-12 21:28:59,782 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:28:59,782 Saving new checkpoint.\n",
            "2020-07-12 21:29:01,000 Example #0\n",
            "2020-07-12 21:29:01,001 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:29:01,001 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:29:01,001 \tHypothesis: Tinali kulalikila m’madela athu amene tinali kutilengamo mbali m’madela akutali\n",
            "2020-07-12 21:29:01,002 Example #1\n",
            "2020-07-12 21:29:01,002 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:29:01,002 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:29:01,003 \tHypothesis: Iye angafune kuti aziimba mlandu cifukwa ca zimene anacita kwa zaka zambili .\n",
            "2020-07-12 21:29:01,003 Example #2\n",
            "2020-07-12 21:29:01,003 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:29:01,003 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:29:01,003 \tHypothesis: Kodi zimene Mose anacita pa zocitika za m’nthawi ya Mose zimasonkhezela mau a Paulo pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:29:01,004 Example #3\n",
            "2020-07-12 21:29:01,004 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:29:01,004 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:29:01,004 \tHypothesis: Tizisangalala Kudziŵa Coonadi Cimene Cimapatsa , July\n",
            "2020-07-12 21:29:01,005 Validation result (greedy) at epoch  25, step    14000: bleu:  13.34, loss: 54857.1328, ppl:   9.1395, duration: 30.6306s\n",
            "2020-07-12 21:29:19,687 Epoch  25 Step:    14100 Batch Loss:     2.246745 Tokens per Sec:    11063, Lr: 0.000300\n",
            "2020-07-12 21:29:38,428 Epoch  25 Step:    14200 Batch Loss:     2.616163 Tokens per Sec:    11016, Lr: 0.000300\n",
            "2020-07-12 21:29:57,088 Epoch  25 Step:    14300 Batch Loss:     2.052552 Tokens per Sec:    11069, Lr: 0.000300\n",
            "2020-07-12 21:30:15,744 Epoch  25 Step:    14400 Batch Loss:     2.588664 Tokens per Sec:    11401, Lr: 0.000300\n",
            "2020-07-12 21:30:34,159 Epoch  25 Step:    14500 Batch Loss:     2.537948 Tokens per Sec:    11180, Lr: 0.000300\n",
            "2020-07-12 21:30:41,916 Epoch  25: total training loss 1265.02\n",
            "2020-07-12 21:30:41,916 EPOCH 26\n",
            "2020-07-12 21:30:52,745 Epoch  26 Step:    14600 Batch Loss:     2.163159 Tokens per Sec:    11315, Lr: 0.000300\n",
            "2020-07-12 21:31:11,302 Epoch  26 Step:    14700 Batch Loss:     2.121996 Tokens per Sec:    11139, Lr: 0.000300\n",
            "2020-07-12 21:31:29,805 Epoch  26 Step:    14800 Batch Loss:     2.149545 Tokens per Sec:    11109, Lr: 0.000300\n",
            "2020-07-12 21:31:48,487 Epoch  26 Step:    14900 Batch Loss:     2.132342 Tokens per Sec:    11199, Lr: 0.000300\n",
            "2020-07-12 21:32:07,100 Epoch  26 Step:    15000 Batch Loss:     2.116457 Tokens per Sec:    11188, Lr: 0.000300\n",
            "2020-07-12 21:32:29,659 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:32:29,659 Saving new checkpoint.\n",
            "2020-07-12 21:32:30,833 Example #0\n",
            "2020-07-12 21:32:30,834 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:32:30,834 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:32:30,834 \tHypothesis: Tinali kugula zakudya za anthu amene tinali kutitenga dziko lapansi\n",
            "2020-07-12 21:32:30,835 Example #1\n",
            "2020-07-12 21:32:30,836 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:32:30,836 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:32:30,836 \tHypothesis: Iye anali ndi maganizo olakwika amene anali nao kale , ngakhale zaka zambili .\n",
            "2020-07-12 21:32:30,837 Example #2\n",
            "2020-07-12 21:32:30,837 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:32:30,837 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:32:30,837 \tHypothesis: Kodi mau a Mose a m’nthawi ya Mose anali kuimila ciani ?\n",
            "2020-07-12 21:32:30,837 Example #3\n",
            "2020-07-12 21:32:30,838 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:32:30,838 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:32:30,838 \tHypothesis: Tili ndi Ciyembekezo Cabwino , July\n",
            "2020-07-12 21:32:30,838 Validation result (greedy) at epoch  26, step    15000: bleu:  13.85, loss: 53885.8867, ppl:   8.7884, duration: 23.7381s\n",
            "2020-07-12 21:32:49,432 Epoch  26 Step:    15100 Batch Loss:     2.359765 Tokens per Sec:    11129, Lr: 0.000300\n",
            "2020-07-12 21:32:53,707 Epoch  26: total training loss 1251.62\n",
            "2020-07-12 21:32:53,707 EPOCH 27\n",
            "2020-07-12 21:33:07,862 Epoch  27 Step:    15200 Batch Loss:     1.612871 Tokens per Sec:    11248, Lr: 0.000300\n",
            "2020-07-12 21:33:26,770 Epoch  27 Step:    15300 Batch Loss:     2.410092 Tokens per Sec:    11272, Lr: 0.000300\n",
            "2020-07-12 21:33:45,135 Epoch  27 Step:    15400 Batch Loss:     2.515933 Tokens per Sec:    11132, Lr: 0.000300\n",
            "2020-07-12 21:34:03,330 Epoch  27 Step:    15500 Batch Loss:     2.067173 Tokens per Sec:    11112, Lr: 0.000300\n",
            "2020-07-12 21:34:21,547 Epoch  27 Step:    15600 Batch Loss:     2.100986 Tokens per Sec:    11403, Lr: 0.000300\n",
            "2020-07-12 21:34:39,988 Epoch  27 Step:    15700 Batch Loss:     2.350737 Tokens per Sec:    11228, Lr: 0.000300\n",
            "2020-07-12 21:34:41,111 Epoch  27: total training loss 1242.77\n",
            "2020-07-12 21:34:41,112 EPOCH 28\n",
            "2020-07-12 21:34:58,730 Epoch  28 Step:    15800 Batch Loss:     1.252520 Tokens per Sec:    11290, Lr: 0.000300\n",
            "2020-07-12 21:35:17,218 Epoch  28 Step:    15900 Batch Loss:     2.261224 Tokens per Sec:    11266, Lr: 0.000300\n",
            "2020-07-12 21:35:35,839 Epoch  28 Step:    16000 Batch Loss:     2.414489 Tokens per Sec:    11158, Lr: 0.000300\n",
            "2020-07-12 21:36:04,598 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:36:04,599 Saving new checkpoint.\n",
            "2020-07-12 21:36:05,824 Example #0\n",
            "2020-07-12 21:36:05,825 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:36:05,825 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:36:05,825 \tHypothesis: Tinali kusiyana kwambili ndi anthu a m’dzikoli\n",
            "2020-07-12 21:36:05,825 Example #1\n",
            "2020-07-12 21:36:05,826 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:36:05,826 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:36:05,826 \tHypothesis: Iye angaganize kuti anali ndi vuto laciwelewele , ndipo anali ndi zaka zambili .\n",
            "2020-07-12 21:36:05,826 Example #2\n",
            "2020-07-12 21:36:05,827 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:36:05,827 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:36:05,827 \tHypothesis: Kodi zocitika za m’nthawi ya Mose zinasonyeza ciani pa mau a Paulo a pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:36:05,827 Example #3\n",
            "2020-07-12 21:36:05,828 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:36:05,828 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:36:05,828 \tHypothesis: Tili ndi Ciyembekezo Cimene Cimene Cimatipatsa , July\n",
            "2020-07-12 21:36:05,828 Validation result (greedy) at epoch  28, step    16000: bleu:  14.68, loss: 53204.4258, ppl:   8.5501, duration: 29.9881s\n",
            "2020-07-12 21:36:25,203 Epoch  28 Step:    16100 Batch Loss:     2.462816 Tokens per Sec:    10942, Lr: 0.000300\n",
            "2020-07-12 21:36:43,579 Epoch  28 Step:    16200 Batch Loss:     2.299822 Tokens per Sec:    11067, Lr: 0.000300\n",
            "2020-07-12 21:36:59,264 Epoch  28: total training loss 1222.03\n",
            "2020-07-12 21:36:59,264 EPOCH 29\n",
            "2020-07-12 21:37:02,248 Epoch  29 Step:    16300 Batch Loss:     1.654314 Tokens per Sec:    10759, Lr: 0.000300\n",
            "2020-07-12 21:37:20,690 Epoch  29 Step:    16400 Batch Loss:     2.241484 Tokens per Sec:    11366, Lr: 0.000300\n",
            "2020-07-12 21:37:39,530 Epoch  29 Step:    16500 Batch Loss:     2.279921 Tokens per Sec:    10965, Lr: 0.000300\n",
            "2020-07-12 21:37:58,074 Epoch  29 Step:    16600 Batch Loss:     2.222358 Tokens per Sec:    11352, Lr: 0.000300\n",
            "2020-07-12 21:38:16,390 Epoch  29 Step:    16700 Batch Loss:     2.419949 Tokens per Sec:    11164, Lr: 0.000300\n",
            "2020-07-12 21:38:34,734 Epoch  29 Step:    16800 Batch Loss:     2.223499 Tokens per Sec:    11140, Lr: 0.000300\n",
            "2020-07-12 21:38:47,135 Epoch  29: total training loss 1219.28\n",
            "2020-07-12 21:38:47,135 EPOCH 30\n",
            "2020-07-12 21:38:53,392 Epoch  30 Step:    16900 Batch Loss:     1.924732 Tokens per Sec:    11642, Lr: 0.000300\n",
            "2020-07-12 21:39:11,811 Epoch  30 Step:    17000 Batch Loss:     2.250193 Tokens per Sec:    11172, Lr: 0.000300\n",
            "2020-07-12 21:39:32,606 Example #0\n",
            "2020-07-12 21:39:32,607 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:39:32,607 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:39:32,607 \tHypothesis: Pokhala anthu amene tinali kusonkhana ndi anthu amene tinali kutitenga padziko lonse lapansi\n",
            "2020-07-12 21:39:32,608 Example #1\n",
            "2020-07-12 21:39:32,608 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:39:32,608 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:39:32,608 \tHypothesis: Iye angaone kuti anali ndi vuto loipa kwambili kwa zaka zambili .\n",
            "2020-07-12 21:39:32,608 Example #2\n",
            "2020-07-12 21:39:32,609 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:39:32,609 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:39:32,609 \tHypothesis: Ndi zocitika ziti zimene Mose anapeleka pamene mau a Paulo analembedwa pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:39:32,609 Example #3\n",
            "2020-07-12 21:39:32,610 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:39:32,610 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:39:32,610 \tHypothesis: Palibe Cikondi Cimene Cimene Cisomo Cili pa July\n",
            "2020-07-12 21:39:32,610 Validation result (greedy) at epoch  30, step    17000: bleu:  14.18, loss: 53287.4102, ppl:   8.5788, duration: 20.7991s\n",
            "2020-07-12 21:39:51,069 Epoch  30 Step:    17100 Batch Loss:     1.996659 Tokens per Sec:    11078, Lr: 0.000300\n",
            "2020-07-12 21:40:09,724 Epoch  30 Step:    17200 Batch Loss:     2.441917 Tokens per Sec:    11321, Lr: 0.000300\n",
            "2020-07-12 21:40:28,184 Epoch  30 Step:    17300 Batch Loss:     2.245591 Tokens per Sec:    11227, Lr: 0.000300\n",
            "2020-07-12 21:40:46,651 Epoch  30 Step:    17400 Batch Loss:     1.991428 Tokens per Sec:    11026, Lr: 0.000300\n",
            "2020-07-12 21:40:55,549 Epoch  30: total training loss 1207.48\n",
            "2020-07-12 21:40:55,550 EPOCH 31\n",
            "2020-07-12 21:41:05,291 Epoch  31 Step:    17500 Batch Loss:     1.878195 Tokens per Sec:    11394, Lr: 0.000300\n",
            "2020-07-12 21:41:23,650 Epoch  31 Step:    17600 Batch Loss:     2.384914 Tokens per Sec:    11117, Lr: 0.000300\n",
            "2020-07-12 21:41:42,167 Epoch  31 Step:    17700 Batch Loss:     2.224703 Tokens per Sec:    11077, Lr: 0.000300\n",
            "2020-07-12 21:42:00,597 Epoch  31 Step:    17800 Batch Loss:     2.243607 Tokens per Sec:    11203, Lr: 0.000300\n",
            "2020-07-12 21:42:19,408 Epoch  31 Step:    17900 Batch Loss:     2.156587 Tokens per Sec:    11277, Lr: 0.000300\n",
            "2020-07-12 21:42:37,706 Epoch  31 Step:    18000 Batch Loss:     1.582895 Tokens per Sec:    11132, Lr: 0.000300\n",
            "2020-07-12 21:43:06,375 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:43:06,375 Saving new checkpoint.\n",
            "2020-07-12 21:43:07,651 Example #0\n",
            "2020-07-12 21:43:07,651 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:43:07,652 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:43:07,652 \tHypothesis: Tinali kugaŵila anthu ambili ku malo osiyanasiyana amene tinali nao padziko lonse\n",
            "2020-07-12 21:43:07,652 Example #1\n",
            "2020-07-12 21:43:07,652 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:43:07,652 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:43:07,653 \tHypothesis: Iye anali kufunitsitsa kuvulaza zinthu zimene anali kucita , ngakhale zaka zambili .\n",
            "2020-07-12 21:43:07,653 Example #2\n",
            "2020-07-12 21:43:07,653 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:43:07,653 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:43:07,654 \tHypothesis: Kodi zocitika za m’nthawi ya Mose zinasonyeza ciani pa mau a Paulo a pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:43:07,654 Example #3\n",
            "2020-07-12 21:43:07,654 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:43:07,654 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:43:07,654 \tHypothesis: Tengelani Citsanzo Cake Cisomo Conse , July\n",
            "2020-07-12 21:43:07,654 Validation result (greedy) at epoch  31, step    18000: bleu:  14.77, loss: 52131.8594, ppl:   8.1881, duration: 29.9481s\n",
            "2020-07-12 21:43:13,475 Epoch  31: total training loss 1202.09\n",
            "2020-07-12 21:43:13,476 EPOCH 32\n",
            "2020-07-12 21:43:26,535 Epoch  32 Step:    18100 Batch Loss:     2.240680 Tokens per Sec:    10869, Lr: 0.000300\n",
            "2020-07-12 21:43:44,600 Epoch  32 Step:    18200 Batch Loss:     1.937984 Tokens per Sec:    11189, Lr: 0.000300\n",
            "2020-07-12 21:44:03,105 Epoch  32 Step:    18300 Batch Loss:     1.852296 Tokens per Sec:    11308, Lr: 0.000300\n",
            "2020-07-12 21:44:21,722 Epoch  32 Step:    18400 Batch Loss:     2.168491 Tokens per Sec:    11348, Lr: 0.000300\n",
            "2020-07-12 21:44:40,145 Epoch  32 Step:    18500 Batch Loss:     2.244599 Tokens per Sec:    11134, Lr: 0.000300\n",
            "2020-07-12 21:44:58,643 Epoch  32 Step:    18600 Batch Loss:     2.211453 Tokens per Sec:    11243, Lr: 0.000300\n",
            "2020-07-12 21:45:01,378 Epoch  32: total training loss 1191.61\n",
            "2020-07-12 21:45:01,379 EPOCH 33\n",
            "2020-07-12 21:45:17,130 Epoch  33 Step:    18700 Batch Loss:     2.181301 Tokens per Sec:    11121, Lr: 0.000300\n",
            "2020-07-12 21:45:35,671 Epoch  33 Step:    18800 Batch Loss:     2.112555 Tokens per Sec:    11139, Lr: 0.000300\n",
            "2020-07-12 21:45:54,058 Epoch  33 Step:    18900 Batch Loss:     1.560102 Tokens per Sec:    11092, Lr: 0.000300\n",
            "2020-07-12 21:46:12,215 Epoch  33 Step:    19000 Batch Loss:     2.184921 Tokens per Sec:    11260, Lr: 0.000300\n",
            "2020-07-12 21:46:31,492 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:46:31,492 Saving new checkpoint.\n",
            "2020-07-12 21:46:32,684 Example #0\n",
            "2020-07-12 21:46:32,685 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:46:32,685 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:46:32,685 \tHypothesis: Tinali kugaŵila mabuku athu pa sitima ya anthu amene tinali kutilengako mbali m’dzikolo\n",
            "2020-07-12 21:46:32,685 Example #1\n",
            "2020-07-12 21:46:32,686 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:46:32,686 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:46:32,686 \tHypothesis: Iye anali kufunitsitsa kuteteza zinthu zina zimene anali kucita m’nthawi yakale , ngakhale kuti zaka zambili zapitazo .\n",
            "2020-07-12 21:46:32,686 Example #2\n",
            "2020-07-12 21:46:32,686 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:46:32,687 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:46:32,687 \tHypothesis: Kodi zocitika za m’nthawi ya Mose zimasonkhezela mau a Paulo pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:46:32,687 Example #3\n",
            "2020-07-12 21:46:32,687 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:46:32,687 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:46:32,688 \tHypothesis: Tengelani Cikondi Cosatha , July\n",
            "2020-07-12 21:46:32,688 Validation result (greedy) at epoch  33, step    19000: bleu:  15.43, loss: 52101.8984, ppl:   8.1782, duration: 20.4726s\n",
            "2020-07-12 21:46:51,651 Epoch  33 Step:    19100 Batch Loss:     2.147182 Tokens per Sec:    11138, Lr: 0.000300\n",
            "2020-07-12 21:47:09,754 Epoch  33: total training loss 1180.51\n",
            "2020-07-12 21:47:09,754 EPOCH 34\n",
            "2020-07-12 21:47:10,167 Epoch  34 Step:    19200 Batch Loss:     1.254606 Tokens per Sec:     7697, Lr: 0.000300\n",
            "2020-07-12 21:47:28,747 Epoch  34 Step:    19300 Batch Loss:     2.237213 Tokens per Sec:    11272, Lr: 0.000300\n",
            "2020-07-12 21:47:47,380 Epoch  34 Step:    19400 Batch Loss:     1.689342 Tokens per Sec:    11192, Lr: 0.000300\n",
            "2020-07-12 21:48:05,737 Epoch  34 Step:    19500 Batch Loss:     1.540750 Tokens per Sec:    11147, Lr: 0.000300\n",
            "2020-07-12 21:48:24,504 Epoch  34 Step:    19600 Batch Loss:     2.257375 Tokens per Sec:    11127, Lr: 0.000300\n",
            "2020-07-12 21:48:42,894 Epoch  34 Step:    19700 Batch Loss:     1.333181 Tokens per Sec:    11175, Lr: 0.000300\n",
            "2020-07-12 21:48:57,675 Epoch  34: total training loss 1170.53\n",
            "2020-07-12 21:48:57,676 EPOCH 35\n",
            "2020-07-12 21:49:01,372 Epoch  35 Step:    19800 Batch Loss:     1.743930 Tokens per Sec:    10966, Lr: 0.000300\n",
            "2020-07-12 21:49:19,865 Epoch  35 Step:    19900 Batch Loss:     1.776612 Tokens per Sec:    11287, Lr: 0.000300\n",
            "2020-07-12 21:49:38,451 Epoch  35 Step:    20000 Batch Loss:     1.984108 Tokens per Sec:    11249, Lr: 0.000300\n",
            "2020-07-12 21:49:56,902 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:49:56,903 Saving new checkpoint.\n",
            "2020-07-12 21:49:58,122 Example #0\n",
            "2020-07-12 21:49:58,123 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:49:58,123 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:49:58,123 \tHypothesis: Tinali kugaŵila ku malo osiyanasiyana kuti tikaloŵe m’dziko latsopano\n",
            "2020-07-12 21:49:58,123 Example #1\n",
            "2020-07-12 21:49:58,124 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:49:58,124 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:49:58,124 \tHypothesis: Iye angaganize kuti anali ndi nkhawa kwambili zokhudza zinthu zimene anacita , ngakhale zaka zambili .\n",
            "2020-07-12 21:49:58,124 Example #2\n",
            "2020-07-12 21:49:58,125 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:49:58,125 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:49:58,125 \tHypothesis: Kodi mau a Paulo a pa 2 Timoteyo 2 : 19 anali ndi zinthu ziti zimene zinacitika m’nthawi ya Mose ?\n",
            "2020-07-12 21:49:58,125 Example #3\n",
            "2020-07-12 21:49:58,125 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:49:58,125 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:49:58,126 \tHypothesis: Tili ndi Ciyembekezo Cabwino Cimene Cimatsogolela , July\n",
            "2020-07-12 21:49:58,126 Validation result (greedy) at epoch  35, step    20000: bleu:  15.35, loss: 51433.5742, ppl:   7.9607, duration: 19.6741s\n",
            "2020-07-12 21:50:17,304 Epoch  35 Step:    20100 Batch Loss:     2.221691 Tokens per Sec:    10735, Lr: 0.000300\n",
            "2020-07-12 21:50:35,839 Epoch  35 Step:    20200 Batch Loss:     2.187058 Tokens per Sec:    11372, Lr: 0.000300\n",
            "2020-07-12 21:50:54,215 Epoch  35 Step:    20300 Batch Loss:     2.000558 Tokens per Sec:    11096, Lr: 0.000300\n",
            "2020-07-12 21:51:05,351 Epoch  35: total training loss 1156.85\n",
            "2020-07-12 21:51:05,352 EPOCH 36\n",
            "2020-07-12 21:51:12,908 Epoch  36 Step:    20400 Batch Loss:     2.145525 Tokens per Sec:    11354, Lr: 0.000300\n",
            "2020-07-12 21:51:31,419 Epoch  36 Step:    20500 Batch Loss:     2.117764 Tokens per Sec:    11099, Lr: 0.000300\n",
            "2020-07-12 21:51:49,883 Epoch  36 Step:    20600 Batch Loss:     2.139196 Tokens per Sec:    11148, Lr: 0.000300\n",
            "2020-07-12 21:52:08,267 Epoch  36 Step:    20700 Batch Loss:     1.803431 Tokens per Sec:    11275, Lr: 0.000300\n",
            "2020-07-12 21:52:26,560 Epoch  36 Step:    20800 Batch Loss:     2.002237 Tokens per Sec:    11100, Lr: 0.000300\n",
            "2020-07-12 21:52:44,998 Epoch  36 Step:    20900 Batch Loss:     2.066098 Tokens per Sec:    11177, Lr: 0.000300\n",
            "2020-07-12 21:52:53,298 Epoch  36: total training loss 1158.20\n",
            "2020-07-12 21:52:53,299 EPOCH 37\n",
            "2020-07-12 21:53:03,462 Epoch  37 Step:    21000 Batch Loss:     1.876578 Tokens per Sec:    11149, Lr: 0.000300\n",
            "2020-07-12 21:53:22,061 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:53:22,062 Saving new checkpoint.\n",
            "2020-07-12 21:53:23,284 Example #0\n",
            "2020-07-12 21:53:23,285 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:53:23,285 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:53:23,285 \tHypothesis: Tinali kugaŵila nchito yoyang’anila dela imene tinali kuyembekezela kudziko lonse\n",
            "2020-07-12 21:53:23,285 Example #1\n",
            "2020-07-12 21:53:23,286 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:53:23,286 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:53:23,286 \tHypothesis: Iye anali ndi nkhawa kwambili cakuti anali atayamba kale , ngakhale zaka zambili zapitazo .\n",
            "2020-07-12 21:53:23,286 Example #2\n",
            "2020-07-12 21:53:23,287 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:53:23,287 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:53:23,287 \tHypothesis: Kodi mau a Mose anali pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:53:23,287 Example #3\n",
            "2020-07-12 21:53:23,288 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:53:23,288 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:53:23,288 \tHypothesis: Tili na Ciyembekezo Cabwino Cimene Cisomo Cimatsogolela , July\n",
            "2020-07-12 21:53:23,288 Validation result (greedy) at epoch  37, step    21000: bleu:  15.83, loss: 51151.2070, ppl:   7.8706, duration: 19.8251s\n",
            "2020-07-12 21:53:42,088 Epoch  37 Step:    21100 Batch Loss:     2.115872 Tokens per Sec:    11126, Lr: 0.000300\n",
            "2020-07-12 21:54:00,599 Epoch  37 Step:    21200 Batch Loss:     2.029389 Tokens per Sec:    11116, Lr: 0.000300\n",
            "2020-07-12 21:54:19,420 Epoch  37 Step:    21300 Batch Loss:     2.030703 Tokens per Sec:    11442, Lr: 0.000300\n",
            "2020-07-12 21:54:37,743 Epoch  37 Step:    21400 Batch Loss:     2.162308 Tokens per Sec:    11304, Lr: 0.000300\n",
            "2020-07-12 21:54:56,431 Epoch  37 Step:    21500 Batch Loss:     2.328736 Tokens per Sec:    11388, Lr: 0.000300\n",
            "2020-07-12 21:55:00,472 Epoch  37: total training loss 1133.69\n",
            "2020-07-12 21:55:00,472 EPOCH 38\n",
            "2020-07-12 21:55:14,795 Epoch  38 Step:    21600 Batch Loss:     1.523328 Tokens per Sec:    10858, Lr: 0.000300\n",
            "2020-07-12 21:55:33,693 Epoch  38 Step:    21700 Batch Loss:     2.188564 Tokens per Sec:    11432, Lr: 0.000300\n",
            "2020-07-12 21:55:52,043 Epoch  38 Step:    21800 Batch Loss:     2.258023 Tokens per Sec:    11232, Lr: 0.000300\n",
            "2020-07-12 21:56:10,512 Epoch  38 Step:    21900 Batch Loss:     2.049119 Tokens per Sec:    11190, Lr: 0.000300\n",
            "2020-07-12 21:56:29,208 Epoch  38 Step:    22000 Batch Loss:     2.075769 Tokens per Sec:    11375, Lr: 0.000300\n",
            "2020-07-12 21:56:55,370 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 21:56:55,370 Saving new checkpoint.\n",
            "2020-07-12 21:56:56,543 Example #0\n",
            "2020-07-12 21:56:56,544 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 21:56:56,544 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 21:56:56,544 \tHypothesis: Tinali kusonkhana ndi anthu ambili amene tinali kukhala nao m’dziko lathu\n",
            "2020-07-12 21:56:56,544 Example #1\n",
            "2020-07-12 21:56:56,545 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 21:56:56,545 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 21:56:56,545 \tHypothesis: Iye anali kufunitsitsa kuteteza zinthu zina zimene anacita , ngakhale zaka zambili zapitazo .\n",
            "2020-07-12 21:56:56,545 Example #2\n",
            "2020-07-12 21:56:56,545 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 21:56:56,545 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:56:56,546 \tHypothesis: Ndi zocitika ziti zimene zinacitikila Mose pamene mau a Paulo a pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 21:56:56,546 Example #3\n",
            "2020-07-12 21:56:56,546 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 21:56:56,546 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 21:56:56,546 \tHypothesis: Tengelani Ciyembekezo Cabwino Cimene Cili pa Cisomo , July\n",
            "2020-07-12 21:56:56,547 Validation result (greedy) at epoch  38, step    22000: bleu:  15.78, loss: 50394.4102, ppl:   7.6340, duration: 27.3318s\n",
            "2020-07-12 21:57:15,159 Epoch  38 Step:    22100 Batch Loss:     1.969265 Tokens per Sec:    10923, Lr: 0.000300\n",
            "2020-07-12 21:57:15,721 Epoch  38: total training loss 1131.84\n",
            "2020-07-12 21:57:15,721 EPOCH 39\n",
            "2020-07-12 21:57:33,739 Epoch  39 Step:    22200 Batch Loss:     1.735266 Tokens per Sec:    11283, Lr: 0.000300\n",
            "2020-07-12 21:57:52,243 Epoch  39 Step:    22300 Batch Loss:     2.093395 Tokens per Sec:    11086, Lr: 0.000300\n",
            "2020-07-12 21:58:10,911 Epoch  39 Step:    22400 Batch Loss:     1.781759 Tokens per Sec:    11207, Lr: 0.000300\n",
            "2020-07-12 21:58:29,487 Epoch  39 Step:    22500 Batch Loss:     1.843250 Tokens per Sec:    11430, Lr: 0.000300\n",
            "2020-07-12 21:58:48,064 Epoch  39 Step:    22600 Batch Loss:     2.179653 Tokens per Sec:    11112, Lr: 0.000300\n",
            "2020-07-12 21:59:03,220 Epoch  39: total training loss 1123.27\n",
            "2020-07-12 21:59:03,221 EPOCH 40\n",
            "2020-07-12 21:59:06,699 Epoch  40 Step:    22700 Batch Loss:     2.049891 Tokens per Sec:    11025, Lr: 0.000300\n",
            "2020-07-12 21:59:24,988 Epoch  40 Step:    22800 Batch Loss:     1.811621 Tokens per Sec:    11263, Lr: 0.000300\n",
            "2020-07-12 21:59:43,458 Epoch  40 Step:    22900 Batch Loss:     2.082726 Tokens per Sec:    11251, Lr: 0.000300\n",
            "2020-07-12 22:00:01,795 Epoch  40 Step:    23000 Batch Loss:     1.971813 Tokens per Sec:    11260, Lr: 0.000300\n",
            "2020-07-12 22:00:23,644 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:00:23,644 Saving new checkpoint.\n",
            "2020-07-12 22:00:24,843 Example #0\n",
            "2020-07-12 22:00:24,844 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:00:24,844 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:00:24,844 \tHypothesis: Tinali kusukulu athu ambili amene tinali kutitenga ku dziko lonse\n",
            "2020-07-12 22:00:24,844 Example #1\n",
            "2020-07-12 22:00:24,845 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:00:24,845 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:00:24,845 \tHypothesis: Mwina iye anali ndi maganizo olakwika ponena za zinthu zimene anacita , ngakhale zaka zambili .\n",
            "2020-07-12 22:00:24,846 Example #2\n",
            "2020-07-12 22:00:24,846 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:00:24,846 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:00:24,847 \tHypothesis: Ndi zocitika ziti zimene Mose anapeleka pamene Paulo analembedwa pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:00:24,847 Example #3\n",
            "2020-07-12 22:00:24,848 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:00:24,848 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:00:24,848 \tHypothesis: Tiziseŵenzetsa Uthenga Wabwino wa Uthenga Wabwino , July\n",
            "2020-07-12 22:00:24,848 Validation result (greedy) at epoch  40, step    23000: bleu:  16.34, loss: 50194.8047, ppl:   7.5727, duration: 23.0523s\n",
            "2020-07-12 22:00:43,690 Epoch  40 Step:    23100 Batch Loss:     2.083548 Tokens per Sec:    11131, Lr: 0.000300\n",
            "2020-07-12 22:01:01,894 Epoch  40 Step:    23200 Batch Loss:     2.167542 Tokens per Sec:    11281, Lr: 0.000300\n",
            "2020-07-12 22:01:13,658 Epoch  40: total training loss 1120.55\n",
            "2020-07-12 22:01:13,658 EPOCH 41\n",
            "2020-07-12 22:01:20,741 Epoch  41 Step:    23300 Batch Loss:     1.966002 Tokens per Sec:    11656, Lr: 0.000300\n",
            "2020-07-12 22:01:39,354 Epoch  41 Step:    23400 Batch Loss:     2.233330 Tokens per Sec:    11330, Lr: 0.000300\n",
            "2020-07-12 22:01:57,616 Epoch  41 Step:    23500 Batch Loss:     2.065772 Tokens per Sec:    11154, Lr: 0.000300\n",
            "2020-07-12 22:02:16,084 Epoch  41 Step:    23600 Batch Loss:     1.971395 Tokens per Sec:    11143, Lr: 0.000300\n",
            "2020-07-12 22:02:34,781 Epoch  41 Step:    23700 Batch Loss:     2.025626 Tokens per Sec:    11371, Lr: 0.000300\n",
            "2020-07-12 22:02:53,147 Epoch  41 Step:    23800 Batch Loss:     2.111092 Tokens per Sec:    10892, Lr: 0.000300\n",
            "2020-07-12 22:03:01,268 Epoch  41: total training loss 1111.13\n",
            "2020-07-12 22:03:01,268 EPOCH 42\n",
            "2020-07-12 22:03:11,680 Epoch  42 Step:    23900 Batch Loss:     1.773509 Tokens per Sec:    11092, Lr: 0.000300\n",
            "2020-07-12 22:03:30,104 Epoch  42 Step:    24000 Batch Loss:     1.953830 Tokens per Sec:    11281, Lr: 0.000300\n",
            "2020-07-12 22:03:51,841 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:03:51,841 Saving new checkpoint.\n",
            "2020-07-12 22:03:53,453 Example #0\n",
            "2020-07-12 22:03:53,454 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:03:53,454 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:03:53,454 \tHypothesis: Padziko lonse lapansi , anthu amene tinali kusamalila dziko lapansi\n",
            "2020-07-12 22:03:53,454 Example #1\n",
            "2020-07-12 22:03:53,455 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:03:53,455 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:03:53,455 \tHypothesis: Iye angafunike kukangana ndi munthu amene anacitapo zinthu zakale , ngakhale kuti zaka zambili zapitazo .\n",
            "2020-07-12 22:03:53,455 Example #2\n",
            "2020-07-12 22:03:53,455 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:03:53,456 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:03:53,456 \tHypothesis: Kodi mau a Paulo a pa 2 Timoteyo 2 : 19 , ati : 19 .\n",
            "2020-07-12 22:03:53,456 Example #3\n",
            "2020-07-12 22:03:53,456 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:03:53,456 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:03:53,457 \tHypothesis: Tizionetsa Kuti Mmene Mumalandila Cikondi Conse , July\n",
            "2020-07-12 22:03:53,457 Validation result (greedy) at epoch  42, step    24000: bleu:  16.58, loss: 50057.7031, ppl:   7.5310, duration: 23.3520s\n",
            "2020-07-12 22:04:12,268 Epoch  42 Step:    24100 Batch Loss:     2.251055 Tokens per Sec:    11104, Lr: 0.000300\n",
            "2020-07-12 22:04:30,993 Epoch  42 Step:    24200 Batch Loss:     2.124293 Tokens per Sec:    11434, Lr: 0.000300\n",
            "2020-07-12 22:04:49,415 Epoch  42 Step:    24300 Batch Loss:     1.837537 Tokens per Sec:    11014, Lr: 0.000300\n",
            "2020-07-12 22:05:07,639 Epoch  42 Step:    24400 Batch Loss:     1.755044 Tokens per Sec:    10997, Lr: 0.000300\n",
            "2020-07-12 22:05:12,681 Epoch  42: total training loss 1109.49\n",
            "2020-07-12 22:05:12,682 EPOCH 43\n",
            "2020-07-12 22:05:26,184 Epoch  43 Step:    24500 Batch Loss:     1.948527 Tokens per Sec:    10963, Lr: 0.000300\n",
            "2020-07-12 22:05:44,812 Epoch  43 Step:    24600 Batch Loss:     1.826027 Tokens per Sec:    11271, Lr: 0.000300\n",
            "2020-07-12 22:06:03,454 Epoch  43 Step:    24700 Batch Loss:     1.973153 Tokens per Sec:    11319, Lr: 0.000300\n",
            "2020-07-12 22:06:21,788 Epoch  43 Step:    24800 Batch Loss:     1.975532 Tokens per Sec:    11232, Lr: 0.000300\n",
            "2020-07-12 22:06:40,262 Epoch  43 Step:    24900 Batch Loss:     1.272901 Tokens per Sec:    11025, Lr: 0.000300\n",
            "2020-07-12 22:06:58,732 Epoch  43 Step:    25000 Batch Loss:     2.216368 Tokens per Sec:    11351, Lr: 0.000300\n",
            "2020-07-12 22:07:22,825 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:07:22,825 Saving new checkpoint.\n",
            "2020-07-12 22:07:24,023 Example #0\n",
            "2020-07-12 22:07:24,024 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:07:24,024 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:07:24,024 \tHypothesis: Tinali kugula makilomita a pa nyumba yathu imene tinali kutitenga padziko lonse lapansi\n",
            "2020-07-12 22:07:24,024 Example #1\n",
            "2020-07-12 22:07:24,024 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:07:24,025 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:07:24,025 \tHypothesis: Iye angagonje ndi kuteteza zinthu zina zimene anacita , ngakhale zaka zambili zapitazo .\n",
            "2020-07-12 22:07:24,025 Example #2\n",
            "2020-07-12 22:07:24,025 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:07:24,025 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:07:24,026 \tHypothesis: Kodi mau a Mose a pa 2 Timoteyo 2 : 19 amati ?\n",
            "2020-07-12 22:07:24,026 Example #3\n",
            "2020-07-12 22:07:24,026 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:07:24,026 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:07:24,026 \tHypothesis: Tiziyamikila Cikondi Cosatha , July\n",
            "2020-07-12 22:07:24,027 Validation result (greedy) at epoch  43, step    25000: bleu:  16.62, loss: 49733.6367, ppl:   7.4332, duration: 25.2938s\n",
            "2020-07-12 22:07:25,611 Epoch  43: total training loss 1100.98\n",
            "2020-07-12 22:07:25,611 EPOCH 44\n",
            "2020-07-12 22:07:43,008 Epoch  44 Step:    25100 Batch Loss:     1.881896 Tokens per Sec:    11029, Lr: 0.000300\n",
            "2020-07-12 22:08:01,573 Epoch  44 Step:    25200 Batch Loss:     1.957312 Tokens per Sec:    11001, Lr: 0.000300\n",
            "2020-07-12 22:08:20,036 Epoch  44 Step:    25300 Batch Loss:     2.177895 Tokens per Sec:    11360, Lr: 0.000300\n",
            "2020-07-12 22:08:38,535 Epoch  44 Step:    25400 Batch Loss:     1.982558 Tokens per Sec:    11313, Lr: 0.000300\n",
            "2020-07-12 22:08:57,047 Epoch  44 Step:    25500 Batch Loss:     1.973275 Tokens per Sec:    11074, Lr: 0.000300\n",
            "2020-07-12 22:09:13,600 Epoch  44: total training loss 1091.23\n",
            "2020-07-12 22:09:13,600 EPOCH 45\n",
            "2020-07-12 22:09:15,716 Epoch  45 Step:    25600 Batch Loss:     1.971170 Tokens per Sec:    11097, Lr: 0.000300\n",
            "2020-07-12 22:09:34,280 Epoch  45 Step:    25700 Batch Loss:     1.735779 Tokens per Sec:    11237, Lr: 0.000300\n",
            "2020-07-12 22:09:53,076 Epoch  45 Step:    25800 Batch Loss:     2.019991 Tokens per Sec:    11135, Lr: 0.000300\n",
            "2020-07-12 22:10:11,497 Epoch  45 Step:    25900 Batch Loss:     2.067264 Tokens per Sec:    11276, Lr: 0.000300\n",
            "2020-07-12 22:10:29,870 Epoch  45 Step:    26000 Batch Loss:     1.724764 Tokens per Sec:    11282, Lr: 0.000300\n",
            "2020-07-12 22:10:58,947 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:10:58,947 Saving new checkpoint.\n",
            "2020-07-12 22:11:00,070 Example #0\n",
            "2020-07-12 22:11:00,071 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:11:00,071 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:11:00,071 \tHypothesis: Nchito yathu inali yosamalila nyumba yathu imene tinali kutitenga padziko lonse lapansi\n",
            "2020-07-12 22:11:00,071 Example #1\n",
            "2020-07-12 22:11:00,072 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:11:00,072 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:11:00,072 \tHypothesis: Iye ayenela kuti anali ndi maganizo olakwika ponena za zimene anacita zaka zambili , ngakhale kuti anali ndi zaka zambili .\n",
            "2020-07-12 22:11:00,072 Example #2\n",
            "2020-07-12 22:11:00,073 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:11:00,073 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:11:00,073 \tHypothesis: Ndi zocitika ziti zimene Mose anapeleka pamene mau a Paulo a pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:11:00,073 Example #3\n",
            "2020-07-12 22:11:00,073 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:11:00,074 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:11:00,074 \tHypothesis: Pitilizani Kukonda Uthenga Wabwino wa Uthenga Wabwino , July\n",
            "2020-07-12 22:11:00,074 Validation result (greedy) at epoch  45, step    26000: bleu:  16.82, loss: 49328.4375, ppl:   7.3127, duration: 30.2034s\n",
            "2020-07-12 22:11:18,810 Epoch  45 Step:    26100 Batch Loss:     2.169725 Tokens per Sec:    10929, Lr: 0.000300\n",
            "2020-07-12 22:11:31,845 Epoch  45: total training loss 1088.19\n",
            "2020-07-12 22:11:31,845 EPOCH 46\n",
            "2020-07-12 22:11:37,251 Epoch  46 Step:    26200 Batch Loss:     1.817476 Tokens per Sec:    10952, Lr: 0.000300\n",
            "2020-07-12 22:11:55,657 Epoch  46 Step:    26300 Batch Loss:     2.044828 Tokens per Sec:    11302, Lr: 0.000300\n",
            "2020-07-12 22:12:14,261 Epoch  46 Step:    26400 Batch Loss:     1.758926 Tokens per Sec:    11395, Lr: 0.000300\n",
            "2020-07-12 22:12:32,741 Epoch  46 Step:    26500 Batch Loss:     1.957885 Tokens per Sec:    11173, Lr: 0.000300\n",
            "2020-07-12 22:12:51,217 Epoch  46 Step:    26600 Batch Loss:     1.730212 Tokens per Sec:    11167, Lr: 0.000300\n",
            "2020-07-12 22:13:09,649 Epoch  46 Step:    26700 Batch Loss:     2.047805 Tokens per Sec:    11242, Lr: 0.000300\n",
            "2020-07-12 22:13:19,165 Epoch  46: total training loss 1079.66\n",
            "2020-07-12 22:13:19,165 EPOCH 47\n",
            "2020-07-12 22:13:28,404 Epoch  47 Step:    26800 Batch Loss:     1.963683 Tokens per Sec:    11235, Lr: 0.000300\n",
            "2020-07-12 22:13:47,092 Epoch  47 Step:    26900 Batch Loss:     1.714284 Tokens per Sec:    11292, Lr: 0.000300\n",
            "2020-07-12 22:14:05,542 Epoch  47 Step:    27000 Batch Loss:     1.860049 Tokens per Sec:    11263, Lr: 0.000300\n",
            "2020-07-12 22:14:33,907 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:14:33,908 Saving new checkpoint.\n",
            "2020-07-12 22:14:35,080 Example #0\n",
            "2020-07-12 22:14:35,080 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:14:35,080 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:14:35,081 \tHypothesis: Pokhala nchito yapamwamba imene tinali kugwilila nchito , tinali kutitenga ku dziko lonse\n",
            "2020-07-12 22:14:35,081 Example #1\n",
            "2020-07-12 22:14:35,081 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:14:35,081 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:14:35,081 \tHypothesis: Iye ayenela kuti anali ndi nkhawa kwambili ponena za zinthu zimene anacita kwa zaka zambili .\n",
            "2020-07-12 22:14:35,081 Example #2\n",
            "2020-07-12 22:14:35,082 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:14:35,082 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:14:35,082 \tHypothesis: Ndi zocitika ziti zimene zinacitikila m’nthawi ya Mose pamene mau a Paulo analembedwa pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:14:35,082 Example #3\n",
            "2020-07-12 22:14:35,083 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:14:35,083 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:14:35,083 \tHypothesis: Tiziyamikila Cisomo Cimene Cisomo Cili pa July\n",
            "2020-07-12 22:14:35,083 Validation result (greedy) at epoch  47, step    27000: bleu:  16.73, loss: 49233.0117, ppl:   7.2846, duration: 29.5404s\n",
            "2020-07-12 22:14:53,720 Epoch  47 Step:    27100 Batch Loss:     1.741834 Tokens per Sec:    11139, Lr: 0.000300\n",
            "2020-07-12 22:15:12,052 Epoch  47 Step:    27200 Batch Loss:     2.265993 Tokens per Sec:    11159, Lr: 0.000300\n",
            "2020-07-12 22:15:30,733 Epoch  47 Step:    27300 Batch Loss:     1.931313 Tokens per Sec:    11214, Lr: 0.000300\n",
            "2020-07-12 22:15:36,313 Epoch  47: total training loss 1070.92\n",
            "2020-07-12 22:15:36,313 EPOCH 48\n",
            "2020-07-12 22:15:49,093 Epoch  48 Step:    27400 Batch Loss:     1.875881 Tokens per Sec:    11401, Lr: 0.000300\n",
            "2020-07-12 22:16:07,463 Epoch  48 Step:    27500 Batch Loss:     1.767903 Tokens per Sec:    11273, Lr: 0.000300\n",
            "2020-07-12 22:16:25,958 Epoch  48 Step:    27600 Batch Loss:     2.175007 Tokens per Sec:    11324, Lr: 0.000300\n",
            "2020-07-12 22:16:44,312 Epoch  48 Step:    27700 Batch Loss:     1.650685 Tokens per Sec:    11127, Lr: 0.000300\n",
            "2020-07-12 22:17:02,594 Epoch  48 Step:    27800 Batch Loss:     1.905820 Tokens per Sec:    11334, Lr: 0.000300\n",
            "2020-07-12 22:17:21,133 Epoch  48 Step:    27900 Batch Loss:     1.916827 Tokens per Sec:    10965, Lr: 0.000300\n",
            "2020-07-12 22:17:23,769 Epoch  48: total training loss 1075.79\n",
            "2020-07-12 22:17:23,770 EPOCH 49\n",
            "2020-07-12 22:17:39,734 Epoch  49 Step:    28000 Batch Loss:     1.575364 Tokens per Sec:    11227, Lr: 0.000300\n",
            "2020-07-12 22:18:05,316 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:18:05,317 Saving new checkpoint.\n",
            "2020-07-12 22:18:06,639 Example #0\n",
            "2020-07-12 22:18:06,640 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:18:06,640 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:18:06,640 \tHypothesis: Nchito yathu yapamwamba imene tinali kugula inali itayamba padziko lonse\n",
            "2020-07-12 22:18:06,640 Example #1\n",
            "2020-07-12 22:18:06,641 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:18:06,641 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:18:06,641 \tHypothesis: Iye ayenela kuti anali kufunitsitsa kuteteza zinthu zimene anacita kale , ngakhale kuti anali ndi zaka zambili .\n",
            "2020-07-12 22:18:06,641 Example #2\n",
            "2020-07-12 22:18:06,642 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:18:06,642 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:18:06,642 \tHypothesis: Kodi mau a Mose a m’nthawi ya Mose analembedwa pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:18:06,642 Example #3\n",
            "2020-07-12 22:18:06,642 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:18:06,643 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:18:06,643 \tHypothesis: Tisamalile Cikondi Cosatha , July\n",
            "2020-07-12 22:18:06,643 Validation result (greedy) at epoch  49, step    28000: bleu:  16.77, loss: 49109.4180, ppl:   7.2484, duration: 26.9085s\n",
            "2020-07-12 22:18:25,231 Epoch  49 Step:    28100 Batch Loss:     1.950407 Tokens per Sec:    11020, Lr: 0.000300\n",
            "2020-07-12 22:18:43,852 Epoch  49 Step:    28200 Batch Loss:     1.889157 Tokens per Sec:    11197, Lr: 0.000300\n",
            "2020-07-12 22:19:02,298 Epoch  49 Step:    28300 Batch Loss:     1.516591 Tokens per Sec:    11353, Lr: 0.000300\n",
            "2020-07-12 22:19:20,672 Epoch  49 Step:    28400 Batch Loss:     1.849100 Tokens per Sec:    10975, Lr: 0.000300\n",
            "2020-07-12 22:19:38,587 Epoch  49: total training loss 1066.71\n",
            "2020-07-12 22:19:38,587 EPOCH 50\n",
            "2020-07-12 22:19:39,232 Epoch  50 Step:    28500 Batch Loss:     2.093219 Tokens per Sec:    10509, Lr: 0.000300\n",
            "2020-07-12 22:19:57,742 Epoch  50 Step:    28600 Batch Loss:     1.851735 Tokens per Sec:    11065, Lr: 0.000300\n",
            "2020-07-12 22:20:16,555 Epoch  50 Step:    28700 Batch Loss:     1.862564 Tokens per Sec:    11354, Lr: 0.000300\n",
            "2020-07-12 22:20:35,050 Epoch  50 Step:    28800 Batch Loss:     2.086834 Tokens per Sec:    11289, Lr: 0.000300\n",
            "2020-07-12 22:20:53,602 Epoch  50 Step:    28900 Batch Loss:     1.967869 Tokens per Sec:    11367, Lr: 0.000300\n",
            "2020-07-12 22:21:12,089 Epoch  50 Step:    29000 Batch Loss:     1.799863 Tokens per Sec:    10979, Lr: 0.000300\n",
            "2020-07-12 22:21:33,248 Hooray! New best validation result [ppl]!\n",
            "2020-07-12 22:21:33,249 Saving new checkpoint.\n",
            "2020-07-12 22:21:34,414 Example #0\n",
            "2020-07-12 22:21:34,415 \tSource:     Our professional ballet careers took us around the world to dance\n",
            "2020-07-12 22:21:34,415 \tReference:  Cifukwa cokhala ndi maluso ovina , tinayenda m’madela ambili padziko lapansi\n",
            "2020-07-12 22:21:34,415 \tHypothesis: Nchito yathu yolalikila inali yosiyana kwambili ndi anthu amene tinali kukhala m’dziko la Japan\n",
            "2020-07-12 22:21:34,415 Example #1\n",
            "2020-07-12 22:21:34,415 \tSource:     He may be plagued with guilt about something he did in the past , even many years ago .\n",
            "2020-07-12 22:21:34,415 \tReference:  Taona kale mmene Davide na Paulo anakambilapo za nkhawa zawo .\n",
            "2020-07-12 22:21:34,416 \tHypothesis: Iye ayenela kuti anali ndi nkhawa kwambili za zinthu zimene anacita zaka zambili , ngakhale kuti anali ndi zaka zambili .\n",
            "2020-07-12 22:21:34,416 Example #2\n",
            "2020-07-12 22:21:34,416 \tSource:     What events in Moses ’ day provide the background for Paul’s words recorded at 2 Timothy 2 : 19 ?\n",
            "2020-07-12 22:21:34,416 \tReference:  Ndi zocitika ziti za m’nthawi ya Mose zimene zimagwilizana ndi zimene Paulo analemba pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:21:34,416 \tHypothesis: Ndi zocitika ziti zimene zinacitikila Mose pa mau a Paulo a pa 2 Timoteyo 2 : 19 ?\n",
            "2020-07-12 22:21:34,417 Example #3\n",
            "2020-07-12 22:21:34,417 \tSource:     Spread the Good News of Undeserved Kindness , July\n",
            "2020-07-12 22:21:34,417 \tReference:  Lalikilani Uthenga Wabwino Wokamba za Kukoma Mtima , July\n",
            "2020-07-12 22:21:34,417 \tHypothesis: Tiziyesetsa Kuonetsa Cikondi Cosatha , July\n",
            "2020-07-12 22:21:34,420 Validation result (greedy) at epoch  50, step    29000: bleu:  16.81, loss: 48810.3203, ppl:   7.1615, duration: 22.3307s\n",
            "2020-07-12 22:21:48,865 Epoch  50: total training loss 1056.90\n",
            "2020-07-12 22:21:48,866 Training ended after  50 epochs.\n",
            "2020-07-12 22:21:48,866 Best validation result (greedy) at step    29000:   7.16 ppl.\n",
            "/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.\n",
            "2020-07-12 22:22:12,383  dev bleu:  17.72 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-07-12 22:22:12,388 Translations saved to: /content/drive/My Drive/masakhane/model-temp/00029000.hyps.dev\n",
            "2020-07-12 22:22:57,597 test bleu:  30.09 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-07-12 22:22:57,602 Translations saved to: /content/drive/My Drive/masakhane/model-temp/00029000.hyps.test\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "MBoDS09JM807",
        "colab": {}
      },
      "source": [
        "# Copy the created models from the temporary storage to main storage on google drive for persistant storage \n",
        "!cp -r \"/content/drive/My Drive/masakhane/model-temp/\"* \"$gdrive_path/models/${src}${tgt}_transformer/\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 510
        },
        "outputId": "fe5a6c4e-32d4-4100-cb98-d7326ce5ad36"
      },
      "source": [
        "# Output our validation accuracy\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 1000\tLoss: 104985.17969\tPPL: 69.02498\tbleu: 0.72913\tLR: 0.00030000\t*\n",
            "Steps: 2000\tLoss: 87342.05469\tPPL: 33.88090\tbleu: 2.09997\tLR: 0.00030000\t*\n",
            "Steps: 3000\tLoss: 78307.31250\tPPL: 23.53396\tbleu: 3.31598\tLR: 0.00030000\t*\n",
            "Steps: 4000\tLoss: 72736.87500\tPPL: 18.79830\tbleu: 5.24072\tLR: 0.00030000\t*\n",
            "Steps: 5000\tLoss: 68773.72656\tPPL: 16.02127\tbleu: 6.90910\tLR: 0.00030000\t*\n",
            "Steps: 6000\tLoss: 66082.85938\tPPL: 14.37346\tbleu: 8.21670\tLR: 0.00030000\t*\n",
            "Steps: 7000\tLoss: 63526.49609\tPPL: 12.96529\tbleu: 9.07820\tLR: 0.00030000\t*\n",
            "Steps: 8000\tLoss: 61911.11719\tPPL: 12.14747\tbleu: 9.82287\tLR: 0.00030000\t*\n",
            "Steps: 9000\tLoss: 60281.30469\tPPL: 11.37461\tbleu: 10.86482\tLR: 0.00030000\t*\n",
            "Steps: 10000\tLoss: 58961.61328\tPPL: 10.78499\tbleu: 11.17248\tLR: 0.00030000\t*\n",
            "Steps: 11000\tLoss: 57576.37891\tPPL: 10.19894\tbleu: 11.80842\tLR: 0.00030000\t*\n",
            "Steps: 12000\tLoss: 56646.62109\tPPL: 9.82355\tbleu: 12.65364\tLR: 0.00030000\t*\n",
            "Steps: 13000\tLoss: 55712.61328\tPPL: 9.46036\tbleu: 13.00193\tLR: 0.00030000\t*\n",
            "Steps: 14000\tLoss: 54857.13281\tPPL: 9.13950\tbleu: 13.33962\tLR: 0.00030000\t*\n",
            "Steps: 15000\tLoss: 53885.88672\tPPL: 8.78839\tbleu: 13.85405\tLR: 0.00030000\t*\n",
            "Steps: 16000\tLoss: 53204.42578\tPPL: 8.55012\tbleu: 14.67869\tLR: 0.00030000\t*\n",
            "Steps: 17000\tLoss: 53287.41016\tPPL: 8.57879\tbleu: 14.17911\tLR: 0.00030000\t\n",
            "Steps: 18000\tLoss: 52131.85938\tPPL: 8.18812\tbleu: 14.77395\tLR: 0.00030000\t*\n",
            "Steps: 19000\tLoss: 52101.89844\tPPL: 8.17823\tbleu: 15.42948\tLR: 0.00030000\t*\n",
            "Steps: 20000\tLoss: 51433.57422\tPPL: 7.96072\tbleu: 15.35123\tLR: 0.00030000\t*\n",
            "Steps: 21000\tLoss: 51151.20703\tPPL: 7.87057\tbleu: 15.82799\tLR: 0.00030000\t*\n",
            "Steps: 22000\tLoss: 50394.41016\tPPL: 7.63396\tbleu: 15.77829\tLR: 0.00030000\t*\n",
            "Steps: 23000\tLoss: 50194.80469\tPPL: 7.57274\tbleu: 16.33550\tLR: 0.00030000\t*\n",
            "Steps: 24000\tLoss: 50057.70312\tPPL: 7.53098\tbleu: 16.58389\tLR: 0.00030000\t*\n",
            "Steps: 25000\tLoss: 49733.63672\tPPL: 7.43319\tbleu: 16.61687\tLR: 0.00030000\t*\n",
            "Steps: 26000\tLoss: 49328.43750\tPPL: 7.31269\tbleu: 16.81934\tLR: 0.00030000\t*\n",
            "Steps: 27000\tLoss: 49233.01172\tPPL: 7.28460\tbleu: 16.73319\tLR: 0.00030000\t*\n",
            "Steps: 28000\tLoss: 49109.41797\tPPL: 7.24838\tbleu: 16.76788\tLR: 0.00030000\t*\n",
            "Steps: 29000\tLoss: 48810.32031\tPPL: 7.16146\tbleu: 16.81243\tLR: 0.00030000\t*\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "565c50c6-de75-409f-db7c-7cb809d8288a"
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\"\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-07-12 22:31:35,573 Hello! This is Joey-NMT.\n",
            "/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.\n",
            "2020-07-12 22:32:01,727  dev bleu:  17.72 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-07-12 22:32:46,046 test bleu:  30.09 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GMM9isXuGu-s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}