en-ish/jw300-baseline/English_to_Esan_BPE_notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "starter_notebook.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ruohoruotsi/masakhane/blob/master/en-ish/jw300-baseline/English_to_Esan_BPE_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Igc5itf-xMGj"
      },
      "source": [
        "# Masakhane - Machine Translation for African Languages (Using JoeyNMT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x4fXCKCf36IK"
      },
      "source": [
        "## Note before beginning:\n",
        "### - The idea is that you should be able to make minimal changes to this in order to get SOME result for your own translation corpus. \n",
        "\n",
        "### - The tl;dr: Go to the **\"TODO\"** comments which will tell you what to update to get up and running\n",
        "\n",
        "### - If you actually want to have a clue what you're doing, read the text and peek at the links\n",
        "\n",
        "### - With 100 epochs, it should take around 7 hours to run in Google Colab\n",
        "\n",
        "### - Once you've gotten a result for your language, please attach and email your notebook that generated it to masakhanetranslation@gmail.com\n",
        "\n",
        "### - If you care enough and get a chance, doing a brief background on your language would be amazing. See examples in  [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l929HimrxS0a"
      },
      "source": [
        "## Retrieve your data & make a parallel corpus\n",
        "\n",
        "If you are wanting to use the JW300 data referenced on the Masakhane website or in our GitHub repo, you can use `opus-tools` to convert the data into a convenient format. `opus_read` from that package provides a convenient tool for reading the native aligned XML files and to convert them to TMX format. The tool can also be used to fetch relevant files from OPUS on the fly and to filter the data as necessary. [Read the documentation](https://pypi.org/project/opustools-pkg/) for more details.\n",
        "\n",
        "Once you have your corpus files in TMX format (an xml structure which will include the sentences in your target language and your source language in a single file), we recommend reading them into a pandas dataframe. Thankfully, Jade wrote a silly `tmx2dataframe` package which converts your tmx file to a pandas dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oGRmDELn7Az0",
        "outputId": "2bf6e5b4-7f40-4215-f56f-5965436a3068",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Cn3tgQLzUxwn",
        "colab": {}
      },
      "source": [
        "# TODO: Set your source and target languages. Keep in mind, these traditionally use language codes as found here:\n",
        "# These will also become the suffix's of all vocab and corpus files used throughout\n",
        "import os\n",
        "source_language = \"en\"\n",
        "target_language = \"ish\" \n",
        "lc = False  # If True, lowercase the data.\n",
        "seed = 42  # Random seed for shuffling.\n",
        "tag = \"baseline\" # Give a unique name to your folder - this is to ensure you don't rewrite any models you've already submitted\n",
        "\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "os.environ[\"tag\"] = tag\n",
        "\n",
        "# This will save it to a folder in our gdrive instead!\n",
        "!mkdir -p \"/content/drive/My Drive/masakhane/$src-$tgt-$tag\"\n",
        "os.environ[\"gdrive_path\"] = \"/content/drive/My Drive/masakhane/%s-%s-%s\" % (source_language, target_language, tag)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSgJHEw7Nvx",
        "outputId": "6e4c35c6-fd0c-4b68-d034-0a3ec43cfb9b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!echo $gdrive_path"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/masakhane/en-ish-baseline\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "gA75Fs9ys8Y9",
        "outputId": "4e11025d-c761-464e-e901-317aef9e0eb4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# Install opus-tools\n",
        "! pip install opustools-pkg"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting opustools-pkg\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6c/9f/e829a0cceccc603450cd18e1ff80807b6237a88d9a8df2c0bb320796e900/opustools_pkg-0.0.52-py3-none-any.whl (80kB)\n",
            "\r\u001b[K     |████                            | 10kB 24.8MB/s eta 0:00:01\r\u001b[K     |████████                        | 20kB 1.7MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 30kB 2.6MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 40kB 1.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 51kB 2.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 61kB 2.5MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▎   | 71kB 2.9MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 81kB 2.5MB/s \n",
            "\u001b[?25hInstalling collected packages: opustools-pkg\n",
            "Successfully installed opustools-pkg-0.0.52\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "xq-tDZVks7ZD",
        "outputId": "d3b5a669-b7d6-44d0-fab0-92e38a38f616",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Downloading our corpus\n",
        "! opus_read -d JW300 -s $src -t $tgt -wm moses -w jw300.$src jw300.$tgt -q\n",
        "\n",
        "# extract the corpus file\n",
        "! gunzip JW300_latest_xml_$src-$tgt.xml.gz"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Alignment file /proj/nlpl/data/OPUS/JW300/latest/xml/en-ish.xml.gz not found. The following files are available for downloading:\n",
            "\n",
            "  60 KB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en-ish.xml.gz\n",
            " 263 MB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/en.zip\n",
            " 624 KB https://object.pouta.csc.fi/OPUS-JW300/v1/xml/ish.zip\n",
            "\n",
            " 264 MB Total size\n",
            "./JW300_latest_xml_en-ish.xml.gz ... 100% of 60 KB\n",
            "./JW300_latest_xml_en.zip ... 100% of 263 MB\n",
            "./JW300_latest_xml_ish.zip ... 100% of 624 KB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n48GDRnP8y2G",
        "colab_type": "code",
        "outputId": "9db9104f-9cd2-4af2-aa1e-67d184495313",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        }
      },
      "source": [
        "# Download the global test set.\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
        "  \n",
        "# And the specific test set for this language pair.\n",
        "os.environ[\"trg\"] = target_language \n",
        "os.environ[\"src\"] = source_language \n",
        "\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.en \n",
        "! mv test.en-$trg.en test.en\n",
        "! wget https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-$trg.$trg \n",
        "! mv test.en-$trg.$trg test.$trg"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-01-26 05:58:13--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-any.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 277791 (271K) [text/plain]\n",
            "Saving to: ‘test.en-any.en’\n",
            "\n",
            "\rtest.en-any.en        0%[                    ]       0  --.-KB/s               \rtest.en-any.en      100%[===================>] 271.28K  --.-KB/s    in 0.06s   \n",
            "\n",
            "2020-01-26 05:58:14 (4.70 MB/s) - ‘test.en-any.en’ saved [277791/277791]\n",
            "\n",
            "--2020-01-26 05:58:14--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ish.en\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 24189 (24K) [text/plain]\n",
            "Saving to: ‘test.en-ish.en’\n",
            "\n",
            "test.en-ish.en      100%[===================>]  23.62K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2020-01-26 05:58:14 (1.82 MB/s) - ‘test.en-ish.en’ saved [24189/24189]\n",
            "\n",
            "--2020-01-26 05:58:16--  https://raw.githubusercontent.com/juliakreutzer/masakhane/master/jw300_utils/test/test.en-ish.ish\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 31664 (31K) [text/plain]\n",
            "Saving to: ‘test.en-ish.ish’\n",
            "\n",
            "test.en-ish.ish     100%[===================>]  30.92K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2020-01-26 05:58:16 (2.32 MB/s) - ‘test.en-ish.ish’ saved [31664/31664]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqDG-CI28y2L",
        "colab_type": "code",
        "outputId": "59a385f1-9d0d-4436-fe82-107c2b18df55",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Read the test data to filter from train and dev splits.\n",
        "# Store english portion in set for quick filtering checks.\n",
        "en_test_sents = set()\n",
        "filter_test_sents = \"test.en-any.en\"\n",
        "j = 0\n",
        "with open(filter_test_sents) as f:\n",
        "  for line in f:\n",
        "    en_test_sents.add(line.strip())\n",
        "    j += 1\n",
        "print('Loaded {} global test sentences to filter from the training/dev data.'.format(j))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded 3571 global test sentences to filter from the training/dev data.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3CNdwLBCfSIl",
        "outputId": "5b6be1fa-f7af-407e-8a30-da859b29a6f5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# TMX file to dataframe\n",
        "source_file = 'jw300.' + source_language\n",
        "target_file = 'jw300.' + target_language\n",
        "\n",
        "source = []\n",
        "target = []\n",
        "skip_lines = []  # Collect the line numbers of the source portion to skip the same lines for the target portion.\n",
        "with open(source_file) as f:\n",
        "    for i, line in enumerate(f):\n",
        "        # Skip sentences that are contained in the test set.\n",
        "        if line.strip() not in en_test_sents:\n",
        "            source.append(line.strip())\n",
        "        else:\n",
        "            skip_lines.append(i)             \n",
        "with open(target_file) as f:\n",
        "    for j, line in enumerate(f):\n",
        "        # Only add to corpus if corresponding source was not skipped.\n",
        "        if j not in skip_lines:\n",
        "            target.append(line.strip())\n",
        "    \n",
        "print('Loaded data and skipped {}/{} lines since contained in test set.'.format(len(skip_lines), i))\n",
        "    \n",
        "df = pd.DataFrame(zip(source, target), columns=['source_sentence', 'target_sentence'])\n",
        "# if you get TypeError: data argument can't be an iterator is because of your zip version run this below\n",
        "#df = pd.DataFrame(list(zip(source, target)), columns=['source_sentence', 'target_sentence'])\n",
        "df.head(3)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded data and skipped 426/6114 lines since contained in test set.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>source_sentence</th>\n",
              "      <th>target_sentence</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3 They Offered Themselves Willingly ​ — In Mad...</td>\n",
              "      <td>3 Eria Ne Tobele Zegbere — bhi Madagascar</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7 “ He Gives Power to the Tired One ”</td>\n",
              "      <td>7 “ Ọle Re Ahu Nin Ọria Nin Egbe Lọle ”</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>When we feel overwhelmed by the pressures of l...</td>\n",
              "      <td>Ahamiẹn ọkakale nan miẹn bhi iẹnlẹn re egbe lọ...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     source_sentence                                    target_sentence\n",
              "0  3 They Offered Themselves Willingly ​ — In Mad...          3 Eria Ne Tobele Zegbere — bhi Madagascar\n",
              "1              7 “ He Gives Power to the Tired One ”            7 “ Ọle Re Ahu Nin Ọria Nin Egbe Lọle ”\n",
              "2  When we feel overwhelmed by the pressures of l...  Ahamiẹn ọkakale nan miẹn bhi iẹnlẹn re egbe lọ..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkuK3B4p2AkN"
      },
      "source": [
        "## Pre-processing and export\n",
        "\n",
        "It is generally a good idea to remove duplicate translations and conflicting translations from the corpus. In practice, these public corpora include some number of these that need to be cleaned.\n",
        "\n",
        "In addition we will split our data into dev/test/train and export to the filesystem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "M_2ouEOH1_1q",
        "outputId": "141a992f-c2db-4768-d8ae-ca79815ff84d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "# drop duplicate translations\n",
        "df_pp = df.drop_duplicates()\n",
        "\n",
        "# drop conflicting translations\n",
        "# (this is optional and something that you might want to comment out \n",
        "# depending on the size of your corpus)\n",
        "df_pp.drop_duplicates(subset='source_sentence', inplace=True)\n",
        "df_pp.drop_duplicates(subset='target_sentence', inplace=True)\n",
        "\n",
        "# Shuffle the data to remove bias in dev set selection.\n",
        "df_pp = df_pp.sample(frac=1, random_state=seed).reset_index(drop=True)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  \n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  import sys\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z_1BwAApEtMk",
        "colab_type": "code",
        "outputId": "ec42bc46-a3fb-4db4-a699-96a663f55a42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 428
        }
      },
      "source": [
        "# Install fuzzy wuzzy to remove \"almost duplicate\" sentences in the\n",
        "# test and training sets.\n",
        "! pip install fuzzywuzzy\n",
        "! pip install python-Levenshtein\n",
        "import time\n",
        "from fuzzywuzzy import process\n",
        "import numpy as np\n",
        "\n",
        "# reset the index of the training set after previous filtering\n",
        "df_pp.reset_index(drop=False, inplace=True)\n",
        "\n",
        "# Remove samples from the training data set if they \"almost overlap\" with the\n",
        "# samples in the test set.\n",
        "\n",
        "# Filtering function. Adjust pad to narrow down the candidate matches to\n",
        "# within a certain length of characters of the given sample.\n",
        "def fuzzfilter(sample, candidates, pad):\n",
        "  candidates = [x for x in candidates if len(x) <= len(sample)+pad and len(x) >= len(sample)-pad] \n",
        "  if len(candidates) > 0:\n",
        "    return process.extractOne(sample, candidates)[1]\n",
        "  else:\n",
        "    return np.nan\n",
        "\n",
        "# NOTE - This might run slow depending on the size of your training set. We are\n",
        "# printing some information to help you track how long it would take. \n",
        "scores = []\n",
        "start_time = time.time()\n",
        "for idx, row in df_pp.iterrows():\n",
        "  scores.append(fuzzfilter(row['source_sentence'], list(en_test_sents), 5))\n",
        "  if idx % 1000 == 0:\n",
        "    hours, rem = divmod(time.time() - start_time, 3600)\n",
        "    minutes, seconds = divmod(rem, 60)\n",
        "    print(\"{:0>2}:{:0>2}:{:05.2f}\".format(int(hours),int(minutes),seconds), \"%0.2f percent complete\" % (100.0*float(idx)/float(len(df_pp))))\n",
        "\n",
        "# Filter out \"almost overlapping samples\"\n",
        "df_pp['scores'] = scores\n",
        "df_pp = df_pp[df_pp['scores'] < 95]"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.17.0\n",
            "Collecting python-Levenshtein\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz (48kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 1.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from python-Levenshtein) (42.0.2)\n",
            "Building wheels for collected packages: python-Levenshtein\n",
            "  Building wheel for python-Levenshtein (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp36-cp36m-linux_x86_64.whl size=144676 sha256=e1c1ba0cb6ba911843732d2acf10e08c1d13193257ce85b35982c9f4eb0a1510\n",
            "  Stored in directory: /root/.cache/pip/wheels/de/c2/93/660fd5f7559049268ad2dc6d81c4e39e9e36518766eaf7e342\n",
            "Successfully built python-Levenshtein\n",
            "Installing collected packages: python-Levenshtein\n",
            "Successfully installed python-Levenshtein-0.12.0\n",
            "00:00:00.02 0.00 percent complete\n",
            "00:00:21.00 19.41 percent complete\n",
            "00:00:41.39 38.81 percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '']\n",
            "WARNING:root:Applied processor reduces input query to empty string, all comparisons will have score 0. [Query: '*']\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "00:01:02.08 58.22 percent complete\n",
            "00:01:23.09 77.62 percent complete\n",
            "00:01:44.76 97.03 percent complete\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "hxxBOCA-xXhy",
        "outputId": "fea25604-9299-408f-ed1f-6d5bd2fe24c7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 819
        }
      },
      "source": [
        "# This section does the split between train/dev for the parallel corpora then saves them as separate files\n",
        "# We use 1000 dev test and the given test set.\n",
        "import csv\n",
        "\n",
        "# Do the split between dev/train and create parallel corpora\n",
        "num_dev_patterns = 1000\n",
        "\n",
        "# Optional: lower case the corpora - this will make it easier to generalize, but without proper casing.\n",
        "if lc:  # Julia: making lowercasing optional\n",
        "    df_pp[\"source_sentence\"] = df_pp[\"source_sentence\"].str.lower()\n",
        "    df_pp[\"target_sentence\"] = df_pp[\"target_sentence\"].str.lower()\n",
        "\n",
        "# Julia: test sets are already generated\n",
        "dev = df_pp.tail(num_dev_patterns) # Herman: Error in original\n",
        "stripped = df_pp.drop(df_pp.tail(num_dev_patterns).index)\n",
        "\n",
        "with open(\"train.\"+source_language, \"w\") as src_file, open(\"train.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in stripped.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "    \n",
        "with open(\"dev.\"+source_language, \"w\") as src_file, open(\"dev.\"+target_language, \"w\") as trg_file:\n",
        "  for index, row in dev.iterrows():\n",
        "    src_file.write(row[\"source_sentence\"]+\"\\n\")\n",
        "    trg_file.write(row[\"target_sentence\"]+\"\\n\")\n",
        "\n",
        "#stripped[[\"source_sentence\"]].to_csv(\"train.\"+source_language, header=False, index=False)  # Herman: Added `header=False` everywhere\n",
        "#stripped[[\"target_sentence\"]].to_csv(\"train.\"+target_language, header=False, index=False)  # Julia: Problematic handling of quotation marks.\n",
        "\n",
        "#dev[[\"source_sentence\"]].to_csv(\"dev.\"+source_language, header=False, index=False)\n",
        "#dev[[\"target_sentence\"]].to_csv(\"dev.\"+target_language, header=False, index=False)\n",
        "\n",
        "# Doublecheck the format below. There should be no extra quotation marks or weird characters.\n",
        "! head train.*\n",
        "! head dev.*"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "==> train.en <==\n",
            "Their unity would give a powerful witness , offering clear evidence that Jehovah had sent Jesus to the earth to do God’s will .\n",
            "In the first century , some Christians sold items that they owned , such as fields or houses , and brought the money to the apostles .\n",
            "Brothers and sisters who remain single because they want to obey the admonition to marry “ only in the Lord ” make up another group that merits encouragement .\n",
            "How can we avoid misusing our God - given freedom ?\n",
            "How grateful we are to be blessed with answered prayers and friendship with Jesus !\n",
            "However , there are some indications of Paul’s appearance .\n",
            "18 What Does It Mean to Be a Spiritual Person ?\n",
            "The psalmist observed : “ It is good to give thanks to Jehovah . . .\n",
            "We do well to ask ourselves : ‘ Do meetings and field service take second place to entertainment ?\n",
            "Joy , in contrast , is a deep - seated quality of the heart .\n",
            "\n",
            "==> train.ish <==\n",
            "Okugbe nọnsele dẹ wo sabọ sọsali ọbhọ ghe , Jehova gene ji Jesu re bhi ọne otọ nan nin ọle dọ lu iho nọnsole . Ahoẹmhọn - egbe hi iyaman nan ha rẹ gene lẹn edibo nesi Jesu .\n",
            "3 : 21 , 22 ; 35 : 22 - 24 ) Bhi ore nọn hẹnhẹn , kristiẹn eso da khiẹn emhin nin ele mhọnlẹn , emhin bọsi uwa , la otọ .\n",
            "Ẹbho ọbhebhe ne yẹ guanọ izebhudu hi , ibhio mhan ne bha lọnmhẹn ọdọ la ne bha re okhuo ranmhude ele ho nin ele rẹkhan adia nọn ribhi Baibo nọn yọle ghe , ọria nọn ribhi oga ọkpa mhan ha rẹkhan . ( 1 Cor . 7 : 39 ) Iriọ sẹyẹ nọn rẹji ene amhẹn .\n",
            "Be imhan ha lu nin mhan hẹi re ọne isẹhoa nin Osẹnobulua re nin mhan khiẹnlẹn ?\n",
            "4 : 13 ) Ẹsele nọn khua wo nọn ghe Osẹnobulua họn erọnmhọn nọnsẹmhan , ghe mhan dẹ sabọ deba iJesu ha mhọn ikolu !\n",
            "Ọkpakinọn , emhin eso ribhọ ne ri ebi Paul diayẹ man .\n",
            "18 Be Ọria Nọn Mun Oga Mhọn Nọnsẹn Rẹ Lu Emhin Yẹ ?\n",
            "Ọnan zẹle nin ọnọn gbẹn Ps . 92 : 1 , 4 bhi Baibo da yọle : “ Ọmhẹnmhin nan rẹ ha re ekhuẹnmhẹn ji Jehova . . .\n",
            "Nin mhan nọọn egbe mhan ene inọnta nan : ‘ iregbe re ahoho mhẹn mun kalo gbera ikolo oga bi itẹmhọn Osẹnobulua ?\n",
            "Ọnyẹnmhẹn , ọle hi eghọnghọn ọsajie nọn vuọn ọria udu .\n",
            "==> dev.en <==\n",
            "I WAS raised in Graz , Austria .\n",
            "In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "We should also strive to help others spiritually .\n",
            "Still others are struggling to provide , not luxury items , but just the basic necessities for their families .\n",
            "Eventually , their love of material things can choke any love they had for God . ​ — Matt .\n",
            "What can help Christian parents succeed in raising their children to serve Jehovah ?\n",
            "Why not be resolved to have a positive influence , promoting unity among your brothers and sisters ?\n",
            "By 1963 there were about 160 Witnesses in Kyrgyzstan , many of them originally from Germany , Ukraine , and Russia .\n",
            "That is so even if it seems to go unnoticed .\n",
            "\n",
            "==> dev.ish <==\n",
            "AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "Ọ manman nọghọ eso rẹ sabọ gbẹloghe azagba - uwa nọnsele .\n",
            "( Matt . 6 : 24 ) Bhiriọ , oyẹẹ nin ele mhọn da ẹfe ki re oyẹẹ nin ele mhọn da Osẹnobulua wẹghẹa . — Matt .\n",
            "Be bhọ ene biẹ ọmọn ha rẹ sabọ bẹẹ imọn nesele wanre yẹ nin ene imọn da ha ga iJehova ?\n",
            "Dọnmhegbe nin uwẹ ha yi ijiẹmhin esili , nin uwẹ da sabọ ha tuẹn okugbe nọn ribhi ẹwẹ mhan okhun .\n",
            "Bhi ukpe 1963 , Esali Jehova ne bun sẹbhi 160 , ele ha ribhi Kyrgyzstan . Germany , Ukraine bi Russia , ọle ene bun nẹ bhi ẹwẹ ele nan vae .\n",
            "Ahamiẹn mhan re ọkhọle rebhe rẹ tuẹ ẹbho , ọ ki sẹ ele otọ - ẹkẹ aharẹmiẹn ọbha tan .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "epeCydmCyS8X"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "## Installation of JoeyNMT\n",
        "\n",
        "JoeyNMT is a simple, minimalist NMT package which is useful for learning and teaching. Check out the documentation for JoeyNMT [here](https://joeynmt.readthedocs.io)  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "iBRMm4kMxZ8L",
        "outputId": "d632e846-5139-4fae-ea21-087beae32f0a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Install JoeyNMT\n",
        "! git clone https://github.com/joeynmt/joeynmt.git\n",
        "! cd joeynmt; pip3 install ."
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'joeynmt'...\n",
            "remote: Enumerating objects: 61, done.\u001b[K\n",
            "remote: Counting objects: 100% (61/61), done.\u001b[K\n",
            "remote: Compressing objects: 100% (39/39), done.\u001b[K\n",
            "remote: Total 2245 (delta 34), reused 34 (delta 22), pack-reused 2184\u001b[K\n",
            "Receiving objects: 100% (2245/2245), 2.63 MiB | 8.83 MiB/s, done.\n",
            "Resolving deltas: 100% (1555/1555), done.\n",
            "Processing /content/joeynmt\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (6.2.2)\n",
            "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.17.5)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (42.0.2)\n",
            "Requirement already satisfied: torch>=1.1 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.3.1)\n",
            "Requirement already satisfied: tensorflow>=1.14 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.3.1)\n",
            "Collecting sacrebleu>=1.3.6\n",
            "  Downloading https://files.pythonhosted.org/packages/45/31/1a135b964c169984b27fb2f7a50280fa7f8e6d9d404d8a9e596180487fd1/sacrebleu-1.4.3-py3-none-any.whl\n",
            "Collecting subword-nmt\n",
            "  Downloading https://files.pythonhosted.org/packages/74/60/6600a7bc09e7ab38bc53a48a20d8cae49b837f93f5842a41fe513a694912/subword_nmt-0.3.7-py2.py3-none-any.whl\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (3.1.2)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (0.9.0)\n",
            "Collecting pyyaml>=5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)\n",
            "\u001b[K     |████████████████████████████████| 276kB 13.9MB/s \n",
            "\u001b[?25hCollecting pylint\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/59/43fc36c5ee316bb9aeb7cf5329cdbdca89e5749c34d5602753827c0aa2dc/pylint-2.4.4-py3-none-any.whl (302kB)\n",
            "\u001b[K     |████████████████████████████████| 307kB 68.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: six==1.12 in /usr/local/lib/python3.6/dist-packages (from joeynmt==0.0.1) (1.12.0)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.10.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (3.1.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.33.6)\n",
            "Requirement already satisfied: tensorflow-estimator==1.15.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.1)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.1.8)\n",
            "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.0.8)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.9.0)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.11.2)\n",
            "Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.2.2)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (1.15.0)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow>=1.14->joeynmt==0.0.1) (0.8.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (2.21.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext->joeynmt==0.0.1) (4.28.1)\n",
            "Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from sacrebleu>=1.3.6->joeynmt==0.0.1) (3.6.6)\n",
            "Collecting portalocker\n",
            "  Downloading https://files.pythonhosted.org/packages/91/db/7bc703c0760df726839e0699b7f78a4d8217fdc9c7fcb1b51b39c5a22a4e/portalocker-1.5.2-py2.py3-none-any.whl\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.6.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (2.4.6)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (0.10.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->joeynmt==0.0.1) (1.1.0)\n",
            "Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (0.25.3)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn->joeynmt==0.0.1) (1.4.1)\n",
            "Collecting isort<5,>=4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e5/b0/c121fd1fa3419ea9bfd55c7f9c4fedfec5143208d8c7ad3ce3db6c623c21/isort-4.3.21-py2.py3-none-any.whl (42kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 10.4MB/s \n",
            "\u001b[?25hCollecting mccabe<0.7,>=0.6\n",
            "  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl\n",
            "Collecting astroid<2.4,>=2.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ad/ae/86734823047962e7b8c8529186a1ac4a7ca19aaf1aa0c7713c022ef593fd/astroid-2.3.3-py3-none-any.whl (205kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 80.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (0.16.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow>=1.14->joeynmt==0.0.1) (3.1.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.8->tensorflow>=1.14->joeynmt==0.0.1) (2.8.0)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext->joeynmt==0.0.1) (2019.11.28)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn->joeynmt==0.0.1) (2018.9)\n",
            "Collecting lazy-object-proxy==1.4.*\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/dd/b1e3407e9e6913cf178e506cd0dee818e58694d9a5cd1984e3f6a8b9a10f/lazy_object_proxy-1.4.3-cp36-cp36m-manylinux1_x86_64.whl (55kB)\n",
            "\u001b[K     |████████████████████████████████| 61kB 10.8MB/s \n",
            "\u001b[?25hCollecting typed-ast<1.5,>=1.4.0; implementation_name == \"cpython\" and python_version < \"3.8\"\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/ed/5459080d95eb87a02fe860d447197be63b6e2b5e9ff73c2b0a85622994f4/typed_ast-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (737kB)\n",
            "\u001b[K     |████████████████████████████████| 747kB 69.6MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: joeynmt, pyyaml\n",
            "  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for joeynmt: filename=joeynmt-0.0.1-cp36-none-any.whl size=72136 sha256=5ff22cc6fb3fa347f93ac52f0b9eebb02dbeba0e941ab1b9fc2720da0647d67d\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-usai0n22/wheels/db/01/db/751cc9f3e7f6faec127c43644ba250a3ea7ad200594aeda70a\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.3-cp36-cp36m-linux_x86_64.whl size=44229 sha256=61c6e1f112712bd9c5557339df670a525dcb3c78fc8ec314447030b2db0236eb\n",
            "  Stored in directory: /root/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191\n",
            "Successfully built joeynmt pyyaml\n",
            "Installing collected packages: portalocker, sacrebleu, subword-nmt, pyyaml, isort, mccabe, lazy-object-proxy, typed-ast, astroid, pylint, joeynmt\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed astroid-2.3.3 isort-4.3.21 joeynmt-0.0.1 lazy-object-proxy-1.4.3 mccabe-0.6.1 portalocker-1.5.2 pylint-2.4.4 pyyaml-5.3 sacrebleu-1.4.3 subword-nmt-0.3.7 typed-ast-1.4.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AaE77Tcppex9"
      },
      "source": [
        "# Preprocessing the Data into Subword BPE Tokens\n",
        "\n",
        "- One of the most powerful improvements for agglutinative languages (a feature of most Bantu languages) is using BPE tokenization [ (Sennrich, 2015) ](https://arxiv.org/abs/1508.07909).\n",
        "\n",
        "- It was also shown that by optimizing the umber of BPE codes we significantly improve results for low-resourced languages [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021) [(Martinus, 2019)](https://arxiv.org/abs/1906.05685)\n",
        "\n",
        "- Below we have the scripts for doing BPE tokenization of our data. We use 4000 tokens as recommended by [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021). You do not need to change anything. Simply running the below will be suitable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "H-TyjtmXB1mL",
        "outputId": "4a2e150f-749f-4235-874d-c397f1502f52",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        }
      },
      "source": [
        "# One of the huge boosts in NMT performance was to use a different method of tokenizing. \n",
        "# Usually, NMT would tokenize by words. However, using a method called BPE gave amazing boosts to performance\n",
        "\n",
        "# Do subword NMT\n",
        "from os import path\n",
        "os.environ[\"src\"] = source_language # Sets them in bash as well, since we often use bash scripts\n",
        "os.environ[\"tgt\"] = target_language\n",
        "\n",
        "# Learn BPEs on the training data.\n",
        "os.environ[\"data_path\"] = path.join(\"joeynmt\", \"data\", source_language + target_language) # Herman! \n",
        "! subword-nmt learn-joint-bpe-and-vocab --input train.$src train.$tgt -s 4000 -o bpe.codes.4000 --write-vocabulary vocab.$src vocab.$tgt\n",
        "\n",
        "# Apply BPE splits to the development and test data.\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < train.$src > train.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < train.$tgt > train.bpe.$tgt\n",
        "\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < dev.$src > dev.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < dev.$tgt > dev.bpe.$tgt\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$src < test.$src > test.bpe.$src\n",
        "! subword-nmt apply-bpe -c bpe.codes.4000 --vocabulary vocab.$tgt < test.$tgt > test.bpe.$tgt\n",
        "\n",
        "# Create directory, move everyone we care about to the correct location\n",
        "! mkdir -p $data_path\n",
        "! cp train.* $data_path\n",
        "! cp test.* $data_path\n",
        "! cp dev.* $data_path\n",
        "! cp bpe.codes.4000 $data_path\n",
        "! ls $data_path\n",
        "\n",
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\"\n",
        "\n",
        "# Create that vocab using build_vocab\n",
        "! sudo chmod 777 joeynmt/scripts/build_vocab.py\n",
        "! joeynmt/scripts/build_vocab.py joeynmt/data/$src$tgt/train.bpe.$src joeynmt/data/$src$tgt/train.bpe.$tgt --output_path joeynmt/data/$src$tgt/vocab.txt\n",
        "\n",
        "# Some output\n",
        "! echo \"BPE Ẹ̀sán Sentences\"\n",
        "! tail -n 5 test.bpe.$tgt\n",
        "! echo \"Combined BPE Vocab\"\n",
        "! tail -n 10 joeynmt/data/$src$tgt/vocab.txt  # Herman\n",
        "! cp joeynmt/data/$src$tgt/vocab.txt \"$gdrive_path\""
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.en\t     test.bpe.ish    test.ish\t    train.en\n",
            "dev.bpe.en\tdev.ish      test.en\t     train.bpe.en   train.ish\n",
            "dev.bpe.ish\ttest.bpe.en  test.en-any.en  train.bpe.ish\n",
            "bpe.codes.4000\tdev.en\t     test.bpe.ish    test.ish\t    train.en\n",
            "dev.bpe.en\tdev.ish      test.en\t     train.bpe.en   train.ish\n",
            "dev.bpe.ish\ttest.bpe.en  test.en-any.en  train.bpe.ish\n",
            "BPE Ẹ̀sán Sentences\n",
            "Agb@@ ada ọsi ẹlinmhin nọn khiale ( Fẹ uduọle 19 - 20 ghe )\n",
            "Jehova dẹ rẹkpa mhan nin mhan da sabọ nin ọle su@@ an .\n",
            "( b ) Emhin nela mhan ha zilo nyan bhi uh@@ ọ@@ mhọn - ọta nọn ki sẹ ọle bhi egbe ?\n",
            "Uwẹ be gbẹlokotọ tie ene ebe Ọ@@ kh@@ ẹ@@ ughe nọn bha sẹ bu@@ ẹ gbe nan ne dagbare ?\n",
            "F@@ ẹghe si uwẹ dẹ sabọ re ewanniẹn ọbhi ene inọnta nan :\n",
            "Combined BPE Vocab\n",
            "LLO\n",
            "fẹ@@\n",
            "Jac@@\n",
            "shar@@\n",
            "ergy\n",
            "beauti@@\n",
            "resp@@\n",
            "Kin@@\n",
            "chy\n",
            "okhan\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IlMitUHR8Qy-",
        "outputId": "41242114-8b4e-4873-a0f0-2385dfa383b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Also move everything we care about to a mounted location in google drive (relevant if running in colab) at gdrive_path\n",
        "! cp train.* \"$gdrive_path\"\n",
        "! cp test.* \"$gdrive_path\"\n",
        "! cp dev.* \"$gdrive_path\"\n",
        "! cp bpe.codes.4000 \"$gdrive_path\"\n",
        "! ls \"$gdrive_path\""
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "bpe.codes.4000\tdev.en\t     test.bpe.ish    test.ish\t    train.en\n",
            "dev.bpe.en\tdev.ish      test.en\t     train.bpe.en   train.ish\n",
            "dev.bpe.ish\ttest.bpe.en  test.en-any.en  train.bpe.ish  vocab.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ixmzi60WsUZ8"
      },
      "source": [
        "# Creating the JoeyNMT Config\n",
        "\n",
        "JoeyNMT requires a yaml config. We provide a template below. We've also set a number of defaults with it, that you may play with!\n",
        "\n",
        "- We used Transformer architecture \n",
        "- We set our dropout to reasonably high: 0.3 (recommended in  [(Sennrich, 2019)](https://www.aclweb.org/anthology/P19-1021))\n",
        "\n",
        "Things worth playing with:\n",
        "- The batch size (also recommended to change for low-resourced languages)\n",
        "- The number of epochs (we've set it at 30 just so it runs in about an hour, for testing purposes)\n",
        "- The decoder options (beam_size, alpha)\n",
        "- Evaluation metrics (BLEU versus Crhf4)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PIs1lY2hxMsl",
        "colab": {}
      },
      "source": [
        "# This creates the config file for our JoeyNMT system. It might seem overwhelming so we've provided a couple of useful parameters you'll need to update\n",
        "# (You can of course play with all the parameters if you'd like!)\n",
        "\n",
        "name = '%s%s' % (source_language, target_language)\n",
        "gdrive_path = os.environ[\"gdrive_path\"]\n",
        "\n",
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"{gdrive_path}/train.bpe\"\n",
        "    dev:   \"{gdrive_path}/dev.bpe\"\n",
        "    test:  \"{gdrive_path}/test.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False\n",
        "    max_sent_length: 100\n",
        "    src_vocab: \"{gdrive_path}/vocab.txt\"\n",
        "    trg_vocab: \"{gdrive_path}/vocab.txt\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "\n",
        "training:\n",
        "    # load_model: \"{gdrive_path}/models/{name}_transformer_orig/142000.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # TODO: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 200                     # TODO: Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 100          # TODO: Set to at least once per epoch.\n",
        "    logging_freq: 100\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"{gdrive_path}/models/{name}_transformer\"\n",
        "    overwrite: True               # TODO: Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_last_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # TODO: Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # TODO: Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, gdrive_path=os.environ[\"gdrive_path\"], source_language=source_language, target_language=target_language)\n",
        "with open(\"joeynmt/configs/transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)\n",
        "\n",
        "! cp joeynmt/configs/transformer_$src$tgt.yaml \"$gdrive_path\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pIifxE3Qzuvs"
      },
      "source": [
        "# Train the Model\n",
        "\n",
        "This single line of joeynmt runs the training using the config we made above"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6ZBPFwT94WpI",
        "outputId": "f13203b6-fae3-4620-ce2e-621404802448",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Train the model\n",
        "# You can press Ctrl-C to stop. And then run the next cell to save your checkpoints! \n",
        "# !cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml\n",
        "!python3 -m joeynmt train \"$gdrive_path/transformer_$src$tgt.yaml\""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-01-26 06:00:56,685 Hello! This is Joey-NMT.\n",
            "2020-01-26 06:00:58,020 Total params: 12055552\n",
            "2020-01-26 06:00:58,021 Trainable parameters: ['decoder.layer_norm.bias', 'decoder.layer_norm.weight', 'decoder.layers.0.dec_layer_norm.bias', 'decoder.layers.0.dec_layer_norm.weight', 'decoder.layers.0.feed_forward.layer_norm.bias', 'decoder.layers.0.feed_forward.layer_norm.weight', 'decoder.layers.0.feed_forward.pwff_layer.0.bias', 'decoder.layers.0.feed_forward.pwff_layer.0.weight', 'decoder.layers.0.feed_forward.pwff_layer.3.bias', 'decoder.layers.0.feed_forward.pwff_layer.3.weight', 'decoder.layers.0.src_trg_att.k_layer.bias', 'decoder.layers.0.src_trg_att.k_layer.weight', 'decoder.layers.0.src_trg_att.output_layer.bias', 'decoder.layers.0.src_trg_att.output_layer.weight', 'decoder.layers.0.src_trg_att.q_layer.bias', 'decoder.layers.0.src_trg_att.q_layer.weight', 'decoder.layers.0.src_trg_att.v_layer.bias', 'decoder.layers.0.src_trg_att.v_layer.weight', 'decoder.layers.0.trg_trg_att.k_layer.bias', 'decoder.layers.0.trg_trg_att.k_layer.weight', 'decoder.layers.0.trg_trg_att.output_layer.bias', 'decoder.layers.0.trg_trg_att.output_layer.weight', 'decoder.layers.0.trg_trg_att.q_layer.bias', 'decoder.layers.0.trg_trg_att.q_layer.weight', 'decoder.layers.0.trg_trg_att.v_layer.bias', 'decoder.layers.0.trg_trg_att.v_layer.weight', 'decoder.layers.0.x_layer_norm.bias', 'decoder.layers.0.x_layer_norm.weight', 'decoder.layers.1.dec_layer_norm.bias', 'decoder.layers.1.dec_layer_norm.weight', 'decoder.layers.1.feed_forward.layer_norm.bias', 'decoder.layers.1.feed_forward.layer_norm.weight', 'decoder.layers.1.feed_forward.pwff_layer.0.bias', 'decoder.layers.1.feed_forward.pwff_layer.0.weight', 'decoder.layers.1.feed_forward.pwff_layer.3.bias', 'decoder.layers.1.feed_forward.pwff_layer.3.weight', 'decoder.layers.1.src_trg_att.k_layer.bias', 'decoder.layers.1.src_trg_att.k_layer.weight', 'decoder.layers.1.src_trg_att.output_layer.bias', 'decoder.layers.1.src_trg_att.output_layer.weight', 'decoder.layers.1.src_trg_att.q_layer.bias', 'decoder.layers.1.src_trg_att.q_layer.weight', 'decoder.layers.1.src_trg_att.v_layer.bias', 'decoder.layers.1.src_trg_att.v_layer.weight', 'decoder.layers.1.trg_trg_att.k_layer.bias', 'decoder.layers.1.trg_trg_att.k_layer.weight', 'decoder.layers.1.trg_trg_att.output_layer.bias', 'decoder.layers.1.trg_trg_att.output_layer.weight', 'decoder.layers.1.trg_trg_att.q_layer.bias', 'decoder.layers.1.trg_trg_att.q_layer.weight', 'decoder.layers.1.trg_trg_att.v_layer.bias', 'decoder.layers.1.trg_trg_att.v_layer.weight', 'decoder.layers.1.x_layer_norm.bias', 'decoder.layers.1.x_layer_norm.weight', 'decoder.layers.2.dec_layer_norm.bias', 'decoder.layers.2.dec_layer_norm.weight', 'decoder.layers.2.feed_forward.layer_norm.bias', 'decoder.layers.2.feed_forward.layer_norm.weight', 'decoder.layers.2.feed_forward.pwff_layer.0.bias', 'decoder.layers.2.feed_forward.pwff_layer.0.weight', 'decoder.layers.2.feed_forward.pwff_layer.3.bias', 'decoder.layers.2.feed_forward.pwff_layer.3.weight', 'decoder.layers.2.src_trg_att.k_layer.bias', 'decoder.layers.2.src_trg_att.k_layer.weight', 'decoder.layers.2.src_trg_att.output_layer.bias', 'decoder.layers.2.src_trg_att.output_layer.weight', 'decoder.layers.2.src_trg_att.q_layer.bias', 'decoder.layers.2.src_trg_att.q_layer.weight', 'decoder.layers.2.src_trg_att.v_layer.bias', 'decoder.layers.2.src_trg_att.v_layer.weight', 'decoder.layers.2.trg_trg_att.k_layer.bias', 'decoder.layers.2.trg_trg_att.k_layer.weight', 'decoder.layers.2.trg_trg_att.output_layer.bias', 'decoder.layers.2.trg_trg_att.output_layer.weight', 'decoder.layers.2.trg_trg_att.q_layer.bias', 'decoder.layers.2.trg_trg_att.q_layer.weight', 'decoder.layers.2.trg_trg_att.v_layer.bias', 'decoder.layers.2.trg_trg_att.v_layer.weight', 'decoder.layers.2.x_layer_norm.bias', 'decoder.layers.2.x_layer_norm.weight', 'decoder.layers.3.dec_layer_norm.bias', 'decoder.layers.3.dec_layer_norm.weight', 'decoder.layers.3.feed_forward.layer_norm.bias', 'decoder.layers.3.feed_forward.layer_norm.weight', 'decoder.layers.3.feed_forward.pwff_layer.0.bias', 'decoder.layers.3.feed_forward.pwff_layer.0.weight', 'decoder.layers.3.feed_forward.pwff_layer.3.bias', 'decoder.layers.3.feed_forward.pwff_layer.3.weight', 'decoder.layers.3.src_trg_att.k_layer.bias', 'decoder.layers.3.src_trg_att.k_layer.weight', 'decoder.layers.3.src_trg_att.output_layer.bias', 'decoder.layers.3.src_trg_att.output_layer.weight', 'decoder.layers.3.src_trg_att.q_layer.bias', 'decoder.layers.3.src_trg_att.q_layer.weight', 'decoder.layers.3.src_trg_att.v_layer.bias', 'decoder.layers.3.src_trg_att.v_layer.weight', 'decoder.layers.3.trg_trg_att.k_layer.bias', 'decoder.layers.3.trg_trg_att.k_layer.weight', 'decoder.layers.3.trg_trg_att.output_layer.bias', 'decoder.layers.3.trg_trg_att.output_layer.weight', 'decoder.layers.3.trg_trg_att.q_layer.bias', 'decoder.layers.3.trg_trg_att.q_layer.weight', 'decoder.layers.3.trg_trg_att.v_layer.bias', 'decoder.layers.3.trg_trg_att.v_layer.weight', 'decoder.layers.3.x_layer_norm.bias', 'decoder.layers.3.x_layer_norm.weight', 'decoder.layers.4.dec_layer_norm.bias', 'decoder.layers.4.dec_layer_norm.weight', 'decoder.layers.4.feed_forward.layer_norm.bias', 'decoder.layers.4.feed_forward.layer_norm.weight', 'decoder.layers.4.feed_forward.pwff_layer.0.bias', 'decoder.layers.4.feed_forward.pwff_layer.0.weight', 'decoder.layers.4.feed_forward.pwff_layer.3.bias', 'decoder.layers.4.feed_forward.pwff_layer.3.weight', 'decoder.layers.4.src_trg_att.k_layer.bias', 'decoder.layers.4.src_trg_att.k_layer.weight', 'decoder.layers.4.src_trg_att.output_layer.bias', 'decoder.layers.4.src_trg_att.output_layer.weight', 'decoder.layers.4.src_trg_att.q_layer.bias', 'decoder.layers.4.src_trg_att.q_layer.weight', 'decoder.layers.4.src_trg_att.v_layer.bias', 'decoder.layers.4.src_trg_att.v_layer.weight', 'decoder.layers.4.trg_trg_att.k_layer.bias', 'decoder.layers.4.trg_trg_att.k_layer.weight', 'decoder.layers.4.trg_trg_att.output_layer.bias', 'decoder.layers.4.trg_trg_att.output_layer.weight', 'decoder.layers.4.trg_trg_att.q_layer.bias', 'decoder.layers.4.trg_trg_att.q_layer.weight', 'decoder.layers.4.trg_trg_att.v_layer.bias', 'decoder.layers.4.trg_trg_att.v_layer.weight', 'decoder.layers.4.x_layer_norm.bias', 'decoder.layers.4.x_layer_norm.weight', 'decoder.layers.5.dec_layer_norm.bias', 'decoder.layers.5.dec_layer_norm.weight', 'decoder.layers.5.feed_forward.layer_norm.bias', 'decoder.layers.5.feed_forward.layer_norm.weight', 'decoder.layers.5.feed_forward.pwff_layer.0.bias', 'decoder.layers.5.feed_forward.pwff_layer.0.weight', 'decoder.layers.5.feed_forward.pwff_layer.3.bias', 'decoder.layers.5.feed_forward.pwff_layer.3.weight', 'decoder.layers.5.src_trg_att.k_layer.bias', 'decoder.layers.5.src_trg_att.k_layer.weight', 'decoder.layers.5.src_trg_att.output_layer.bias', 'decoder.layers.5.src_trg_att.output_layer.weight', 'decoder.layers.5.src_trg_att.q_layer.bias', 'decoder.layers.5.src_trg_att.q_layer.weight', 'decoder.layers.5.src_trg_att.v_layer.bias', 'decoder.layers.5.src_trg_att.v_layer.weight', 'decoder.layers.5.trg_trg_att.k_layer.bias', 'decoder.layers.5.trg_trg_att.k_layer.weight', 'decoder.layers.5.trg_trg_att.output_layer.bias', 'decoder.layers.5.trg_trg_att.output_layer.weight', 'decoder.layers.5.trg_trg_att.q_layer.bias', 'decoder.layers.5.trg_trg_att.q_layer.weight', 'decoder.layers.5.trg_trg_att.v_layer.bias', 'decoder.layers.5.trg_trg_att.v_layer.weight', 'decoder.layers.5.x_layer_norm.bias', 'decoder.layers.5.x_layer_norm.weight', 'encoder.layer_norm.bias', 'encoder.layer_norm.weight', 'encoder.layers.0.feed_forward.layer_norm.bias', 'encoder.layers.0.feed_forward.layer_norm.weight', 'encoder.layers.0.feed_forward.pwff_layer.0.bias', 'encoder.layers.0.feed_forward.pwff_layer.0.weight', 'encoder.layers.0.feed_forward.pwff_layer.3.bias', 'encoder.layers.0.feed_forward.pwff_layer.3.weight', 'encoder.layers.0.layer_norm.bias', 'encoder.layers.0.layer_norm.weight', 'encoder.layers.0.src_src_att.k_layer.bias', 'encoder.layers.0.src_src_att.k_layer.weight', 'encoder.layers.0.src_src_att.output_layer.bias', 'encoder.layers.0.src_src_att.output_layer.weight', 'encoder.layers.0.src_src_att.q_layer.bias', 'encoder.layers.0.src_src_att.q_layer.weight', 'encoder.layers.0.src_src_att.v_layer.bias', 'encoder.layers.0.src_src_att.v_layer.weight', 'encoder.layers.1.feed_forward.layer_norm.bias', 'encoder.layers.1.feed_forward.layer_norm.weight', 'encoder.layers.1.feed_forward.pwff_layer.0.bias', 'encoder.layers.1.feed_forward.pwff_layer.0.weight', 'encoder.layers.1.feed_forward.pwff_layer.3.bias', 'encoder.layers.1.feed_forward.pwff_layer.3.weight', 'encoder.layers.1.layer_norm.bias', 'encoder.layers.1.layer_norm.weight', 'encoder.layers.1.src_src_att.k_layer.bias', 'encoder.layers.1.src_src_att.k_layer.weight', 'encoder.layers.1.src_src_att.output_layer.bias', 'encoder.layers.1.src_src_att.output_layer.weight', 'encoder.layers.1.src_src_att.q_layer.bias', 'encoder.layers.1.src_src_att.q_layer.weight', 'encoder.layers.1.src_src_att.v_layer.bias', 'encoder.layers.1.src_src_att.v_layer.weight', 'encoder.layers.2.feed_forward.layer_norm.bias', 'encoder.layers.2.feed_forward.layer_norm.weight', 'encoder.layers.2.feed_forward.pwff_layer.0.bias', 'encoder.layers.2.feed_forward.pwff_layer.0.weight', 'encoder.layers.2.feed_forward.pwff_layer.3.bias', 'encoder.layers.2.feed_forward.pwff_layer.3.weight', 'encoder.layers.2.layer_norm.bias', 'encoder.layers.2.layer_norm.weight', 'encoder.layers.2.src_src_att.k_layer.bias', 'encoder.layers.2.src_src_att.k_layer.weight', 'encoder.layers.2.src_src_att.output_layer.bias', 'encoder.layers.2.src_src_att.output_layer.weight', 'encoder.layers.2.src_src_att.q_layer.bias', 'encoder.layers.2.src_src_att.q_layer.weight', 'encoder.layers.2.src_src_att.v_layer.bias', 'encoder.layers.2.src_src_att.v_layer.weight', 'encoder.layers.3.feed_forward.layer_norm.bias', 'encoder.layers.3.feed_forward.layer_norm.weight', 'encoder.layers.3.feed_forward.pwff_layer.0.bias', 'encoder.layers.3.feed_forward.pwff_layer.0.weight', 'encoder.layers.3.feed_forward.pwff_layer.3.bias', 'encoder.layers.3.feed_forward.pwff_layer.3.weight', 'encoder.layers.3.layer_norm.bias', 'encoder.layers.3.layer_norm.weight', 'encoder.layers.3.src_src_att.k_layer.bias', 'encoder.layers.3.src_src_att.k_layer.weight', 'encoder.layers.3.src_src_att.output_layer.bias', 'encoder.layers.3.src_src_att.output_layer.weight', 'encoder.layers.3.src_src_att.q_layer.bias', 'encoder.layers.3.src_src_att.q_layer.weight', 'encoder.layers.3.src_src_att.v_layer.bias', 'encoder.layers.3.src_src_att.v_layer.weight', 'encoder.layers.4.feed_forward.layer_norm.bias', 'encoder.layers.4.feed_forward.layer_norm.weight', 'encoder.layers.4.feed_forward.pwff_layer.0.bias', 'encoder.layers.4.feed_forward.pwff_layer.0.weight', 'encoder.layers.4.feed_forward.pwff_layer.3.bias', 'encoder.layers.4.feed_forward.pwff_layer.3.weight', 'encoder.layers.4.layer_norm.bias', 'encoder.layers.4.layer_norm.weight', 'encoder.layers.4.src_src_att.k_layer.bias', 'encoder.layers.4.src_src_att.k_layer.weight', 'encoder.layers.4.src_src_att.output_layer.bias', 'encoder.layers.4.src_src_att.output_layer.weight', 'encoder.layers.4.src_src_att.q_layer.bias', 'encoder.layers.4.src_src_att.q_layer.weight', 'encoder.layers.4.src_src_att.v_layer.bias', 'encoder.layers.4.src_src_att.v_layer.weight', 'encoder.layers.5.feed_forward.layer_norm.bias', 'encoder.layers.5.feed_forward.layer_norm.weight', 'encoder.layers.5.feed_forward.pwff_layer.0.bias', 'encoder.layers.5.feed_forward.pwff_layer.0.weight', 'encoder.layers.5.feed_forward.pwff_layer.3.bias', 'encoder.layers.5.feed_forward.pwff_layer.3.weight', 'encoder.layers.5.layer_norm.bias', 'encoder.layers.5.layer_norm.weight', 'encoder.layers.5.src_src_att.k_layer.bias', 'encoder.layers.5.src_src_att.k_layer.weight', 'encoder.layers.5.src_src_att.output_layer.bias', 'encoder.layers.5.src_src_att.output_layer.weight', 'encoder.layers.5.src_src_att.q_layer.bias', 'encoder.layers.5.src_src_att.q_layer.weight', 'encoder.layers.5.src_src_att.v_layer.bias', 'encoder.layers.5.src_src_att.v_layer.weight', 'src_embed.lut.weight']\n",
            "2020-01-26 06:01:07,828 cfg.name                           : enish_transformer\n",
            "2020-01-26 06:01:07,829 cfg.data.src                       : en\n",
            "2020-01-26 06:01:07,829 cfg.data.trg                       : ish\n",
            "2020-01-26 06:01:07,829 cfg.data.train                     : /content/drive/My Drive/masakhane/en-ish-baseline/train.bpe\n",
            "2020-01-26 06:01:07,829 cfg.data.dev                       : /content/drive/My Drive/masakhane/en-ish-baseline/dev.bpe\n",
            "2020-01-26 06:01:07,829 cfg.data.test                      : /content/drive/My Drive/masakhane/en-ish-baseline/test.bpe\n",
            "2020-01-26 06:01:07,829 cfg.data.level                     : bpe\n",
            "2020-01-26 06:01:07,829 cfg.data.lowercase                 : False\n",
            "2020-01-26 06:01:07,829 cfg.data.max_sent_length           : 100\n",
            "2020-01-26 06:01:07,830 cfg.data.src_vocab                 : /content/drive/My Drive/masakhane/en-ish-baseline/vocab.txt\n",
            "2020-01-26 06:01:07,830 cfg.data.trg_vocab                 : /content/drive/My Drive/masakhane/en-ish-baseline/vocab.txt\n",
            "2020-01-26 06:01:07,830 cfg.testing.beam_size              : 5\n",
            "2020-01-26 06:01:07,830 cfg.testing.alpha                  : 1.0\n",
            "2020-01-26 06:01:07,830 cfg.training.random_seed           : 42\n",
            "2020-01-26 06:01:07,830 cfg.training.optimizer             : adam\n",
            "2020-01-26 06:01:07,830 cfg.training.normalization         : tokens\n",
            "2020-01-26 06:01:07,831 cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2020-01-26 06:01:07,831 cfg.training.scheduling            : plateau\n",
            "2020-01-26 06:01:07,831 cfg.training.patience              : 5\n",
            "2020-01-26 06:01:07,831 cfg.training.learning_rate_factor  : 0.5\n",
            "2020-01-26 06:01:07,831 cfg.training.learning_rate_warmup  : 1000\n",
            "2020-01-26 06:01:07,831 cfg.training.decrease_factor       : 0.7\n",
            "2020-01-26 06:01:07,831 cfg.training.loss                  : crossentropy\n",
            "2020-01-26 06:01:07,832 cfg.training.learning_rate         : 0.0003\n",
            "2020-01-26 06:01:07,832 cfg.training.learning_rate_min     : 1e-08\n",
            "2020-01-26 06:01:07,832 cfg.training.weight_decay          : 0.0\n",
            "2020-01-26 06:01:07,832 cfg.training.label_smoothing       : 0.1\n",
            "2020-01-26 06:01:07,832 cfg.training.batch_size            : 4096\n",
            "2020-01-26 06:01:07,832 cfg.training.batch_type            : token\n",
            "2020-01-26 06:01:07,832 cfg.training.eval_batch_size       : 3600\n",
            "2020-01-26 06:01:07,832 cfg.training.eval_batch_type       : token\n",
            "2020-01-26 06:01:07,832 cfg.training.batch_multiplier      : 1\n",
            "2020-01-26 06:01:07,833 cfg.training.early_stopping_metric : ppl\n",
            "2020-01-26 06:01:07,833 cfg.training.epochs                : 200\n",
            "2020-01-26 06:01:07,833 cfg.training.validation_freq       : 100\n",
            "2020-01-26 06:01:07,833 cfg.training.logging_freq          : 100\n",
            "2020-01-26 06:01:07,833 cfg.training.eval_metric           : bleu\n",
            "2020-01-26 06:01:07,833 cfg.training.model_dir             : /content/drive/My Drive/masakhane/en-ish-baseline/models/enish_transformer\n",
            "2020-01-26 06:01:07,833 cfg.training.overwrite             : True\n",
            "2020-01-26 06:01:07,833 cfg.training.shuffle               : True\n",
            "2020-01-26 06:01:07,833 cfg.training.use_cuda              : True\n",
            "2020-01-26 06:01:07,834 cfg.training.max_output_length     : 100\n",
            "2020-01-26 06:01:07,834 cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2020-01-26 06:01:07,834 cfg.training.keep_last_ckpts       : 3\n",
            "2020-01-26 06:01:07,834 cfg.model.initializer              : xavier\n",
            "2020-01-26 06:01:07,834 cfg.model.bias_initializer         : zeros\n",
            "2020-01-26 06:01:07,834 cfg.model.init_gain                : 1.0\n",
            "2020-01-26 06:01:07,834 cfg.model.embed_initializer        : xavier\n",
            "2020-01-26 06:01:07,835 cfg.model.embed_init_gain          : 1.0\n",
            "2020-01-26 06:01:07,835 cfg.model.tied_embeddings          : True\n",
            "2020-01-26 06:01:07,835 cfg.model.tied_softmax             : True\n",
            "2020-01-26 06:01:07,835 cfg.model.encoder.type             : transformer\n",
            "2020-01-26 06:01:07,835 cfg.model.encoder.num_layers       : 6\n",
            "2020-01-26 06:01:07,835 cfg.model.encoder.num_heads        : 4\n",
            "2020-01-26 06:01:07,835 cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2020-01-26 06:01:07,835 cfg.model.encoder.embeddings.scale : True\n",
            "2020-01-26 06:01:07,836 cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2020-01-26 06:01:07,836 cfg.model.encoder.hidden_size      : 256\n",
            "2020-01-26 06:01:07,836 cfg.model.encoder.ff_size          : 1024\n",
            "2020-01-26 06:01:07,836 cfg.model.encoder.dropout          : 0.3\n",
            "2020-01-26 06:01:07,836 cfg.model.decoder.type             : transformer\n",
            "2020-01-26 06:01:07,836 cfg.model.decoder.num_layers       : 6\n",
            "2020-01-26 06:01:07,836 cfg.model.decoder.num_heads        : 4\n",
            "2020-01-26 06:01:07,836 cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2020-01-26 06:01:07,837 cfg.model.decoder.embeddings.scale : True\n",
            "2020-01-26 06:01:07,837 cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2020-01-26 06:01:07,837 cfg.model.decoder.hidden_size      : 256\n",
            "2020-01-26 06:01:07,837 cfg.model.decoder.ff_size          : 1024\n",
            "2020-01-26 06:01:07,837 cfg.model.decoder.dropout          : 0.3\n",
            "2020-01-26 06:01:07,837 Data set sizes: \n",
            "\ttrain 4126,\n",
            "\tvalid 1000,\n",
            "\ttest 343\n",
            "2020-01-26 06:01:07,837 First training example:\n",
            "\t[SRC] The@@ ir unity would give a powerful witness , offering clear evidence that Jehovah had sent Jesus to the earth to do God’s will .\n",
            "\t[TRG] O@@ kugbe nọnsele dẹ wo sabọ sọ@@ sali ọbhọ ghe , Jehova gene ji Jesu re bhi ọne otọ nan nin ọle dọ lu iho nọns@@ ole . A@@ hoẹmhọn - egbe hi iy@@ aman nan ha rẹ gene lẹn edibo nesi Jesu .\n",
            "2020-01-26 06:01:07,838 First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) nin (7) ha (8) the (9) mhan\n",
            "2020-01-26 06:01:07,838 First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) . (5) , (6) nin (7) ha (8) the (9) mhan\n",
            "2020-01-26 06:01:07,839 Number of Src words (types): 3888\n",
            "2020-01-26 06:01:07,839 Number of Trg words (types): 3888\n",
            "2020-01-26 06:01:07,840 Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=3888),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=3888))\n",
            "2020-01-26 06:01:07,853 EPOCH 1\n",
            "2020-01-26 06:01:14,952 Epoch   1: total training loss 264.31\n",
            "2020-01-26 06:01:14,952 EPOCH 2\n",
            "2020-01-26 06:01:21,821 Epoch   2: total training loss 232.20\n",
            "2020-01-26 06:01:21,822 EPOCH 3\n",
            "2020-01-26 06:01:22,683 Epoch   3 Step:      100 Batch Loss:     4.786351 Tokens per Sec:    13717, Lr: 0.000300\n",
            "2020-01-26 06:02:05,690 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:02:05,690 Saving new checkpoint.\n",
            "2020-01-26 06:02:07,204 Example #0\n",
            "2020-01-26 06:02:07,205 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:02:07,206 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:02:07,206 \tHypothesis: : , , , , , , , , , .\n",
            "2020-01-26 06:02:07,206 Example #1\n",
            "2020-01-26 06:02:07,206 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:02:07,206 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:02:07,206 \tHypothesis: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , .\n",
            "2020-01-26 06:02:07,206 Example #2\n",
            "2020-01-26 06:02:07,207 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:02:07,207 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:02:07,207 \tHypothesis: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , .\n",
            "2020-01-26 06:02:07,207 Example #3\n",
            "2020-01-26 06:02:07,207 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:02:07,207 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:02:07,207 \tHypothesis: : : , , , .\n",
            "2020-01-26 06:02:07,208 Validation result (greedy) at epoch   3, step      100: bleu:   0.00, loss: 116888.2266, ppl: 126.0511, duration: 44.5245s\n",
            "2020-01-26 06:02:13,683 Epoch   3: total training loss 229.42\n",
            "2020-01-26 06:02:13,683 EPOCH 4\n",
            "2020-01-26 06:02:20,898 Epoch   4: total training loss 217.70\n",
            "2020-01-26 06:02:20,898 EPOCH 5\n",
            "2020-01-26 06:02:22,513 Epoch   5 Step:      200 Batch Loss:     4.343826 Tokens per Sec:    12279, Lr: 0.000300\n",
            "2020-01-26 06:03:07,298 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:03:07,298 Saving new checkpoint.\n",
            "2020-01-26 06:03:08,655 Example #0\n",
            "2020-01-26 06:03:08,656 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:03:08,656 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:03:08,656 \tHypothesis: ( : : : ( : ( : ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:03:08,656 Example #1\n",
            "2020-01-26 06:03:08,656 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:03:08,656 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:03:08,656 \tHypothesis: ( : Mhan , : : ( : ( : ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:03:08,656 Example #2\n",
            "2020-01-26 06:03:08,657 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:03:08,657 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:03:08,657 \tHypothesis: Mhan , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,\n",
            "2020-01-26 06:03:08,657 Example #3\n",
            "2020-01-26 06:03:08,657 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:03:08,657 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:03:08,657 \tHypothesis: ( : : : Be : : Be : ( : ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:03:08,657 Validation result (greedy) at epoch   5, step      200: bleu:   0.00, loss: 112468.1953, ppl: 104.9823, duration: 46.1441s\n",
            "2020-01-26 06:03:14,601 Epoch   5: total training loss 216.89\n",
            "2020-01-26 06:03:14,601 EPOCH 6\n",
            "2020-01-26 06:03:22,081 Epoch   6: total training loss 203.88\n",
            "2020-01-26 06:03:22,081 EPOCH 7\n",
            "2020-01-26 06:03:24,613 Epoch   7 Step:      300 Batch Loss:     4.409139 Tokens per Sec:    12810, Lr: 0.000300\n",
            "2020-01-26 06:04:10,739 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:04:10,739 Saving new checkpoint.\n",
            "2020-01-26 06:04:11,882 Example #0\n",
            "2020-01-26 06:04:11,883 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:04:11,883 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:04:11,883 \tHypothesis: ( : 1 : ( : ( : ( 1 : ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:04:11,883 Example #1\n",
            "2020-01-26 06:04:11,884 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:04:11,884 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:04:11,884 \tHypothesis: ( : ( : ( : ( 1 ) ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:04:11,884 Example #2\n",
            "2020-01-26 06:04:11,884 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:04:11,884 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:04:11,884 \tHypothesis: ( : ( ) ( ) ( ) ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:04:11,885 Example #3\n",
            "2020-01-26 06:04:11,885 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:04:11,885 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:04:11,885 \tHypothesis: ( : 1 : ( : ( : ( 1 : ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (\n",
            "2020-01-26 06:04:11,885 Validation result (greedy) at epoch   7, step      300: bleu:   0.00, loss: 101796.1875, ppl:  67.5047, duration: 47.2717s\n",
            "2020-01-26 06:04:16,689 Epoch   7: total training loss 195.63\n",
            "2020-01-26 06:04:16,689 EPOCH 8\n",
            "2020-01-26 06:04:24,165 Epoch   8: total training loss 189.38\n",
            "2020-01-26 06:04:24,165 EPOCH 9\n",
            "2020-01-26 06:04:27,495 Epoch   9 Step:      400 Batch Loss:     3.988022 Tokens per Sec:    13391, Lr: 0.000300\n",
            "2020-01-26 06:05:13,678 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:05:13,679 Saving new checkpoint.\n",
            "2020-01-26 06:05:14,812 Example #0\n",
            "2020-01-26 06:05:14,813 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:05:14,813 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:05:14,813 \tHypothesis: Bhi , ọle da ha re egbe .\n",
            "2020-01-26 06:05:14,813 Example #1\n",
            "2020-01-26 06:05:14,814 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:05:14,814 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:05:14,814 \tHypothesis: Bhi , ọle da ha ha re obọ bhi ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne\n",
            "2020-01-26 06:05:14,814 Example #2\n",
            "2020-01-26 06:05:14,814 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:05:14,814 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:05:14,814 \tHypothesis: Bhi , ọle da ha ha re ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne\n",
            "2020-01-26 06:05:14,814 Example #3\n",
            "2020-01-26 06:05:14,815 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:05:14,815 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:05:14,815 \tHypothesis: Bhi , ọle da ha ha re egbe .\n",
            "2020-01-26 06:05:14,815 Validation result (greedy) at epoch   9, step      400: bleu:   0.29, loss: 95543.6719, ppl:  52.1162, duration: 47.3192s\n",
            "2020-01-26 06:05:19,267 Epoch   9: total training loss 184.09\n",
            "2020-01-26 06:05:19,268 EPOCH 10\n",
            "2020-01-26 06:05:26,809 Epoch  10: total training loss 177.01\n",
            "2020-01-26 06:05:26,809 EPOCH 11\n",
            "2020-01-26 06:05:31,040 Epoch  11 Step:      500 Batch Loss:     3.519213 Tokens per Sec:    12817, Lr: 0.000300\n",
            "2020-01-26 06:06:16,695 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:06:16,695 Saving new checkpoint.\n",
            "2020-01-26 06:06:17,717 Example #0\n",
            "2020-01-26 06:06:17,717 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:06:17,718 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:06:17,718 \tHypothesis: ( 1 : 1 ) Mhan da ha re egbe .\n",
            "2020-01-26 06:06:17,718 Example #1\n",
            "2020-01-26 06:06:17,718 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:06:17,718 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:06:17,718 \tHypothesis: Mhan da ha ha re egbe bhi ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne oga .\n",
            "2020-01-26 06:06:17,718 Example #2\n",
            "2020-01-26 06:06:17,718 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:06:17,719 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:06:17,719 \tHypothesis: Bhi ọsi ene bunbun , ọle da ha re ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne ọne agbọn nan .\n",
            "2020-01-26 06:06:17,719 Example #3\n",
            "2020-01-26 06:06:17,719 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:06:17,719 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:06:17,719 \tHypothesis: Be bhọ mhan ha ha lu emhin nin mhan rẹ ha lu emhin nin mhan rẹ ha lu emhin nin mhan .\n",
            "2020-01-26 06:06:17,719 Validation result (greedy) at epoch  11, step      500: bleu:   0.97, loss: 90320.1328, ppl:  41.9859, duration: 46.6791s\n",
            "2020-01-26 06:06:20,827 Epoch  11: total training loss 173.63\n",
            "2020-01-26 06:06:20,827 EPOCH 12\n",
            "2020-01-26 06:06:28,309 Epoch  12: total training loss 167.96\n",
            "2020-01-26 06:06:28,309 EPOCH 13\n",
            "2020-01-26 06:06:33,432 Epoch  13 Step:      600 Batch Loss:     3.638573 Tokens per Sec:    13123, Lr: 0.000300\n",
            "2020-01-26 06:07:19,125 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:07:19,126 Saving new checkpoint.\n",
            "2020-01-26 06:07:20,267 Example #0\n",
            "2020-01-26 06:07:20,269 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:07:20,269 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:07:20,269 \tHypothesis: Bhi ọsi ẹmhọanta , ọle da ha re obọ .\n",
            "2020-01-26 06:07:20,269 Example #1\n",
            "2020-01-26 06:07:20,269 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:07:20,269 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:07:20,270 \tHypothesis: Bhi ọsi ẹmhọanta , ọle da ha re obọ bhi ọne agbọn nan , ọle da ha re obọ .\n",
            "2020-01-26 06:07:20,270 Example #2\n",
            "2020-01-26 06:07:20,270 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:07:20,270 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:07:20,270 \tHypothesis: Bhi ọsi ijiẹmhin , ọle da ha re obọ bhi ọne agbọn nan , ọle da ha re obọ bhi ọne agbọn nan .\n",
            "2020-01-26 06:07:20,270 Example #3\n",
            "2020-01-26 06:07:20,271 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:07:20,271 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:07:20,271 \tHypothesis: ( b ) Be bhọ mhan ha rẹ ha rẹ ha lu emhin nin mhan rẹ ha lu emhin .\n",
            "2020-01-26 06:07:20,271 Validation result (greedy) at epoch  13, step      600: bleu:   1.40, loss: 86108.0625, ppl:  35.2704, duration: 46.8388s\n",
            "2020-01-26 06:07:22,416 Epoch  13: total training loss 164.86\n",
            "2020-01-26 06:07:22,416 EPOCH 14\n",
            "2020-01-26 06:07:29,710 Epoch  14: total training loss 161.66\n",
            "2020-01-26 06:07:29,711 EPOCH 15\n",
            "2020-01-26 06:07:36,152 Epoch  15 Step:      700 Batch Loss:     3.295609 Tokens per Sec:    12699, Lr: 0.000300\n",
            "2020-01-26 06:08:21,896 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:08:21,896 Saving new checkpoint.\n",
            "2020-01-26 06:08:23,045 Example #0\n",
            "2020-01-26 06:08:23,046 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:08:23,046 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:08:23,046 \tHypothesis: Ọle da wo ha yi mẹn bhi ọne agbọn nan .\n",
            "2020-01-26 06:08:23,046 Example #1\n",
            "2020-01-26 06:08:23,046 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:08:23,046 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:08:23,046 \tHypothesis: Bhi ọsi ẹmhọanta , ọle da ha re obọ , ọle da ha re obọ rẹkhan emhin nin ele rẹ ha re obọ .\n",
            "2020-01-26 06:08:23,046 Example #2\n",
            "2020-01-26 06:08:23,047 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:08:23,047 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:08:23,047 \tHypothesis: Bhi ọsi ẹmhọanta , mhan ki ha re obọ , ọle da ha re obọ rẹkhan adia nin ele rẹ ha re obọ , ọle da ha re izebhudu nin ele rẹ ha re obọ rẹkhan emhin nin ele .\n",
            "2020-01-26 06:08:23,047 Example #3\n",
            "2020-01-26 06:08:23,047 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:08:23,047 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:08:23,047 \tHypothesis: Be bhọ mhan ha rẹ ha re egbe khọkhọ Osẹnobulua .\n",
            "2020-01-26 06:08:23,047 Validation result (greedy) at epoch  15, step      700: bleu:   1.91, loss: 83404.9141, ppl:  31.5379, duration: 46.8946s\n",
            "2020-01-26 06:08:24,255 Epoch  15: total training loss 158.51\n",
            "2020-01-26 06:08:24,255 EPOCH 16\n",
            "2020-01-26 06:08:31,413 Epoch  16: total training loss 155.32\n",
            "2020-01-26 06:08:31,413 EPOCH 17\n",
            "2020-01-26 06:08:38,563 Epoch  17 Step:      800 Batch Loss:     3.353125 Tokens per Sec:    13172, Lr: 0.000300\n",
            "2020-01-26 06:09:24,611 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:09:24,612 Saving new checkpoint.\n",
            "2020-01-26 06:09:25,718 Example #0\n",
            "2020-01-26 06:09:25,719 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:09:25,719 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:09:25,719 \tHypothesis: Bhi ọsi ẹmhọanta , mẹn da ha yi mẹn bhi ọne agbọn nan .\n",
            "2020-01-26 06:09:25,719 Example #1\n",
            "2020-01-26 06:09:25,719 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:09:25,719 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:09:25,719 \tHypothesis: Bhi ọsi ẹmhọanta , ọle da ha yi ikpe ne bunbun ne bunbun ne bunbun ne bunbun ne bunbun .\n",
            "2020-01-26 06:09:25,720 Example #2\n",
            "2020-01-26 06:09:25,720 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:09:25,720 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:09:25,720 \tHypothesis: ( 1 Cor . 1 : 1 ) Mhan da ha re izebhudu nin mhan rẹ ha re obọ rẹkhan adia nọnsi Jehova , ọle ki ha re izebhudu nin mhan rẹ ha re obọ rẹkhan adia nọnsi Jehova , bi ene biẹ ele da ha mhọn emhin ne bunbun .\n",
            "2020-01-26 06:09:25,720 Example #3\n",
            "2020-01-26 06:09:25,720 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:09:25,721 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:09:25,721 \tHypothesis: ( b ) Be bhọ mhan ha rẹ ha re egbe khọkhọ mhan , mhan ki sabọ ha re egbe khọkhọ mhan .\n",
            "2020-01-26 06:09:25,721 Validation result (greedy) at epoch  17, step      800: bleu:   2.38, loss: 81618.1953, ppl:  29.2903, duration: 47.1577s\n",
            "2020-01-26 06:09:25,910 Epoch  17: total training loss 152.99\n",
            "2020-01-26 06:09:25,910 EPOCH 18\n",
            "2020-01-26 06:09:33,208 Epoch  18: total training loss 150.52\n",
            "2020-01-26 06:09:33,208 EPOCH 19\n",
            "2020-01-26 06:09:40,890 Epoch  19: total training loss 148.12\n",
            "2020-01-26 06:09:40,891 EPOCH 20\n",
            "2020-01-26 06:09:41,688 Epoch  20 Step:      900 Batch Loss:     2.226691 Tokens per Sec:    11629, Lr: 0.000300\n",
            "2020-01-26 06:10:27,566 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:10:27,566 Saving new checkpoint.\n",
            "2020-01-26 06:10:28,669 Example #0\n",
            "2020-01-26 06:10:28,670 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:10:28,670 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:10:28,670 \tHypothesis: Bhi ọsi ẹmhọanta , mẹn da wo ha yi mẹn mẹn .\n",
            "2020-01-26 06:10:28,670 Example #1\n",
            "2020-01-26 06:10:28,671 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:10:28,671 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:10:28,671 \tHypothesis: Bhi ọsi ẹmhọanta , ọle da ha yi ikpe ne bunbun ne bunbun ne bunbun ne bunbun ne bunbun .\n",
            "2020-01-26 06:10:28,671 Example #2\n",
            "2020-01-26 06:10:28,672 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:10:28,672 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:10:28,672 \tHypothesis: ( John 4 : 1 ) Mhan da ha yi emhin nọn ribhi Baibo , mhan ki ha re izebhudu nin mhan rẹ ha re izebhudu nin mhan rẹ ha lu iwẹnna itẹmhọn Osẹnobulua .\n",
            "2020-01-26 06:10:28,672 Example #3\n",
            "2020-01-26 06:10:28,673 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:10:28,673 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:10:28,673 \tHypothesis: Be bhọ mhan ha rẹ sabọ rẹkpa mhan rẹ ha lu emhin nin mhan rẹ ha lu emhin .\n",
            "2020-01-26 06:10:28,673 Validation result (greedy) at epoch  20, step      900: bleu:   2.58, loss: 79700.1953, ppl:  27.0556, duration: 46.9852s\n",
            "2020-01-26 06:10:35,284 Epoch  20: total training loss 145.45\n",
            "2020-01-26 06:10:35,285 EPOCH 21\n",
            "2020-01-26 06:10:42,872 Epoch  21: total training loss 143.53\n",
            "2020-01-26 06:10:42,872 EPOCH 22\n",
            "2020-01-26 06:10:44,565 Epoch  22 Step:     1000 Batch Loss:     3.257029 Tokens per Sec:    13712, Lr: 0.000300\n",
            "2020-01-26 06:11:30,524 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:11:30,525 Saving new checkpoint.\n",
            "2020-01-26 06:11:31,577 Example #0\n",
            "2020-01-26 06:11:31,577 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:11:31,578 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:11:31,578 \tHypothesis: Ọle da wo ha yi mẹn bhi ọne okhuo .\n",
            "2020-01-26 06:11:31,578 Example #1\n",
            "2020-01-26 06:11:31,578 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:11:31,578 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:11:31,578 \tHypothesis: Bhi ọsi ẹmhọanta , ele da ha yi ikpe ne bunbun ne bunbun ne bunbun ne bunbun ne bunbun .\n",
            "2020-01-26 06:11:31,578 Example #2\n",
            "2020-01-26 06:11:31,579 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:11:31,579 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:11:31,579 \tHypothesis: Mhan ha re obọ rẹkhan adia nin mhan rẹ ha yi emhin ne ribhi Baibo , mhan ki ha yi emhin ne ribhi Baibo , mhan ki sabọ ha yi emhin ne ribhi Baibo , ranmhude emhin ne bunbun ne bunbun ne bunbun ne bunbun .\n",
            "2020-01-26 06:11:31,579 Example #3\n",
            "2020-01-26 06:11:31,579 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:11:31,579 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:11:31,579 \tHypothesis: Mhan ha sabọ rẹkpa mhan rẹ ha mhọn emhin bhi iẹnlẹn .\n",
            "2020-01-26 06:11:31,579 Validation result (greedy) at epoch  22, step     1000: bleu:   2.65, loss: 78288.6562, ppl:  25.5206, duration: 47.0134s\n",
            "2020-01-26 06:11:37,287 Epoch  22: total training loss 141.50\n",
            "2020-01-26 06:11:37,287 EPOCH 23\n",
            "2020-01-26 06:11:44,976 Epoch  23: total training loss 139.42\n",
            "2020-01-26 06:11:44,977 EPOCH 24\n",
            "2020-01-26 06:11:47,556 Epoch  24 Step:     1100 Batch Loss:     2.881152 Tokens per Sec:    13532, Lr: 0.000300\n",
            "2020-01-26 06:12:33,313 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:12:33,313 Saving new checkpoint.\n",
            "2020-01-26 06:12:34,530 Example #0\n",
            "2020-01-26 06:12:34,531 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:12:34,532 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:12:34,532 \tHypothesis: Ọle da wo ha yi mẹn bhi ọne okhuo mẹn .\n",
            "2020-01-26 06:12:34,532 Example #1\n",
            "2020-01-26 06:12:34,532 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:12:34,532 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:12:34,532 \tHypothesis: Ọ da wo ha yi emhin nin ele rẹ ha yi emhin nin ele rẹ ha yi emhin nin ele ha mhọn da ele , ele da ha yi emhin nin ele rẹ ha yi emhin nin ele .\n",
            "2020-01-26 06:12:34,532 Example #2\n",
            "2020-01-26 06:12:34,533 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:12:34,533 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:12:34,533 \tHypothesis: Mhan ha re obọ rẹkhan adia nesi Jesu , mhan ki ha re izebhudu nin mhan , nin mhan rẹ ha re izebhudu nin ele , nin ele da sabọ ha re izebhudu nin ele , bi emhin nin ele rẹ ha re obọ rẹkhan adia nọnsi Osẹnobulua .\n",
            "2020-01-26 06:12:34,533 Example #3\n",
            "2020-01-26 06:12:34,533 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:12:34,533 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:12:34,533 \tHypothesis: Mhan ha sabọ rẹkpa mhan rẹ ha mhọn urẹọbhọ nọnsi Jehova .\n",
            "2020-01-26 06:12:34,533 Validation result (greedy) at epoch  24, step     1100: bleu:   2.94, loss: 77336.5234, ppl:  24.5347, duration: 46.9770s\n",
            "2020-01-26 06:12:39,420 Epoch  24: total training loss 137.79\n",
            "2020-01-26 06:12:39,420 EPOCH 25\n",
            "2020-01-26 06:12:46,852 Epoch  25: total training loss 135.86\n",
            "2020-01-26 06:12:46,853 EPOCH 26\n",
            "2020-01-26 06:12:50,379 Epoch  26 Step:     1200 Batch Loss:     2.626782 Tokens per Sec:    13105, Lr: 0.000300\n",
            "2020-01-26 06:13:36,244 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:13:36,245 Saving new checkpoint.\n",
            "2020-01-26 06:13:37,309 Example #0\n",
            "2020-01-26 06:13:37,310 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:13:37,310 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:13:37,310 \tHypothesis: Mẹn da wo ha yi mẹn bhi enin .\n",
            "2020-01-26 06:13:37,311 Example #1\n",
            "2020-01-26 06:13:37,311 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:13:37,311 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:13:37,311 \tHypothesis: Ọle da ha re egbe khọkhọ ene biẹ ọmọn , ele da ha re egbe khọkhọ ele , ele da ha re egbe khọkhọ ele .\n",
            "2020-01-26 06:13:37,312 Example #2\n",
            "2020-01-26 06:13:37,312 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:13:37,312 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:13:37,312 \tHypothesis: Mhan ha re obọ rẹkhan adia nin ele , mhan ki ha re obọ rẹkhan adia nin ele , ele ki ha re egbe khọkhọ ọne agbọn nan , bi emhin nin ele rẹ ha re obọ rẹkhan ọlẹn .\n",
            "2020-01-26 06:13:37,313 Example #3\n",
            "2020-01-26 06:13:37,313 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:13:37,313 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:13:37,313 \tHypothesis: Be bhọ mhan ha rẹ sabọ rẹkpa mhan yẹ , mhan ki sabọ ha mhọn urẹọbhọ nọnsi Jehova .\n",
            "2020-01-26 06:13:37,313 Validation result (greedy) at epoch  26, step     1200: bleu:   3.21, loss: 76556.4766, ppl:  23.7554, duration: 46.9339s\n",
            "2020-01-26 06:13:41,182 Epoch  26: total training loss 134.28\n",
            "2020-01-26 06:13:41,182 EPOCH 27\n",
            "2020-01-26 06:13:48,485 Epoch  27: total training loss 131.93\n",
            "2020-01-26 06:13:48,485 EPOCH 28\n",
            "2020-01-26 06:13:52,991 Epoch  28 Step:     1300 Batch Loss:     2.923473 Tokens per Sec:    13603, Lr: 0.000300\n",
            "2020-01-26 06:14:38,869 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:14:38,870 Saving new checkpoint.\n",
            "2020-01-26 06:14:40,121 Example #0\n",
            "2020-01-26 06:14:40,122 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:14:40,123 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:14:40,123 \tHypothesis: Mẹn da wo ha yi mẹn bhi obọ mẹn .\n",
            "2020-01-26 06:14:40,123 Example #1\n",
            "2020-01-26 06:14:40,123 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:14:40,123 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:14:40,123 \tHypothesis: Ọle da ha yi ọkpa bhi ọne otọ nan , ọle da ha yi ọria nọn ha yi ọria nọn ha yi emhin nin ele ha lu .\n",
            "2020-01-26 06:14:40,124 Example #2\n",
            "2020-01-26 06:14:40,124 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:14:40,124 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:14:40,124 \tHypothesis: Mhan dẹ sabọ ha re obọ rẹkhan adia nesi Jehova , mhan ki ha re obọ rẹkhan adia nesi Jehova , la sade mhan re obọ rẹkhan adia nesi Jehova , la sade mhan re obọ rẹkhan adia nesi Jehova .\n",
            "2020-01-26 06:14:40,124 Example #3\n",
            "2020-01-26 06:14:40,124 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:14:40,125 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:14:40,125 \tHypothesis: Mhan dẹ sabọ ha re egbe khọkhọ iJehova .\n",
            "2020-01-26 06:14:40,125 Validation result (greedy) at epoch  28, step     1300: bleu:   3.73, loss: 75516.3906, ppl:  22.7547, duration: 47.1329s\n",
            "2020-01-26 06:14:42,975 Epoch  28: total training loss 130.09\n",
            "2020-01-26 06:14:42,975 EPOCH 29\n",
            "2020-01-26 06:14:50,506 Epoch  29: total training loss 132.18\n",
            "2020-01-26 06:14:50,507 EPOCH 30\n",
            "2020-01-26 06:14:55,892 Epoch  30 Step:     1400 Batch Loss:     2.957211 Tokens per Sec:    13033, Lr: 0.000300\n",
            "2020-01-26 06:15:41,685 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:15:41,686 Saving new checkpoint.\n",
            "2020-01-26 06:15:43,226 Example #0\n",
            "2020-01-26 06:15:43,226 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:15:43,226 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:15:43,227 \tHypothesis: Mẹn da wo ha yi mẹn bhi enin .\n",
            "2020-01-26 06:15:43,227 Example #1\n",
            "2020-01-26 06:15:43,227 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:15:43,227 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:15:43,227 \tHypothesis: ( Prov . 6 : 1 ) Bhiriọ , ele da ha re izebhudu nin ele rẹ ha re izebhudu nin ele .\n",
            "2020-01-26 06:15:43,227 Example #2\n",
            "2020-01-26 06:15:43,228 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:15:43,228 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:15:43,228 \tHypothesis: Mhan ha miẹn elele bhi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn bi ene biẹ ọmọn .\n",
            "2020-01-26 06:15:43,228 Example #3\n",
            "2020-01-26 06:15:43,228 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:15:43,228 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:15:43,228 \tHypothesis: Mhan ha miẹn emhin bhi egbe .\n",
            "2020-01-26 06:15:43,228 Validation result (greedy) at epoch  30, step     1400: bleu:   3.51, loss: 74975.6016, ppl:  22.2512, duration: 47.3361s\n",
            "2020-01-26 06:15:45,306 Epoch  30: total training loss 126.63\n",
            "2020-01-26 06:15:45,306 EPOCH 31\n",
            "2020-01-26 06:15:52,868 Epoch  31: total training loss 125.45\n",
            "2020-01-26 06:15:52,869 EPOCH 32\n",
            "2020-01-26 06:15:58,897 Epoch  32 Step:     1500 Batch Loss:     2.731450 Tokens per Sec:    13521, Lr: 0.000300\n",
            "2020-01-26 06:16:45,044 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:16:45,044 Saving new checkpoint.\n",
            "2020-01-26 06:16:46,048 Example #0\n",
            "2020-01-26 06:16:46,048 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:16:46,048 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:16:46,048 \tHypothesis: Mẹn da wo ha yi mẹn bhi obọ mẹn .\n",
            "2020-01-26 06:16:46,048 Example #1\n",
            "2020-01-26 06:16:46,049 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:16:46,049 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:16:46,049 \tHypothesis: Bhi ọsi ẹmhọanta , ẹbho ne bunbun ne bunbun ne bunbun dọ ha mhọn ẹmhọn Osẹnobulua .\n",
            "2020-01-26 06:16:46,049 Example #2\n",
            "2020-01-26 06:16:46,049 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:16:46,049 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:16:46,049 \tHypothesis: Mhan ha re obọ rẹkhan adia nesi Jehova , mhan ki ha re obọ rẹkhan adia nesi Jehova , ranmhude mhan mhọn isẹhoa nin ele rẹ ha mhọn bhi ọne iwẹnna nan .\n",
            "2020-01-26 06:16:46,049 Example #3\n",
            "2020-01-26 06:16:46,050 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:16:46,050 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:16:46,050 \tHypothesis: Mhan ha miẹn elele bhi egbe .\n",
            "2020-01-26 06:16:46,050 Validation result (greedy) at epoch  32, step     1500: bleu:   3.62, loss: 74599.6406, ppl:  21.9077, duration: 47.1525s\n",
            "2020-01-26 06:16:47,239 Epoch  32: total training loss 123.27\n",
            "2020-01-26 06:16:47,240 EPOCH 33\n",
            "2020-01-26 06:16:54,372 Epoch  33: total training loss 121.43\n",
            "2020-01-26 06:16:54,373 EPOCH 34\n",
            "2020-01-26 06:17:01,662 Epoch  34 Step:     1600 Batch Loss:     2.268804 Tokens per Sec:    12931, Lr: 0.000300\n",
            "2020-01-26 06:17:47,829 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:17:47,829 Saving new checkpoint.\n",
            "2020-01-26 06:17:48,951 Example #0\n",
            "2020-01-26 06:17:48,952 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:17:48,952 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:17:48,953 \tHypothesis: Mẹn da ha yi mẹn bhi okhuo mẹn .\n",
            "2020-01-26 06:17:48,953 Example #1\n",
            "2020-01-26 06:17:48,953 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:17:48,954 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:17:48,954 \tHypothesis: Ọne ẹghe nan , ọle da ha re izebhudu nan , ele da ha re izebhudu nin ele .\n",
            "2020-01-26 06:17:48,954 Example #2\n",
            "2020-01-26 06:17:48,955 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:17:48,955 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:17:48,955 \tHypothesis: Mhan da ha re obọ rẹkhan uhi bi ene ewanlẹn bhi agbotu , mhan ki ha mhọn isẹhoa nin ele rẹ ha mhọn bhi agbaẹbho , bi ene ẹbho ne bunbun da ha mhọn isẹhoa nin ele rẹ ha mhọn ọne iwẹnna nan .\n",
            "2020-01-26 06:17:48,955 Example #3\n",
            "2020-01-26 06:17:48,956 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:17:48,956 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:17:48,957 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn isẹhoa ọsaje .\n",
            "2020-01-26 06:17:48,957 Validation result (greedy) at epoch  34, step     1600: bleu:   3.95, loss: 74023.3047, ppl:  21.3914, duration: 47.2945s\n",
            "2020-01-26 06:17:49,171 Epoch  34: total training loss 119.27\n",
            "2020-01-26 06:17:49,172 EPOCH 35\n",
            "2020-01-26 06:17:56,662 Epoch  35: total training loss 118.11\n",
            "2020-01-26 06:17:56,663 EPOCH 36\n",
            "2020-01-26 06:18:03,945 Epoch  36: total training loss 119.21\n",
            "2020-01-26 06:18:03,945 EPOCH 37\n",
            "2020-01-26 06:18:04,611 Epoch  37 Step:     1700 Batch Loss:     2.369793 Tokens per Sec:    14355, Lr: 0.000300\n",
            "2020-01-26 06:18:50,528 Example #0\n",
            "2020-01-26 06:18:50,529 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:18:50,529 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:18:50,529 \tHypothesis: Mẹn da ha yi mẹn bhi Esali Jehova .\n",
            "2020-01-26 06:18:50,530 Example #1\n",
            "2020-01-26 06:18:50,530 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:18:50,530 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:18:50,530 \tHypothesis: Ọle da ha re obọ rẹkhan adia nesi Jehova , ele ki ha re obọ rẹkhan adia nesi Jehova .\n",
            "2020-01-26 06:18:50,531 Example #2\n",
            "2020-01-26 06:18:50,531 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:18:50,531 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:18:50,531 \tHypothesis: Mhan ha re obọ rẹkhan uwedẹ nin ele rẹ ha mhọn ọne isẹhoa nan rẹ ha ribhi ọne agbọn nan , bi ene biẹ ọmọn ki ha mhọn isẹhoa nin ele rẹ ha mhọn ọne isẹhoa nan .\n",
            "2020-01-26 06:18:50,531 Example #3\n",
            "2020-01-26 06:18:50,532 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:18:50,532 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:18:50,532 \tHypothesis: Mhan ha sabọ rẹkpa mhan rẹ sabọ ha mhọn urẹọbhọ bọsi eria .\n",
            "2020-01-26 06:18:50,532 Validation result (greedy) at epoch  37, step     1700: bleu:   3.81, loss: 74392.2812, ppl:  21.7205, duration: 45.9213s\n",
            "2020-01-26 06:18:57,037 Epoch  37: total training loss 115.12\n",
            "2020-01-26 06:18:57,037 EPOCH 38\n",
            "2020-01-26 06:19:04,241 Epoch  38: total training loss 112.91\n",
            "2020-01-26 06:19:04,241 EPOCH 39\n",
            "2020-01-26 06:19:05,798 Epoch  39 Step:     1800 Batch Loss:     2.360721 Tokens per Sec:    13783, Lr: 0.000300\n",
            "2020-01-26 06:19:51,795 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:19:51,796 Saving new checkpoint.\n",
            "2020-01-26 06:19:52,972 Example #0\n",
            "2020-01-26 06:19:52,972 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:19:52,972 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:19:52,972 \tHypothesis: Mẹn da wo ha yi mẹn bhi agbotu .\n",
            "2020-01-26 06:19:52,973 Example #1\n",
            "2020-01-26 06:19:52,973 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:19:52,973 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:19:52,973 \tHypothesis: Ene ẹbho ne bunbun wo manman ha mhọn urẹọbhọ da ẹbho ne bunbun .\n",
            "2020-01-26 06:19:52,973 Example #2\n",
            "2020-01-26 06:19:52,973 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:19:52,974 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:19:52,974 \tHypothesis: Mhan da ha re ọne isẹhoa nan rẹ ha mhọn isẹhoa nin ele rẹ ha ribhi ọne agbọn nan , bi ene ga iJehova , bi ene biẹ ọmọn bi ene ga iJehova .\n",
            "2020-01-26 06:19:52,974 Example #3\n",
            "2020-01-26 06:19:52,974 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:19:52,974 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:19:52,974 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ sabọ ha mhọn urẹọbhọ bọsi eria .\n",
            "2020-01-26 06:19:52,974 Validation result (greedy) at epoch  39, step     1800: bleu:   4.18, loss: 73928.2891, ppl:  21.3075, duration: 47.1763s\n",
            "2020-01-26 06:19:58,715 Epoch  39: total training loss 111.43\n",
            "2020-01-26 06:19:58,716 EPOCH 40\n",
            "2020-01-26 06:20:06,311 Epoch  40: total training loss 109.70\n",
            "2020-01-26 06:20:06,312 EPOCH 41\n",
            "2020-01-26 06:20:08,691 Epoch  41 Step:     1900 Batch Loss:     2.476414 Tokens per Sec:    13159, Lr: 0.000300\n",
            "2020-01-26 06:20:54,570 Hooray! New best validation result [ppl]!\n",
            "2020-01-26 06:20:54,570 Saving new checkpoint.\n",
            "2020-01-26 06:20:55,659 Example #0\n",
            "2020-01-26 06:20:55,659 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:20:55,659 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:20:55,660 \tHypothesis: Mẹn da ha khian ọne isikulu , mẹn da dọ ha khian ọne isikulu .\n",
            "2020-01-26 06:20:55,660 Example #1\n",
            "2020-01-26 06:20:55,660 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:20:55,660 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:20:55,660 \tHypothesis: Ene ẹbho nan wo manman hoẹmhọn egbe ele , ele ki ha mhọn ọne isẹhoa nan rẹ ha ga iJehova .\n",
            "2020-01-26 06:20:55,660 Example #2\n",
            "2020-01-26 06:20:55,660 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:20:55,661 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:20:55,661 \tHypothesis: Bhi ọsi ẹmhọanta , mhan dẹ sabọ re izebhudu nin ene ga iJehova , sade mhan re obọ kpa bhi agbaẹbho kẹkẹ , la sade mhan re obọ rẹkhan ọlẹn bhi agbaẹbho ọbhebhe , la sade a re obọ kpa bhi ọne agbọn nan .\n",
            "2020-01-26 06:20:55,661 Example #3\n",
            "2020-01-26 06:20:55,661 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:20:55,661 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:20:55,661 \tHypothesis: Mhan dẹ sabọ rẹkpa ene biẹ mhan rẹ ha mhọn urẹọbhọ da iJehova .\n",
            "2020-01-26 06:20:55,661 Validation result (greedy) at epoch  41, step     1900: bleu:   4.38, loss: 73768.7266, ppl:  21.1673, duration: 46.9699s\n",
            "2020-01-26 06:21:00,460 Epoch  41: total training loss 108.00\n",
            "2020-01-26 06:21:00,460 EPOCH 42\n",
            "2020-01-26 06:21:07,846 Epoch  42: total training loss 106.55\n",
            "2020-01-26 06:21:07,846 EPOCH 43\n",
            "2020-01-26 06:21:11,280 Epoch  43 Step:     2000 Batch Loss:     2.588126 Tokens per Sec:    13353, Lr: 0.000300\n",
            "2020-01-26 06:21:57,059 Example #0\n",
            "2020-01-26 06:21:57,060 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:21:57,060 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:21:57,060 \tHypothesis: Mẹn da ha yi mẹn bhi agbotu .\n",
            "2020-01-26 06:21:57,060 Example #1\n",
            "2020-01-26 06:21:57,061 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:21:57,061 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:21:57,061 \tHypothesis: Ene ẹbho nan wo manman hoẹmhọn egbe ele .\n",
            "2020-01-26 06:21:57,061 Example #2\n",
            "2020-01-26 06:21:57,062 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:21:57,062 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:21:57,062 \tHypothesis: Mhan da ha re obọ rẹkhan uhi bi ene ribhi agbotu nọnsẹmhan , mhan ki ha mhọn isẹhoa nin ele rẹ ha zẹ ebi ele ha lu yẹ , bi emhin ne bunbun ne bunbun ne bunbun ne bunbun .\n",
            "2020-01-26 06:21:57,062 Example #3\n",
            "2020-01-26 06:21:57,062 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:21:57,063 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:21:57,063 \tHypothesis: Mhan dẹ sabọ rẹkpa ene biẹ mhan rẹ sabọ mun oga mhọn nọnsẹn .\n",
            "2020-01-26 06:21:57,063 Validation result (greedy) at epoch  43, step     2000: bleu:   4.61, loss: 73770.9844, ppl:  21.1693, duration: 45.7826s\n",
            "2020-01-26 06:22:00,972 Epoch  43: total training loss 106.94\n",
            "2020-01-26 06:22:00,972 EPOCH 44\n",
            "2020-01-26 06:22:08,112 Epoch  44: total training loss 104.02\n",
            "2020-01-26 06:22:08,112 EPOCH 45\n",
            "2020-01-26 06:22:12,281 Epoch  45 Step:     2100 Batch Loss:     2.218198 Tokens per Sec:    13917, Lr: 0.000300\n",
            "2020-01-26 06:22:58,089 Example #0\n",
            "2020-01-26 06:22:58,090 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:22:58,090 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:22:58,091 \tHypothesis: Mẹn da ha yi mẹn bhi agbotu .\n",
            "2020-01-26 06:22:58,091 Example #1\n",
            "2020-01-26 06:22:58,091 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:22:58,092 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:22:58,092 \tHypothesis: Ene ẹbho ne bunbun wo manman hoẹmhọn egbe ele .\n",
            "2020-01-26 06:22:58,092 Example #2\n",
            "2020-01-26 06:22:58,093 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:22:58,093 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:22:58,093 \tHypothesis: Bhi ọsi ẹmhọanta , mhan ki sabọ ha mhọn isẹhoa nin ele rẹ ha mhọn bhi ọne agbọn nan , bi emhin ne bunbun , bi emhin ne bunbun ne bunbun ne bunbun .\n",
            "2020-01-26 06:22:58,093 Example #3\n",
            "2020-01-26 06:22:58,094 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:22:58,094 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:22:58,094 \tHypothesis: Mhan dẹ sabọ rẹkpa ene eria nan rẹ sabọ rẹkpa ele .\n",
            "2020-01-26 06:22:58,094 Validation result (greedy) at epoch  45, step     2100: bleu:   4.43, loss: 73995.4219, ppl:  21.3668, duration: 45.8125s\n",
            "2020-01-26 06:23:01,107 Epoch  45: total training loss 101.78\n",
            "2020-01-26 06:23:01,107 EPOCH 46\n",
            "2020-01-26 06:23:08,284 Epoch  46: total training loss 100.12\n",
            "2020-01-26 06:23:08,285 EPOCH 47\n",
            "2020-01-26 06:23:13,242 Epoch  47 Step:     2200 Batch Loss:     2.125690 Tokens per Sec:    13449, Lr: 0.000300\n",
            "2020-01-26 06:23:59,097 Example #0\n",
            "2020-01-26 06:23:59,098 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:23:59,098 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:23:59,098 \tHypothesis: Mẹn da ha yi mẹn bhi isikulu .\n",
            "2020-01-26 06:23:59,098 Example #1\n",
            "2020-01-26 06:23:59,099 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:23:59,099 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:23:59,099 \tHypothesis: Ene ẹbho nan wo manman hoẹmhọn egbe ele .\n",
            "2020-01-26 06:23:59,099 Example #2\n",
            "2020-01-26 06:23:59,100 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:23:59,100 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:23:59,100 \tHypothesis: Bhi ọsi ẹmhọanta , mhan da ha mhọn isẹhoa ọsaje , bi emhin ne bunbun ne bunbun ne bunbun , la mhan ha mhọn ọne agbọn nan , bi emhin ne bunbun .\n",
            "2020-01-26 06:23:59,100 Example #3\n",
            "2020-01-26 06:23:59,100 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:23:59,100 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:23:59,101 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ sabọ ha mhọn urẹọbhọ bọsi eria .\n",
            "2020-01-26 06:23:59,101 Validation result (greedy) at epoch  47, step     2200: bleu:   4.77, loss: 74653.0703, ppl:  21.9562, duration: 45.8578s\n",
            "2020-01-26 06:24:01,261 Epoch  47: total training loss 98.90\n",
            "2020-01-26 06:24:01,262 EPOCH 48\n",
            "2020-01-26 06:24:08,514 Epoch  48: total training loss 97.66\n",
            "2020-01-26 06:24:08,514 EPOCH 49\n",
            "2020-01-26 06:24:14,456 Epoch  49 Step:     2300 Batch Loss:     2.433468 Tokens per Sec:    13251, Lr: 0.000300\n",
            "2020-01-26 06:25:00,375 Example #0\n",
            "2020-01-26 06:25:00,376 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:25:00,376 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:25:00,376 \tHypothesis: Mẹn da ha khian uwa ọsi mẹn .\n",
            "2020-01-26 06:25:00,376 Example #1\n",
            "2020-01-26 06:25:00,377 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:25:00,377 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:25:00,377 \tHypothesis: Ene ẹbho ne bunbun wo lẹn ghe , ele guanọ nin ele ha ga Osẹnobulua , ele ki ha re iẹnlẹn nọnsele man ẹbho .\n",
            "2020-01-26 06:25:00,377 Example #2\n",
            "2020-01-26 06:25:00,378 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:25:00,378 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:25:00,378 \tHypothesis: Bhi ọsi ẹmhọanta , mhan ki sabọ ha mhọn isẹhoa nin ele rẹ ha zẹ ebi a rẹ mun oga mhọn nọnsẹn yẹ , nin ẹbho ne bunbun da sabọ ha mhọn isẹhoa nin ele rẹ ha mhọn ọne ẹmhọn , bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu bi ozughu nan .\n",
            "2020-01-26 06:25:00,378 Example #3\n",
            "2020-01-26 06:25:00,378 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:25:00,379 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:25:00,379 \tHypothesis: Mhan dẹ sabọ rẹkpa mhan rẹ ha mhọn urẹọbhọ bọsi eria ne mhọn nọnsẹn .\n",
            "2020-01-26 06:25:00,379 Validation result (greedy) at epoch  49, step     2300: bleu:   5.01, loss: 74774.7734, ppl:  22.0671, duration: 45.9223s\n",
            "2020-01-26 06:25:01,675 Epoch  49: total training loss 98.46\n",
            "2020-01-26 06:25:01,675 EPOCH 50\n",
            "2020-01-26 06:25:08,910 Epoch  50: total training loss 94.65\n",
            "2020-01-26 06:25:08,910 EPOCH 51\n",
            "2020-01-26 06:25:15,597 Epoch  51 Step:     2400 Batch Loss:     1.754878 Tokens per Sec:    13267, Lr: 0.000300\n",
            "2020-01-26 06:26:01,261 Example #0\n",
            "2020-01-26 06:26:01,261 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:26:01,262 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:26:01,262 \tHypothesis: Mẹn da ha khian uwa ọsi mẹn .\n",
            "2020-01-26 06:26:01,262 Example #1\n",
            "2020-01-26 06:26:01,262 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:26:01,262 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:26:01,262 \tHypothesis: Ene ẹbho nan wo manman hoẹmhọn egbe ele .\n",
            "2020-01-26 06:26:01,262 Example #2\n",
            "2020-01-26 06:26:01,262 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:26:01,262 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:26:01,263 \tHypothesis: Bhi ọsi ẹmhọanta , mhan da ha mhọn isẹhoa ọsaje nin ọle rẹ ha zẹ ebi ọle ha lu , bi emhin ne bunbun yẹ , mhan ki yẹ ha noo ọne agbọn nan rebhe .\n",
            "2020-01-26 06:26:01,263 Example #3\n",
            "2020-01-26 06:26:01,263 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:26:01,263 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:26:01,263 \tHypothesis: Mhan dẹ sabọ rẹkpa ene ibhio mhan nan rẹ sabọ rẹkpa mhan rẹ ha mhọn emhanmhan .\n",
            "2020-01-26 06:26:01,263 Validation result (greedy) at epoch  51, step     2400: bleu:   5.39, loss: 75081.5938, ppl:  22.3490, duration: 45.6658s\n",
            "2020-01-26 06:26:01,846 Epoch  51: total training loss 95.07\n",
            "2020-01-26 06:26:01,846 EPOCH 52\n",
            "2020-01-26 06:26:08,992 Epoch  52: total training loss 91.61\n",
            "2020-01-26 06:26:08,993 EPOCH 53\n",
            "2020-01-26 06:26:16,198 Epoch  53: total training loss 90.01\n",
            "2020-01-26 06:26:16,199 EPOCH 54\n",
            "2020-01-26 06:26:16,496 Epoch  54 Step:     2500 Batch Loss:     2.151499 Tokens per Sec:    14123, Lr: 0.000300\n",
            "2020-01-26 06:27:02,181 Example #0\n",
            "2020-01-26 06:27:02,182 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:27:02,182 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:27:02,182 \tHypothesis: Mẹn da ha yi mẹn bhi isikulu , mẹn da dọ ha yo isikulu .\n",
            "2020-01-26 06:27:02,183 Example #1\n",
            "2020-01-26 06:27:02,183 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:27:02,183 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:27:02,183 \tHypothesis: Ene ẹbho nan wo manman hoẹmhọn egbe ele , ele ọkpa ele ọkpa ele re eghọnghọn lu emhin nin ele lu .\n",
            "2020-01-26 06:27:02,184 Example #2\n",
            "2020-01-26 06:27:02,184 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:27:02,184 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:27:02,184 \tHypothesis: Bhi ọsi ẹmhọanta , mhan sabọ rẹ ha mhọn isẹhoa nin ele rẹ zẹ ebi mhan ha lu , bi emhin ne bunbun nin ele ha da sabọ ha mhọn isẹhoa nin ele rẹ ha zẹ emhin bhi obọ .\n",
            "2020-01-26 06:27:02,184 Example #3\n",
            "2020-01-26 06:27:02,185 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:27:02,185 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:27:02,185 \tHypothesis: Mhan dẹ sabọ rẹkpa ene ibhio mhan nan rẹ sabọ rẹkpa mhan .\n",
            "2020-01-26 06:27:02,185 Validation result (greedy) at epoch  54, step     2500: bleu:   5.43, loss: 75066.7734, ppl:  22.3353, duration: 45.6891s\n",
            "2020-01-26 06:27:09,116 Epoch  54: total training loss 89.34\n",
            "2020-01-26 06:27:09,116 EPOCH 55\n",
            "2020-01-26 06:27:16,457 Epoch  55: total training loss 87.67\n",
            "2020-01-26 06:27:16,458 EPOCH 56\n",
            "2020-01-26 06:27:17,456 Epoch  56 Step:     2600 Batch Loss:     2.244434 Tokens per Sec:    13313, Lr: 0.000210\n",
            "2020-01-26 06:28:03,417 Example #0\n",
            "2020-01-26 06:28:03,418 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:28:03,419 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:28:03,419 \tHypothesis: Mẹn da ha yi mẹn bhi isikulu .\n",
            "2020-01-26 06:28:03,419 Example #1\n",
            "2020-01-26 06:28:03,420 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:28:03,420 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:28:03,420 \tHypothesis: Ene ẹbho nan wo manman hoẹmhọn egbe ele , ele ọkpa ele ọkpa ele mun kalo gbera emhin nin ele lu .\n",
            "2020-01-26 06:28:03,420 Example #2\n",
            "2020-01-26 06:28:03,421 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:28:03,421 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:28:03,421 \tHypothesis: Bhi ọsi ẹmhọanta , mhan da sabọ ha mhọn isẹhoa ọsaje nin ọria rẹ ha zẹ ebi ọne ẹmhọn nan rẹ lu emhin yẹ , bi emhin ne bunbun wo ti ọne agbọn nan .\n",
            "2020-01-26 06:28:03,421 Example #3\n",
            "2020-01-26 06:28:03,422 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:28:03,422 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:28:03,422 \tHypothesis: Emhin kpataki nọn nin mhan ha rẹ sabọ rẹkpa mhan rẹ sabọ ha mhọn okugbe .\n",
            "2020-01-26 06:28:03,423 Validation result (greedy) at epoch  56, step     2600: bleu:   5.76, loss: 75907.1953, ppl:  23.1257, duration: 45.9662s\n",
            "2020-01-26 06:28:09,646 Epoch  56: total training loss 85.80\n",
            "2020-01-26 06:28:09,647 EPOCH 57\n",
            "2020-01-26 06:28:16,958 Epoch  57: total training loss 83.85\n",
            "2020-01-26 06:28:16,958 EPOCH 58\n",
            "2020-01-26 06:28:18,835 Epoch  58 Step:     2700 Batch Loss:     1.853516 Tokens per Sec:    13053, Lr: 0.000210\n",
            "2020-01-26 06:29:04,798 Example #0\n",
            "2020-01-26 06:29:04,799 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:29:04,799 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:29:04,800 \tHypothesis: Mẹn da ha yi mẹn bhi agbaẹbho natiọle Multin mẹn yo isikulu .\n",
            "2020-01-26 06:29:04,800 Example #1\n",
            "2020-01-26 06:29:04,800 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:29:04,800 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:29:04,800 \tHypothesis: Ene ẹbho nan wo hoẹmhọn egbe ele , ele ọkpa ele ọkpa ele mun kalo gbera ẹbho .\n",
            "2020-01-26 06:29:04,801 Example #2\n",
            "2020-01-26 06:29:04,801 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:29:04,801 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:29:04,801 \tHypothesis: Bhi ọsi ẹmhọanta , mhan ki ha mhọn isẹhoa ọsaje , nin ele rẹ ha zẹ emhin bhi obọ , la ebe nesẹmhan , la ọria soso iribhọ nọn kokhun nẹ bhi ọne agbọn nan rebhe .\n",
            "2020-01-26 06:29:04,801 Example #3\n",
            "2020-01-26 06:29:04,802 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:29:04,802 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:29:04,802 \tHypothesis: Mhan dẹ sabọ rẹkpa ene ibhio mhan nan rẹ sabọ rẹkpa mhan rẹ ha mhọn ọfure .\n",
            "2020-01-26 06:29:04,802 Validation result (greedy) at epoch  58, step     2700: bleu:   5.95, loss: 75744.6484, ppl:  22.9707, duration: 45.9668s\n",
            "2020-01-26 06:29:10,306 Epoch  58: total training loss 83.99\n",
            "2020-01-26 06:29:10,307 EPOCH 59\n",
            "2020-01-26 06:29:17,548 Epoch  59: total training loss 82.05\n",
            "2020-01-26 06:29:17,548 EPOCH 60\n",
            "2020-01-26 06:29:20,148 Epoch  60 Step:     2800 Batch Loss:     1.447856 Tokens per Sec:    13417, Lr: 0.000210\n",
            "2020-01-26 06:30:06,063 Example #0\n",
            "2020-01-26 06:30:06,064 \tSource:     I WAS raised in Graz , Austria .\n",
            "2020-01-26 06:30:06,064 \tReference:  AGBAẸBHO natiọle Graz bhi Austria , ọle mẹn da wanre .\n",
            "2020-01-26 06:30:06,064 \tHypothesis: Mẹn da ha yi mẹn bhi ọne isikulu .\n",
            "2020-01-26 06:30:06,064 Example #1\n",
            "2020-01-26 06:30:06,065 \tSource:     In contrast with the people who show the widespread lack of love today , those who worship Jehovah have genuine love for their fellow man .\n",
            "2020-01-26 06:30:06,065 \tReference:  Ene ga iJehova ẹlẹnan wo mhọn oyẹẹ da ibo ele , ele bha diabe ene iga Jehova ne bha hoẹmhọn ibo ele .\n",
            "2020-01-26 06:30:06,065 \tHypothesis: Ene ẹbho nan wo hoẹmhọn egbe ele , ele ọkpa ele ọkpa ele mun kalo bhi iẹnlẹn .\n",
            "2020-01-26 06:30:06,065 Example #2\n",
            "2020-01-26 06:30:06,066 \tSource:     Today , we live as foreigners , as it were , in a world that has been morally and spiritually corrupted by Babylon the Great , the world empire of false religion , “ a dwelling place of demons . ”\n",
            "2020-01-26 06:30:06,066 \tReference:  Inian mhan diabi eriọbhe bhi ẹkẹ ọne agbọn nan nin oga ohoghe ria a . Babylon nọn khua a tie ene oga ohoghe nan , ele da dọ kiẹn eji ẹlinmhin ne imhẹn da nyẹnlẹn . ( Rev .\n",
            "2020-01-26 06:30:06,066 \tHypothesis: Bhi ọsi ẹmhọanta , mhan ki ha mhọn isẹhoa nin ele rẹ ha zẹ ebi ele ha lu , bi emhin ne bunbun wo ti ele bhọ rẹ ha mhọn ọne isẹhoa nan , bi ozughu bi ozughu bi ozughu nan mun obọ .\n",
            "2020-01-26 06:30:06,066 Example #3\n",
            "2020-01-26 06:30:06,067 \tSource:     We should also strive to help others spiritually .\n",
            "2020-01-26 06:30:06,067 \tReference:  Ahamiẹn mhan re ẹghe bhi otọ rẹ ha luẹ iBaibo , yẹ deba ene gene guanọ nin ele ga iJehova ha muobọ , ọ dẹ rẹkpa mhan rẹ ziẹn ikolu nin mhan bi Jehova koko mhọnlẹn .\n",
            "2020-01-26 06:30:06,067 \tHypothesis: Emhin kpataki nọn nin mhan rẹ rẹkpa ẹbho rẹ sabọ ha zọn egbe .\n",
            "2020-01-26 06:30:06,067 Validation result (greedy) at epoch  60, step     2800: bleu:   6.42, loss: 76520.3438, ppl:  23.7199, duration: 45.9189s\n",
            "2020-01-26 06:30:10,740 Epoch  60: total training loss 81.03\n",
            "2020-01-26 06:30:10,741 EPOCH 61\n",
            "2020-01-26 06:30:18,022 Epoch  61: total training loss 80.01\n",
            "2020-01-26 06:30:18,022 EPOCH 62\n",
            "2020-01-26 06:30:21,552 Epoch  62 Step:     2900 Batch Loss:     1.625466 Tokens per Sec:    13614, Lr: 0.000210\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "n94wlrCjVc17",
        "outputId": "e977d469-999d-4d30-9f59-72932ba14755",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 663
        }
      },
      "source": [
        "# Output our validation accuracy epoch 1-30\n",
        "! cat \"$gdrive_path/models/${src}${tgt}_transformer/validations.txt\""
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Steps: 100\tLoss: 116888.22656\tPPL: 126.05112\tbleu: 0.00000\tLR: 0.00030000\t*\n",
            "Steps: 200\tLoss: 112468.19531\tPPL: 104.98232\tbleu: 0.00000\tLR: 0.00030000\t*\n",
            "Steps: 300\tLoss: 101796.18750\tPPL: 67.50474\tbleu: 0.00000\tLR: 0.00030000\t*\n",
            "Steps: 400\tLoss: 95543.67188\tPPL: 52.11624\tbleu: 0.29204\tLR: 0.00030000\t*\n",
            "Steps: 500\tLoss: 90320.13281\tPPL: 41.98586\tbleu: 0.96838\tLR: 0.00030000\t*\n",
            "Steps: 600\tLoss: 86108.06250\tPPL: 35.27036\tbleu: 1.39655\tLR: 0.00030000\t*\n",
            "Steps: 700\tLoss: 83404.91406\tPPL: 31.53791\tbleu: 1.91092\tLR: 0.00030000\t*\n",
            "Steps: 800\tLoss: 81618.19531\tPPL: 29.29035\tbleu: 2.37779\tLR: 0.00030000\t*\n",
            "Steps: 900\tLoss: 79700.19531\tPPL: 27.05559\tbleu: 2.58315\tLR: 0.00030000\t*\n",
            "Steps: 1000\tLoss: 78288.65625\tPPL: 25.52059\tbleu: 2.64520\tLR: 0.00030000\t*\n",
            "Steps: 1100\tLoss: 77336.52344\tPPL: 24.53468\tbleu: 2.94157\tLR: 0.00030000\t*\n",
            "Steps: 1200\tLoss: 76556.47656\tPPL: 23.75541\tbleu: 3.20514\tLR: 0.00030000\t*\n",
            "Steps: 1300\tLoss: 75516.39062\tPPL: 22.75473\tbleu: 3.73430\tLR: 0.00030000\t*\n",
            "Steps: 1400\tLoss: 74975.60156\tPPL: 22.25119\tbleu: 3.50584\tLR: 0.00030000\t*\n",
            "Steps: 1500\tLoss: 74599.64062\tPPL: 21.90772\tbleu: 3.62337\tLR: 0.00030000\t*\n",
            "Steps: 1600\tLoss: 74023.30469\tPPL: 21.39144\tbleu: 3.94679\tLR: 0.00030000\t*\n",
            "Steps: 1700\tLoss: 74392.28125\tPPL: 21.72055\tbleu: 3.80602\tLR: 0.00030000\t\n",
            "Steps: 1800\tLoss: 73928.28906\tPPL: 21.30751\tbleu: 4.18474\tLR: 0.00030000\t*\n",
            "Steps: 1900\tLoss: 73768.72656\tPPL: 21.16728\tbleu: 4.38409\tLR: 0.00030000\t*\n",
            "Steps: 2000\tLoss: 73770.98438\tPPL: 21.16926\tbleu: 4.61327\tLR: 0.00030000\t\n",
            "Steps: 2100\tLoss: 73995.42188\tPPL: 21.36678\tbleu: 4.42531\tLR: 0.00030000\t\n",
            "Steps: 2200\tLoss: 74653.07031\tPPL: 21.95621\tbleu: 4.76705\tLR: 0.00030000\t\n",
            "Steps: 2300\tLoss: 74774.77344\tPPL: 22.06705\tbleu: 5.00587\tLR: 0.00030000\t\n",
            "Steps: 2400\tLoss: 75081.59375\tPPL: 22.34900\tbleu: 5.38832\tLR: 0.00030000\t\n",
            "Steps: 2500\tLoss: 75066.77344\tPPL: 22.33530\tbleu: 5.42747\tLR: 0.00021000\t\n",
            "Steps: 2600\tLoss: 75907.19531\tPPL: 23.12569\tbleu: 5.75667\tLR: 0.00021000\t\n",
            "Steps: 2700\tLoss: 75744.64844\tPPL: 22.97066\tbleu: 5.95162\tLR: 0.00021000\t\n",
            "Steps: 2800\tLoss: 76520.34375\tPPL: 23.71992\tbleu: 6.42372\tLR: 0.00021000\t\n",
            "Steps: 2900\tLoss: 76849.28125\tPPL: 24.04498\tbleu: 6.23793\tLR: 0.00021000\t\n",
            "Steps: 3000\tLoss: 77257.02344\tPPL: 24.45410\tbleu: 6.59527\tLR: 0.00021000\t\n",
            "Steps: 3100\tLoss: 77712.66406\tPPL: 24.91953\tbleu: 6.60808\tLR: 0.00014700\t\n",
            "Steps: 3200\tLoss: 77657.35156\tPPL: 24.86256\tbleu: 6.72748\tLR: 0.00014700\t\n",
            "Steps: 3300\tLoss: 78290.96875\tPPL: 25.52304\tbleu: 6.96522\tLR: 0.00014700\t\n",
            "Steps: 3400\tLoss: 78602.87500\tPPL: 25.85458\tbleu: 6.77731\tLR: 0.00014700\t\n",
            "Steps: 3500\tLoss: 79068.42969\tPPL: 26.35748\tbleu: 7.13205\tLR: 0.00014700\t\n",
            "Steps: 3600\tLoss: 79626.85156\tPPL: 26.97360\tbleu: 6.78547\tLR: 0.00014700\t\n",
            "Steps: 3700\tLoss: 79842.32031\tPPL: 27.21517\tbleu: 6.87354\tLR: 0.00010290\t\n",
            "Steps: 3800\tLoss: 80084.18750\tPPL: 27.48891\tbleu: 6.91555\tLR: 0.00010290\t\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "66WhRE9lIhoD",
        "outputId": "ab884d35-2e53-4a65-ce90-01f07c73d4c4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Test our model\n",
        "! cd joeynmt; python3 -m joeynmt test \"$gdrive_path/models/${src}${tgt}_transformer/config.yaml\""
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-01-26 07:41:48,000 Hello! This is Joey-NMT.\n",
            "2020-01-26 07:42:22,635  dev bleu:   4.94 [Beam search decoding with beam size = 5 and alpha = 1.0]\n",
            "2020-01-26 07:42:27,157 test bleu:   6.25 [Beam search decoding with beam size = 5 and alpha = 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}