{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4cRE8IbIrIV"
      },
      "source": [
        "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and execute it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MOsHUjgdIrIW",
        "outputId": "3e8d972f-e441-43c1-d9ab-bf6cc28e86c6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers[sentencepiece]) (2024.11.6)\n",
            "Requirement already satisfied: tokenizers<0.22,>=0.21 in /usr/local/lib/python3.10/dist-packages (from transformers[sentencepiece]) (0.21.0)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers[sentencepiece]) (0.4.5)\n",
            "Requirement already satisfied: sentencepiece!=0.1.92,>=0.1.91 in /usr/local/lib/python3.10/dist-packages (from transformers[sentencepiece]) (0.2.0)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from transformers[sentencepiece]) (4.25.5)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.4.4)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.2)\n",
            "Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (24.3.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.5.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.1.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (0.2.1)\n",
            "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.18.3)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->datasets) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2024.12.14)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)\n"
          ]
        }
      ],
      "source": [
        "! pip install datasets transformers[sentencepiece]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFASsisvIrIb"
      },
      "source": [
        "If you're opening this notebook locally, make sure your environment has an install from the last version of Datasets and a source install of Transformers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mq6gZGf9zlB5"
      },
      "outputs": [],
      "source": [
        "from transformers.utils import send_example_telemetry\n",
        "\n",
        "send_example_telemetry(\"tokenizer_training_notebook\", framework=\"none\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whPRbBNbIrIl"
      },
      "source": [
        "## Loading the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d9L_UKQjzlB6"
      },
      "source": [
        "# Training your own tokenizer from scratch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FOPtHPrZzlB6"
      },
      "source": [
        "In this notebook, we will see several ways to train your own tokenizer from scratch on a given corpus, so you can then use it to train a language model from scratch.\n",
        "\n",
        "Why would you need to *train* a tokenizer? That's because Transformer models very often use subword tokenization algorithms, and they need to be trained to identify the parts of words that are often present in the corpus you are using. We recommend you take a look at the [tokenization chapter](https://huggingface.co/course/chapter2/4?fw=pt) of the Hugging Face course for a general introduction on tokenizers, and at the [tokenizers summary](https://huggingface.co/transformers/tokenizer_summary.html) for a look at the differences between the subword tokenization algorithms."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "czSxaIhFzlB7"
      },
      "source": [
        "## Getting a corpus"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D1U5Be81zlB7"
      },
      "source": [
        "We will need texts to train our tokenizer. We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download our text data, which can be easily done with the `load_dataset` function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "ayr4l2ThzlB8"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zk3BQcA_zlB9"
      },
      "source": [
        "For this example, we will use Wikitext-2 (which contains 4.5MB of texts so training goes fast for our example) but you can use any dataset you want (and in any language, just not English)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "6MFsO1cJzlB9",
        "outputId": "1cc90fec-44d8-4b74-adc4-6e6785e65f44",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81,
          "referenced_widgets": [
            "0bf443c9813a487ebc412c907cd36576",
            "11d68bff925d40e3aeb8c6b3bdc30a93",
            "ab98f87f8fc2441bb4e577391662bf13",
            "7ecd0925e1f0464b8d8bbce8b67acc05",
            "6443115ea23a42a8b8e97a8924c57d31",
            "5c03f1965d91467194e6f31f51c58b34",
            "28e888c0de3344a09f08e79379dc8247",
            "0e3b7e667e1241a481c348cbd10acb62",
            "b85fd95ef1f345cba25801d140128cd9",
            "a525142c054a4b50833ebcc96eb16957",
            "17732775a1a54057b53b01e98577b188",
            "c348c4d541964607a076ddeb28adff13",
            "c41b71281e604f609c38d45e672ee31f",
            "7a3bffa557064ffba3ba872711f471c3",
            "c3664f48956e47709ed0a26f064c4718",
            "a7fa406d984c4069908a43e603ea221b",
            "76342cf3c6b74f4da205f3a162dd53b0",
            "90d6f5365ddc4a86954818b6b951c51d",
            "35fb71a2cb7a491b934f8de77e4f1000",
            "97ae67e8c51a4944808076222d562045",
            "db0105053d364f72920bdfdff0881c13",
            "40a5d4f2057b4a3984fcee9b96e134d4"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Resolving data files:   0%|          | 0/17 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0bf443c9813a487ebc412c907cd36576"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading dataset shards:   0%|          | 0/17 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c348c4d541964607a076ddeb28adff13"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "dataset = load_dataset(\"wikimedia/wikipedia\", name=\"20231101.fr\", split=\"train\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y5DipK_azlB-"
      },
      "source": [
        "We can have a look at the dataset, which as 36,718 texts:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "wG_PEMgUzlB-",
        "outputId": "2212cfce-319c-45d9-e6cc-f97a829b1e3b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['text'],\n",
              "    num_rows: 2564646\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tO0vfhz5zlB-"
      },
      "source": [
        "To access an element, we just have to provide its index:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "cUgwNBgBzlB-",
        "outputId": "7ec12a0a-5e44-40ae-a67e-0b8245096ccc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 146
        }
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'dataset' is not defined",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-1ec12ad25fa7>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'dataset' is not defined"
          ]
        }
      ],
      "source": [
        "dataset[1]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset=dataset.remove_columns([\"title\",\"url\",\"id\"])"
      ],
      "metadata": {
        "id": "WObI4Mz07FEU"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hOHYPA-GzlB-"
      },
      "source": [
        "We can also access a slice directly, in which case we get a dictionary with the key `\"text\"` and a list of texts as value:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "87CZ4fx5zlB_",
        "outputId": "086e9cc2-852d-4078-c0c3-7d9a5ca82cdd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 146
        }
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "name 'dataset' is not defined",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-28b8b8fb4071>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdataset\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'dataset' is not defined"
          ]
        }
      ],
      "source": [
        "dataset[:1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VzouGlkkzlB_"
      },
      "source": [
        "The API to train our tokenizer will require an iterator of batch of texts, for instance a list of list of texts:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "OZj2DYztzlB_"
      },
      "outputs": [],
      "source": [
        "batch_size = 10000\n",
        "# all_texts = [dataset[i : i + batch_size][\"text\"] for i in range(0, len(dataset), batch_size)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ymvwVLM3zlB_"
      },
      "source": [
        "To avoid loading everything into memory (since the Datasets library keeps the element on disk and only load them in memory when requested), we define a Python iterator. This is particularly useful if you have a huge dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "lHYatY4MzlB_"
      },
      "outputs": [],
      "source": [
        "def batch_iterator():\n",
        "    for i in range(0, len(dataset), batch_size):\n",
        "        yield dataset[i : i + batch_size][\"text\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHzb68rfzlB_"
      },
      "source": [
        "Now let's see how we can use this corpus to train a new tokenizer! There are two APIs to do this: the first one uses an existing tokenizer and will train a new version of it on your corpus in one line of code, the second is to actually build your tokenizer block by block, so lets you customize every step!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HDiDoF7FzlCA"
      },
      "source": [
        "## Using an existing tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x2cKgKlzzlCA"
      },
      "source": [
        "If you want to train a tokenizer with the exact same algorithms and parameters as an existing one, you can just use the `train_new_from_iterator` API. For instance, let's train a new version of the GPT-2 tokenzier on Wikitext-2 using the same tokenization algorithm.\n",
        "\n",
        "First we need to load the tokenizer we want to use as a model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "6nRvVJp4zlCA",
        "outputId": "ac890070-d94a-47bf-cd16-477814ae277f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 183,
          "referenced_widgets": [
            "72c0b4be3c65414fa849caab05bfc9f9",
            "21c1a8cfe1084e46b5fb90c626992923",
            "96fa41e9125848399763a9db272ee698",
            "0a95f31bb0a2476fa7f60a72606c9d21",
            "7b17cf1792b8479783452da0b3e89e65",
            "6c84842f9ae0443096109fc193f07d11",
            "26480a27342a460395c0f8ea9c3ec71f",
            "773733e7909c49f1828253f1d29e0fdb",
            "0ef285af8d0a486384d3f58764181a82",
            "e7a173cb3fa242fbabdf1148c2bd4d0f",
            "8193c9700d7a474da3a0b3921171ef76",
            "71e81051f60940668f584ec148c20031",
            "baa0203ac370435884eee023706735e6",
            "3d3202a940c4490aa45d4fd4c5ba0c01",
            "192ffa5491fb4f10910560ba663d72d1",
            "5877f2b7ac664493bdd6145fbf989e38",
            "292ce492607d425db89d30569348ddd0",
            "20b810d770944032aded94562df65c7b",
            "aedd4ffb42c04e848620529e4d003cf7",
            "e0f8a8ce5b814cd88df7980eba53c8ad",
            "54cb39abec8d41aa841e59d323090b35",
            "8aae96010b764b79b37f74c37b0293d7",
            "bb46e20e956c4f60b2380c3389a56767",
            "716b18627be34341af35c7c82b7cf25e",
            "0f18ee1f3dee451fa76ded6462ec0918",
            "6468534df4234cf3a01888a373fe26b4",
            "6d02cf7636da4289b18850681798c932",
            "e0d574cc69e84578b19a48d8e27c5129",
            "1c30ec867025464eb4b1d2dfedbb01a4",
            "8d5a60c36d154497aac99d4af953d84c",
            "4a75202ac15b4e6693f89fe1f3e1169e",
            "cd5c392eb11d4b798bbf255e0fbbe374",
            "780947ca386c49f88f7e4816bfe2ecf6",
            "2845640b0e94460399ffaba34defcef6",
            "0043b028448a42f6afcf17fd2cec799e",
            "fe96aee8b357490f9c622d4dd3dddc3e",
            "db3a2c6435eb41ea91e7d151b510f4ad",
            "a6d08a7fc9c24e21a516843a25de9943",
            "28af8d2fb34a4f95880b44d34c9afc26",
            "1b2376cef95c4398b7fb65f7dd93bf94",
            "1afad3b2d2654ebbbbea22466671f182",
            "20fc1bfc68cc4304b586dd064c34104b",
            "bc63063fd8fb405d97395f2a8f260c9e",
            "912e50236f5646178a6465ed58f27f8b"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "0it [00:00, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "72c0b4be3c65414fa849caab05bfc9f9"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/20.8k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "71e81051f60940668f584ec148c20031"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/2.13M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bb46e20e956c4f60b2380c3389a56767"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/694 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2845640b0e94460399ffaba34defcef6"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"answerdotai/ModernBERT-base\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "roZ0ejDYzlCA"
      },
      "source": [
        "Make sure that the tokenizer you picked as a *fast* version (backed by the 🤗 Tokenizers library) otherwise the rest of the notebook will not run:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "text=\"\"\"\n",
        "À votre niveau d'utilisation actuel, cette exécution peut durer jusqu'à 38 heures 20 minutes.\n",
        "\"\"\""
      ],
      "metadata": {
        "id": "fSAx2QMbI_XB"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "v-yE5edqzlCA",
        "outputId": "75a80a9e-66f3-40b1-d9e5-2017db645e43",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Ċ', 'Ã', 'Ģ', 'Ġvotre', 'Ġn', 'iveau', 'Ġd', \"'\", 'util', 'isation', 'Ġact', 'uel', ',', 'Ġcette', 'Ġex', 'éc', 'ution', 'Ġpeut', 'Ġd', 'urer', 'Ġjus', 'qu', \"'\", 'Ãł', 'Ġ38', 'Ġhe', 'ures', 'Ġ20', 'Ġminutes', '.', 'Ċ']\n",
            "50280\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer.tokenize(text))\n",
        "print(tokenizer.vocab_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WT2RJsDHzlCA"
      },
      "source": [
        "Then we feed the training corpus (either the list of list or the iterator we defined earlier) to the `train_new_from_iterator` method. We also have to specify the vocabulary size we want to use:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8a6JXYf8zlCA"
      },
      "outputs": [],
      "source": [
        "new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=50280)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X3fMJE4_zlCB"
      },
      "source": [
        "And that's all there is to it! The training goes very fast thanks to the 🤗 Tokenizers library, backed by Rust.\n",
        "\n",
        "You now have a new tokenizer ready to preprocess your data and train a language model. You can feed it input texts as usual:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zsUZmM_xzlCB",
        "outputId": "3a586e8c-6f45-48ba-f8c0-be4e88a566bd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [[], [238, 8576, 9441, 2987, 238, 252], [], [4657, 74, 4762, 826, 8576, 428, 466, 609, 6881, 412, 204, 9441, 311, 2746, 466, 10816, 168, 99, 150, 192, 112, 14328, 3983, 112, 4446, 94, 18288, 4446, 193, 3983, 98, 3983, 22171, 95, 19, 201, 6374, 209, 8576, 218, 198, 3455, 1972, 428, 310, 201, 5099, 3242, 227, 281, 8576, 9441, 2987, 2553, 1759, 201, 301, 196, 13996, 1496, 277, 2330, 1464, 674, 1898, 307, 742, 3541, 225, 7514, 14, 54, 719, 274, 198, 4777, 15522, 209, 19895, 221, 1341, 1633, 221, 1759, 201, 322, 301, 198, 1368, 674, 221, 198, 8576, 843, 209, 2468, 1795, 223, 198, 1049, 9595, 218, 13996, 225, 1563, 277, 582, 6493, 281, 457, 14371, 201, 198, 1422, 3373, 7452, 227, 198, 455, 674, 225, 4687, 198, 239, 21976, 239, 201, 196, 21657, 1680, 3773, 5591, 198, 4196, 218, 4679, 427, 661, 198, 3518, 1288, 220, 1051, 516, 889, 3947, 1922, 2500, 225, 390, 2065, 744, 872, 198, 7592, 3773, 239, 1975, 251, 208, 89, 22351, 239, 209, 252], [261, 674, 959, 1921, 221, 1462, 201, 7600, 547, 196, 1178, 4753, 218, 198, 630, 3591, 263, 8576, 9441, 1180, 209, 1831, 322, 7568, 198, 3621, 2240, 218, 198, 843, 201, 322, 471, 9575, 5291, 16591, 967, 201, 781, 281, 1815, 198, 674, 604, 10344, 1252, 274, 843, 664, 3147, 320, 209, 13290, 8751, 8124, 2528, 6023, 74, 235, 225, 7445, 10040, 17384, 241, 11487, 8950, 857, 1835, 340, 1382, 22582, 201, 1008, 296, 8576, 9441, 1180, 2436, 21134, 5337, 19463, 5161, 209, 240, 1178, 927, 218, 3776, 8650, 198, 3355, 209, 261, 674, 268, 83, 2511, 3472, 258, 8288, 307, 1010, 268, 78, 209, 252]], 'attention_mask': [[], [1, 1, 1, 1, 1, 1], [], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "new_tokenizer(dataset[:5][\"text\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j5XQ-3jwzlCB"
      },
      "source": [
        "You can save it locally with the `save_pretrained` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U81f-9mSzlCB",
        "outputId": "5a16b212-bd39-420d-eb89-3decb7c5f491"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "('my-new-tokenizer/tokenizer_config.json',\n",
              " 'my-new-tokenizer/special_tokens_map.json',\n",
              " 'my-new-tokenizer/vocab.json',\n",
              " 'my-new-tokenizer/merges.txt',\n",
              " 'my-new-tokenizer/added_tokens.json',\n",
              " 'my-new-tokenizer/tokenizer.json')"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "new_tokenizer.save_pretrained(\"my-new-tokenizer\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DrH4hf9kzlCC"
      },
      "source": [
        "Or even push it to the [Hugging Face Hub](https://huggingface.co/models) to use that new tokenzier from anywhere. Just make sure you have your authentication token stored by executing `huggingface-cli login` in a terminal or executing the following cell:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JqH5-qxpzlCC"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import notebook_login\n",
        "\n",
        "notebook_login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LpGF84DAzlCC"
      },
      "source": [
        "We are almost there, it is also necessary that you have `git lfs` installed. You can do it directly from this notebook by uncommenting the following cells:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uU34E4xvzlCD"
      },
      "outputs": [],
      "source": [
        "# !apt install git-lfs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WjUlxe3wzlCD",
        "outputId": "1931e220-5d97-49e3-e451-94515c64904b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'https://huggingface.co/sgugger/my-new-shiny-tokenizer/commit/4714349893fd1aba4ed1e84910a13d631d07395a'"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "new_tokenizer.push_to_hub(\"my-new-shiny-tokenizer\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sNQtw_sAzlCJ"
      },
      "source": [
        "The tokenizer can now be reloaded on this machine with:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lKJ1Mw2SzlCK"
      },
      "outputs": [],
      "source": [
        "tok = new_tokenizer.from_pretrained(\"my-new-tokenizer\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0_Vd7KPzlCK"
      },
      "source": [
        "Or from anywhere using the repo ID, which is your namespace followed by a slash an the name you gave in the `push_to_hub` method, so for instance:\n",
        "\n",
        "```python\n",
        "tok = new_tokenizer.from_pretrained(\"sgugger/my-new-shiny-tokenizer\")\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "beLoP7p9zlCK"
      },
      "source": [
        "Now if you want to create and a train a new tokenizer that doesn't look like anything in existence, you will need to build it from scratch using the 🤗 Tokenizers library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dP13dY2czlCK"
      },
      "source": [
        "## Building your tokenizer from scratch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QbTcJUu6zlCK"
      },
      "source": [
        "To understand how to build your tokenizer from scratch, we have to dive a little bit more in the 🤗 Tokenizers library and the tokenization pipeline. This pipeline takes several steps:\n",
        "\n",
        "- **Normalization**: Executes all the initial transformations over the initial input string. For example when you need to lowercase some text, maybe strip it, or even apply one of the common unicode normalization process, you will add a Normalizer.\n",
        "- **Pre-tokenization**: In charge of splitting the initial input string. That's the component that decides where and how to pre-segment the origin string. The simplest example would be to simply split on spaces.\n",
        "- **Model**: Handles all the sub-token discovery and generation, this is the part that is trainable and really dependent of your input data.\n",
        "- **Post-Processing**: Provides advanced construction features to be compatible with some of the Transformers-based SoTA models. For instance, for BERT it would wrap the tokenized sentence around [CLS] and [SEP] tokens.\n",
        "\n",
        "And to go in the other direction:\n",
        "\n",
        "- **Decoding**: In charge of mapping back a tokenized input to the original string. The decoder is usually chosen according to the `PreTokenizer` we used previously.\n",
        "\n",
        "For the training of the model, the 🤗 Tokenizers library provides a `Trainer` class that we will use.\n",
        "\n",
        "All of these building blocks can be combined to create working tokenization pipelines. To give you some examples, we will show three full pipelines here: how to replicate GPT-2, BERT and T5 (which will give you an example of BPE, WordPiece and Unigram tokenizer)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mf64EcfyzlCK"
      },
      "source": [
        "### WordPiece model like BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oZkD53q3zlCK"
      },
      "source": [
        "Let's have a look at how we can create a WordPiece tokenizer like the one used for training BERT. The first step is to create a `Tokenizer` with an empty `WordPiece` model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eCFxuFFfzlCL"
      },
      "outputs": [],
      "source": [
        "from tokenizers import decoders, models, normalizers, pre_tokenizers, processors, trainers, Tokenizer\n",
        "\n",
        "tokenizer = Tokenizer(models.WordPiece(unk_token=\"[UNK]\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LC1nAGJuzlCL"
      },
      "source": [
        "This `tokenizer` is not ready for training yet. We have to add some preprocessing steps: the normalization (which is optional) and the pre-tokenizer, which will split inputs into the chunks we will call words. The tokens will then be part of those words (but can't be larger than that).\n",
        "\n",
        "In the case of BERT, the normalization is lowercasing. Since BERT is such a popular model, it has its own normalizer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zRKVjWOrzlCL"
      },
      "outputs": [],
      "source": [
        "tokenizer.normalizer = normalizers.BertNormalizer(lowercase=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UIQt6IGOzlCL"
      },
      "source": [
        "If you want to customize it, you can use the existing blocks and compose them in a sequence: here for instance we lower case, apply NFD normalization and strip the accents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ReQQOSoQzlCL"
      },
      "outputs": [],
      "source": [
        "tokenizer.normalizer = normalizers.Sequence(\n",
        "    [normalizers.NFD(), normalizers.Lowercase(), normalizers.StripAccents()]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1So-B6VlzlCL"
      },
      "source": [
        "There is also a `BertPreTokenizer` we can use directly. It pre-tokenizes using white space and punctuation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JbwEbKRgzlCM"
      },
      "outputs": [],
      "source": [
        "tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6z2P7IwMzlCM"
      },
      "source": [
        "Like for the normalizer, we can combine several pre-tokenizers in a `Sequence`. If we want to have a quick look at how it preprocesses the inputs, we can call the `pre_tokenize_str` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M5eXOHDHzlCM",
        "outputId": "ee1f6a33-e520-4b31-a74c-bb2f67656e93"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('This', (0, 4)),\n",
              " ('is', (5, 7)),\n",
              " ('an', (8, 10)),\n",
              " ('example', (11, 18)),\n",
              " ('!', (18, 19))]"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer.pre_tokenizer.pre_tokenize_str(\"This is an example!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1AFPdglQzlCM"
      },
      "source": [
        "Note that the pre-tokenizer not only split the text into words but keeps the offsets, that is the beginning and start of each of those words inside the original text. This is what will allow the final tokenizer to be able to match each token to the part of the text that it comes from (a feature we use for question answering or token classification tasks).\n",
        "\n",
        "We can now train our tokenizer (the pipeline is not entirely finished but we will need a trained tokenizer to build the post-processor), we use a `WordPieceTrainer` for that. The key thing to remember is to pass along the special tokens to the trainer, as they won't be seen in the corpus."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2vay_KRkzlCN"
      },
      "outputs": [],
      "source": [
        "special_tokens = [\"[UNK]\", \"[PAD]\", \"[CLS]\", \"[SEP]\", \"[MASK]\"]\n",
        "trainer = trainers.WordPieceTrainer(vocab_size=25000, special_tokens=special_tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kMwVfsdazlCN"
      },
      "source": [
        "To actually train the tokenizer, the method looks like what we used before: we can either pass some text files, or an iterator of batches of texts:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OisTCDadzlCN"
      },
      "outputs": [],
      "source": [
        "tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OGFqWhDBzlCN"
      },
      "source": [
        "Now that the tokenizer is trained, we can define the post-processor: we need to add the CLS token at the beginning and the SEP token at the end (for single sentences) or several SEP tokens (for pairs of sentences). We use a [`TemplateProcessing`](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html#tokenizers.processors.TemplateProcessing) to do this, which requires to know the IDs of the CLS and SEP token (which is why we waited for the training).\n",
        "\n",
        "So let's first grab the ids of the two special tokens:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h_aOgfnIzlCN",
        "outputId": "58a4db51-5ea0-408a-ce97-13b529944aab"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2 3\n"
          ]
        }
      ],
      "source": [
        "cls_token_id = tokenizer.token_to_id(\"[CLS]\")\n",
        "sep_token_id = tokenizer.token_to_id(\"[SEP]\")\n",
        "print(cls_token_id, sep_token_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFB6GxrKzlCO"
      },
      "source": [
        "And here is how we can build our post processor. We have to indicate in the template how to organize the special tokens with one sentence (`$A`) or two sentences (`$A` and `$B`). The `:` followed by a number indicates the token type ID to give to each part."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o5VFJ8W5zlCO"
      },
      "outputs": [],
      "source": [
        "tokenizer.post_processor = processors.TemplateProcessing(\n",
        "    single=f\"[CLS]:0 $A:0 [SEP]:0\",\n",
        "    pair=f\"[CLS]:0 $A:0 [SEP]:0 $B:1 [SEP]:1\",\n",
        "    special_tokens=[\n",
        "        (\"[CLS]\", cls_token_id),\n",
        "        (\"[SEP]\", sep_token_id),\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1Tc8tbIzlCO"
      },
      "source": [
        "We can check we get the expected results by encoding a pair of sentences for instance:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HWeSN-vKzlCP"
      },
      "outputs": [],
      "source": [
        "encoding = tokenizer.encode(\"This is one sentence.\", \"With this one we have a pair.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s3dSNbBczlCP"
      },
      "source": [
        "We can look at the tokens to check the special tokens have been inserted in the right places:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mkUSK_CDzlCP",
        "outputId": "6f5cae67-7d54-4745-eda9-873aa378020d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['[CLS]',\n",
              " 'this',\n",
              " 'is',\n",
              " 'one',\n",
              " 'sentence',\n",
              " '.',\n",
              " '[SEP]',\n",
              " 'with',\n",
              " 'this',\n",
              " 'one',\n",
              " 'we',\n",
              " 'have',\n",
              " 'a',\n",
              " 'pair',\n",
              " '.',\n",
              " '[SEP]']"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "encoding.tokens"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PXFCLh4OzlCP"
      },
      "source": [
        "And we can check the token type ids are correct:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XDP9b7AEzlCP",
        "outputId": "7678d588-68c1-43e7-a3d2-2d8711b1e0bd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "encoding.type_ids"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bDles1ymzlCQ"
      },
      "source": [
        "The last piece in this tokenizer is the decoder, we use a `WordPiece` decoder and indicate the special prefix `##`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NDUoSvZPzlCQ"
      },
      "outputs": [],
      "source": [
        "tokenizer.decoder = decoders.WordPiece(prefix=\"##\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bSHxHEfIzlCQ"
      },
      "source": [
        "Now that our tokenizer is finished, we need to wrap it inside a Transformers object to be able to use it with the Transformers library. More specifically, we have to put it inside the class of tokenizer fast corresponding to the model we want to use, here a `BertTokenizerFast`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JdQKmoWazlCQ"
      },
      "outputs": [],
      "source": [
        "from transformers import BertTokenizerFast\n",
        "\n",
        "new_tokenizer = BertTokenizerFast(tokenizer_object=tokenizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KEUb_y8KzlCQ"
      },
      "source": [
        "And like before, we can use this tokenizer as a normal Transformers tokenizer, and use the `save_pretrained` or `push_to_hub` methods.\n",
        "\n",
        "If the tokenizer you are building does not match any class in Transformers because it's really special, you can wrap it in `PreTrainedTokenizerFast`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pt7kywzPzlCQ"
      },
      "source": [
        "### BPE model like GPT-2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mryoQaJPzlCQ"
      },
      "source": [
        "Let's now have a look at how we can create a BPE tokenizer like the one used for training GPT-2. The first step is to create a `Tokenizer` with an empty `BPE` model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XslZwJYRzlCR"
      },
      "outputs": [],
      "source": [
        "tokenizer = Tokenizer(models.BPE())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hM54TP4izlCR"
      },
      "source": [
        "Like before, we have to add the optional normalization (not used in the case of GPT-2) and we need to specify a pre-tokenizer before training. In the case of GPT-2, the pre-tokenizer used is a byte level pre-tokenizer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9CBjTXKYzlCR"
      },
      "outputs": [],
      "source": [
        "tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hHYh4rC8zlCR"
      },
      "source": [
        "If we want to have a quick look at how it preprocesses the inputs, we can call the `pre_tokenize_str` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ueQ90Kw1zlCR",
        "outputId": "486d37db-3057-4af8-ec8a-83f5db6d49cf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('This', (0, 4)),\n",
              " ('Ġis', (4, 7)),\n",
              " ('Ġan', (7, 10)),\n",
              " ('Ġexample', (10, 18)),\n",
              " ('!', (18, 19))]"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer.pre_tokenizer.pre_tokenize_str(\"This is an example!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7WYJiQl-zlCS"
      },
      "source": [
        "We used the same default as for GPT-2 for the prefix space, so you can see that each word gets an initial `'Ġ'` added at the beginning, except the first one.\n",
        "\n",
        "We can now train our tokenizer! This time we use a `BpeTrainer`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iMOC3FIFzlCS"
      },
      "outputs": [],
      "source": [
        "trainer = trainers.BpeTrainer(vocab_size=25000, special_tokens=[\"<|endoftext|>\"])\n",
        "tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4P6XJQ62zlCV"
      },
      "source": [
        "To finish the whole pipeline, we have to include the post-processor and decoder:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OtWnj-fhzlCV"
      },
      "outputs": [],
      "source": [
        "tokenizer.post_processor = processors.ByteLevel(trim_offsets=False)\n",
        "tokenizer.decoder = decoders.ByteLevel()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5WBYGaZzlCW"
      },
      "source": [
        "And like before, we finish by wrapping this in a Transformers tokenizer object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n_2pHt2GzlCW"
      },
      "outputs": [],
      "source": [
        "from transformers import GPT2TokenizerFast\n",
        "\n",
        "new_tokenizer = GPT2TokenizerFast(tokenizer_object=tokenizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwMMjQDkzlCW"
      },
      "source": [
        "### Unigram model like Albert"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KtwmAouHzlCW"
      },
      "source": [
        "Let's now have a look at how we can create a Unigram tokenizer like the one used for training T5. The first step is to create a `Tokenizer` with an empty `Unigram` model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_aLA3BUSzlCW"
      },
      "outputs": [],
      "source": [
        "tokenizer = Tokenizer(models.Unigram())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cOfa2k_zzlCX"
      },
      "source": [
        "Like before, we have to add the optional normalization (here some replaces and lower-casing) and we need to specify a pre-tokenizer before training. The pre-tokenizer used is a `Metaspace` pre-tokenizer: it replaces all spaces by a special character (defaulting to ▁) and then splits on that character."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VdaKiT7uzlCX"
      },
      "outputs": [],
      "source": [
        "tokenizer.normalizer = normalizers.Sequence(\n",
        "    [normalizers.Replace(\"``\", '\"'), normalizers.Replace(\"''\", '\"'), normalizers.Lowercase()]\n",
        ")\n",
        "tokenizer.pre_tokenizer = pre_tokenizers.Metaspace()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lQwvcOVhzlCX"
      },
      "source": [
        "If we want to have a quick look at how it preprocesses the inputs, we can call the `pre_tokenize_str` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_Dw5Gad_zlCX",
        "outputId": "b37e600f-c07b-4811-cc43-c1e212c95d0d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[('▁This', (0, 4)), ('▁is', (4, 7)), ('▁an', (7, 10)), ('▁example!', (10, 19))]"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer.pre_tokenizer.pre_tokenize_str(\"This is an example!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dw69NzmFzlCX"
      },
      "source": [
        "You can see that each word gets an initial `▁` added at the beginning, as is usually done by sentencepiece.\n",
        "\n",
        "We can now train our tokenizer! This time we use a `UnigramTrainer`.\"We have to explicitely set the unknown token in this trainer otherwise it will forget it afterward."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8oFpQofhzlCX"
      },
      "outputs": [],
      "source": [
        "trainer = trainers.UnigramTrainer(vocab_size=25000, special_tokens=[\"[CLS]\", \"[SEP]\", \"<unk>\", \"<pad>\", \"[MASK]\"], unk_token=\"<unk>\")\n",
        "tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rynY7037zlCX"
      },
      "source": [
        "To finish the whole pipeline, we have to include the post-processor and decoder. The post-processor is very similar to what we saw with BERT, the decoder is just `Metaspace`, like for the pre-tokenizer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2ZzEKstFzlCY"
      },
      "outputs": [],
      "source": [
        "cls_token_id = tokenizer.token_to_id(\"[CLS]\")\n",
        "sep_token_id = tokenizer.token_to_id(\"[SEP]\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YM4RQMyWzlCY"
      },
      "outputs": [],
      "source": [
        "tokenizer.post_processor = processors.TemplateProcessing(\n",
        "    single=\"[CLS]:0 $A:0 [SEP]:0\",\n",
        "    pair=\"[CLS]:0 $A:0 [SEP]:0 $B:1 [SEP]:1\",\n",
        "    special_tokens=[\n",
        "        (\"[CLS]\", cls_token_id),\n",
        "        (\"[SEP]\", sep_token_id),\n",
        "    ],\n",
        ")\n",
        "tokenizer.decoder = decoders.Metaspace()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gGmfF_UnzlCY"
      },
      "source": [
        "And like before, we finish by wrapping this in a Transformers tokenizer object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x4ZXGtg2zlCY"
      },
      "outputs": [],
      "source": [
        "from transformers import AlbertTokenizerFast\n",
        "\n",
        "new_tokenizer = AlbertTokenizerFast(tokenizer_object=tokenizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4l8mWX03zlCY"
      },
      "source": [
        "## Use your new tokenizer to train a language model!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X5I-yLezzlCY"
      },
      "source": [
        "You can either use your new tokenizer in the language modeling from scratch notebook [Link to come] or use the `--tokenizer_name` argument in the [language modeling scripts](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling) to use it there to train a model from scratch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R66sgzfQzlCY"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "name": "Train your tokenizer",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.8"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0bf443c9813a487ebc412c907cd36576": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_11d68bff925d40e3aeb8c6b3bdc30a93",
              "IPY_MODEL_ab98f87f8fc2441bb4e577391662bf13",
              "IPY_MODEL_7ecd0925e1f0464b8d8bbce8b67acc05"
            ],
            "layout": "IPY_MODEL_6443115ea23a42a8b8e97a8924c57d31"
          }
        },
        "11d68bff925d40e3aeb8c6b3bdc30a93": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5c03f1965d91467194e6f31f51c58b34",
            "placeholder": "​",
            "style": "IPY_MODEL_28e888c0de3344a09f08e79379dc8247",
            "value": "Resolving data files: 100%"
          }
        },
        "ab98f87f8fc2441bb4e577391662bf13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0e3b7e667e1241a481c348cbd10acb62",
            "max": 17,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b85fd95ef1f345cba25801d140128cd9",
            "value": 17
          }
        },
        "7ecd0925e1f0464b8d8bbce8b67acc05": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a525142c054a4b50833ebcc96eb16957",
            "placeholder": "​",
            "style": "IPY_MODEL_17732775a1a54057b53b01e98577b188",
            "value": " 17/17 [00:00&lt;00:00, 90.00it/s]"
          }
        },
        "6443115ea23a42a8b8e97a8924c57d31": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5c03f1965d91467194e6f31f51c58b34": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "28e888c0de3344a09f08e79379dc8247": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0e3b7e667e1241a481c348cbd10acb62": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b85fd95ef1f345cba25801d140128cd9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a525142c054a4b50833ebcc96eb16957": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "17732775a1a54057b53b01e98577b188": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c348c4d541964607a076ddeb28adff13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c41b71281e604f609c38d45e672ee31f",
              "IPY_MODEL_7a3bffa557064ffba3ba872711f471c3",
              "IPY_MODEL_c3664f48956e47709ed0a26f064c4718"
            ],
            "layout": "IPY_MODEL_a7fa406d984c4069908a43e603ea221b"
          }
        },
        "c41b71281e604f609c38d45e672ee31f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_76342cf3c6b74f4da205f3a162dd53b0",
            "placeholder": "​",
            "style": "IPY_MODEL_90d6f5365ddc4a86954818b6b951c51d",
            "value": "Loading dataset shards: 100%"
          }
        },
        "7a3bffa557064ffba3ba872711f471c3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_35fb71a2cb7a491b934f8de77e4f1000",
            "max": 17,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_97ae67e8c51a4944808076222d562045",
            "value": 17
          }
        },
        "c3664f48956e47709ed0a26f064c4718": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_db0105053d364f72920bdfdff0881c13",
            "placeholder": "​",
            "style": "IPY_MODEL_40a5d4f2057b4a3984fcee9b96e134d4",
            "value": " 17/17 [00:20&lt;00:00,  1.10it/s]"
          }
        },
        "a7fa406d984c4069908a43e603ea221b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "76342cf3c6b74f4da205f3a162dd53b0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90d6f5365ddc4a86954818b6b951c51d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "35fb71a2cb7a491b934f8de77e4f1000": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "97ae67e8c51a4944808076222d562045": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "db0105053d364f72920bdfdff0881c13": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "40a5d4f2057b4a3984fcee9b96e134d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "72c0b4be3c65414fa849caab05bfc9f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_21c1a8cfe1084e46b5fb90c626992923",
              "IPY_MODEL_96fa41e9125848399763a9db272ee698",
              "IPY_MODEL_0a95f31bb0a2476fa7f60a72606c9d21"
            ],
            "layout": "IPY_MODEL_7b17cf1792b8479783452da0b3e89e65"
          }
        },
        "21c1a8cfe1084e46b5fb90c626992923": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6c84842f9ae0443096109fc193f07d11",
            "placeholder": "​",
            "style": "IPY_MODEL_26480a27342a460395c0f8ea9c3ec71f",
            "value": ""
          }
        },
        "96fa41e9125848399763a9db272ee698": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_773733e7909c49f1828253f1d29e0fdb",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0ef285af8d0a486384d3f58764181a82",
            "value": 0
          }
        },
        "0a95f31bb0a2476fa7f60a72606c9d21": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e7a173cb3fa242fbabdf1148c2bd4d0f",
            "placeholder": "​",
            "style": "IPY_MODEL_8193c9700d7a474da3a0b3921171ef76",
            "value": " 0/0 [00:00&lt;?, ?it/s]"
          }
        },
        "7b17cf1792b8479783452da0b3e89e65": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6c84842f9ae0443096109fc193f07d11": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "26480a27342a460395c0f8ea9c3ec71f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "773733e7909c49f1828253f1d29e0fdb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "0ef285af8d0a486384d3f58764181a82": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e7a173cb3fa242fbabdf1148c2bd4d0f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8193c9700d7a474da3a0b3921171ef76": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "71e81051f60940668f584ec148c20031": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_baa0203ac370435884eee023706735e6",
              "IPY_MODEL_3d3202a940c4490aa45d4fd4c5ba0c01",
              "IPY_MODEL_192ffa5491fb4f10910560ba663d72d1"
            ],
            "layout": "IPY_MODEL_5877f2b7ac664493bdd6145fbf989e38"
          }
        },
        "baa0203ac370435884eee023706735e6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_292ce492607d425db89d30569348ddd0",
            "placeholder": "​",
            "style": "IPY_MODEL_20b810d770944032aded94562df65c7b",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "3d3202a940c4490aa45d4fd4c5ba0c01": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aedd4ffb42c04e848620529e4d003cf7",
            "max": 20837,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e0f8a8ce5b814cd88df7980eba53c8ad",
            "value": 20837
          }
        },
        "192ffa5491fb4f10910560ba663d72d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_54cb39abec8d41aa841e59d323090b35",
            "placeholder": "​",
            "style": "IPY_MODEL_8aae96010b764b79b37f74c37b0293d7",
            "value": " 20.8k/20.8k [00:00&lt;00:00, 624kB/s]"
          }
        },
        "5877f2b7ac664493bdd6145fbf989e38": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "292ce492607d425db89d30569348ddd0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20b810d770944032aded94562df65c7b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "aedd4ffb42c04e848620529e4d003cf7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0f8a8ce5b814cd88df7980eba53c8ad": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "54cb39abec8d41aa841e59d323090b35": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8aae96010b764b79b37f74c37b0293d7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bb46e20e956c4f60b2380c3389a56767": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_716b18627be34341af35c7c82b7cf25e",
              "IPY_MODEL_0f18ee1f3dee451fa76ded6462ec0918",
              "IPY_MODEL_6468534df4234cf3a01888a373fe26b4"
            ],
            "layout": "IPY_MODEL_6d02cf7636da4289b18850681798c932"
          }
        },
        "716b18627be34341af35c7c82b7cf25e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e0d574cc69e84578b19a48d8e27c5129",
            "placeholder": "​",
            "style": "IPY_MODEL_1c30ec867025464eb4b1d2dfedbb01a4",
            "value": "tokenizer.json: 100%"
          }
        },
        "0f18ee1f3dee451fa76ded6462ec0918": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8d5a60c36d154497aac99d4af953d84c",
            "max": 2132967,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4a75202ac15b4e6693f89fe1f3e1169e",
            "value": 2132967
          }
        },
        "6468534df4234cf3a01888a373fe26b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cd5c392eb11d4b798bbf255e0fbbe374",
            "placeholder": "​",
            "style": "IPY_MODEL_780947ca386c49f88f7e4816bfe2ecf6",
            "value": " 2.13M/2.13M [00:00&lt;00:00, 20.1MB/s]"
          }
        },
        "6d02cf7636da4289b18850681798c932": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e0d574cc69e84578b19a48d8e27c5129": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1c30ec867025464eb4b1d2dfedbb01a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8d5a60c36d154497aac99d4af953d84c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4a75202ac15b4e6693f89fe1f3e1169e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cd5c392eb11d4b798bbf255e0fbbe374": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "780947ca386c49f88f7e4816bfe2ecf6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2845640b0e94460399ffaba34defcef6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0043b028448a42f6afcf17fd2cec799e",
              "IPY_MODEL_fe96aee8b357490f9c622d4dd3dddc3e",
              "IPY_MODEL_db3a2c6435eb41ea91e7d151b510f4ad"
            ],
            "layout": "IPY_MODEL_a6d08a7fc9c24e21a516843a25de9943"
          }
        },
        "0043b028448a42f6afcf17fd2cec799e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_28af8d2fb34a4f95880b44d34c9afc26",
            "placeholder": "​",
            "style": "IPY_MODEL_1b2376cef95c4398b7fb65f7dd93bf94",
            "value": "special_tokens_map.json: 100%"
          }
        },
        "fe96aee8b357490f9c622d4dd3dddc3e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1afad3b2d2654ebbbbea22466671f182",
            "max": 694,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_20fc1bfc68cc4304b586dd064c34104b",
            "value": 694
          }
        },
        "db3a2c6435eb41ea91e7d151b510f4ad": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bc63063fd8fb405d97395f2a8f260c9e",
            "placeholder": "​",
            "style": "IPY_MODEL_912e50236f5646178a6465ed58f27f8b",
            "value": " 694/694 [00:00&lt;00:00, 34.1kB/s]"
          }
        },
        "a6d08a7fc9c24e21a516843a25de9943": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "28af8d2fb34a4f95880b44d34c9afc26": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b2376cef95c4398b7fb65f7dd93bf94": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1afad3b2d2654ebbbbea22466671f182": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "20fc1bfc68cc4304b586dd064c34104b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bc63063fd8fb405d97395f2a8f260c9e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "912e50236f5646178a6465ed58f27f8b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}