VictorSanh
commited on
Commit
•
40cd111
1
Parent(s):
1253cd9
Update README.md
Browse files
README.md
CHANGED
@@ -58,7 +58,7 @@ If you want to use another checkpoint, please replace the path in `AutoTokenizer
|
|
58 |
|
59 |
# Training procedure
|
60 |
|
61 |
-
T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on
|
62 |
|
63 |
At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.
|
64 |
|
|
|
58 |
|
59 |
# Training procedure
|
60 |
|
61 |
+
T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapated T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.
|
62 |
|
63 |
At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.
|
64 |
|