---
license: apache-2.0
datasets:
- HuggingFaceTB/smollm-corpus
language:
- en
pipeline_tag: text2text-generation
library_name: transformers
---


# tFINE-900m-e16-d32-1024ctx


Pretrained T5 model with [nanoT5](https://github.com/pszemraj/nanoT5/tree/fineweb-edu-test):

- ~900m parameters, 16 layers in encoder, 32 layers in decoder
- sentencepiece tokenizer with 48k vocab & byte-pair fallback
  - handles whitespaces etc correctly (_unlike original T5 tokenizer_)
- 1024 ctx during pretrain
- `relative_attention_num_buckets` increased to 48 from 32 for context length upscaling

## Experiment logs

Training consisted of two phases:

- [phase one](https://wandb.ai/pszemraj/nanoT5/runs/l0y9uuv3) - ~30k steps at context length 512
- [phase two](https://wandb.ai/pszemraj/nanoT5/runs/mao0tqjy) - 20k steps at context length 1024