cmarkea/bloomz-3b-sft-chat · Maximum input length

Apr 3

Hello,

Thanks for publishing your models.
How can we determine the maximum input length of any model of your collections, esp. of Bloomz-Chat collection?

(At some point, I came across tokenizer's "model_max_length" but was surprised by the value: 1000000000000000019884624838656.)

Thanks

Cyrile

Credit Mutuel Arkea org Apr 3

Hello JeromeL-DT,

Thank you for your interest in our modeling. Indeed, you will find this size limit on all models that have a relative positional encoder (as in the case of Bloom: Alibi). This is to indicate that in theory, there is no size limit. However, that's theory; in practice, we have been limited by the type of data (often chat type data has a relatively short format) and the architecture (GPU with 40 GB of VRAM). We observed a loss of modeling performance beyond a context of 5k tokens.

JeromeL-DT changed discussion status to closed Apr 4