bigscience

/

bloom

like 4.86k

Follow

BigScience Workshop 962

BLOOM was trained with sequences of length 2048, but uses ALiBi position embeddings (https://arxiv.org/abs/2108.12409) -- meaning it can be used with longer sequences.

So ALIBI allows input of what length to be considered? You write that it allows for “longer sequences” but this way I only know it is more than 2048…

Any update on this? @cakiki

I'd refer you to both the paper I linked to and empirical experimentation to answer that question. The practical limit will likely sooner be your hardware than any theoretical one.