32k or 8k context?
#1
by
mclassHF2023
- opened
This shows up in oobabooga as 8192 context and also seems to generate gibberish when set to higher context. I can't test the original model, but is this meant to be genuine 32k context?
From what I can see, the mnodel has a base context size of 8192, and uses rope scaling to get to 32k. Anything more you should ask on the original model - this is just quants of it. See the model card for a link.
Just tested with f16 - seems pretty coherent at 32k context size. Make sure your tool is uptodate w.r.t. llama3.
mradermacher
changed discussion status to
closed