4096?
When using it in role play, after i pass the 4k context window it begins to spit out nonsense. Too bad as i was looking for a model that can keep the character details for a bit longer than the usual 4k
hi, are you using GGUF, AWQ, or GPTQ?
Q5_K_M.GGUF. I have been looking for a good large context model for a while and thought this one was the one to claim the prize, but when i hit 4k it began to make no sense anymore and spit out gibberish. i had set it to 8k for testing.
hmm. i'm not sure. i use the same quant and it works for me for as far out as i've tested, which is past 7000 context. sorry, i wish i knew how to help more.
i had my friend test it on his machine and it's working for him as well. GGUF on apple silicon
i'm going to download the Q8 and try again and report back.
Loaded the Q2 for faster testing here is what it looks like when i go past the 4k size. I have no clue what causes this tbh as other models like mistral or mixtral ones do work fine beyond 4k.