LLM just keep repating the same words
#5
by
leong99
- opened
Hi the Bloke, simillar things also happen to me in jyputer notebook, I am new to LLM can you teach me how to resolve the problem. really thanful!
howdy -- likely you are not setting the rope scaling (to 8 in this case). See, the 16k context is from the fine tuning of this model by lmsys. The pretrained LLama2 was 2k, so you need to set the rope scaling to 8 (16/2).
That's correct, you need to set rope scaling. Llama2 was trained to 4K, so you need a rope scaling of 4 for 16K. That's represented as 0.25, eg --rope-freq-scale 0.25