config.json paramaters
On this model, it reads :
"max_position_embeddings": 2048,
"torch_dtype": "float16",
Shouldn't it be like on the original CodeLlama model and Airoboros c34b 2.1?
"max_position_embeddings": 16384,
"torch_dtype": "bfloat16",
Also, wouldn't "rope_theta": 1000000, be useful, as per the original CodeLlama model?
Thanks for publishing this, as well as for Samantha C34 1.11 (who is a real challenge for context obedient prompt and model schizophrenia between the Assistant and the Character Samantha), Eric!
I didn't set any of those values. I used CodeLlama as the base model
From where? Did you use a non-official source perhaps? The correct values are here: https://huggingface.co/codellama/CodeLlama-34b-hf/blob/main/config.json
It should have 16K and rope_theta 1M yeah. I'll change it for my quants and I put in a PR
Thank you Tom