What -ctx and -chunks parameters did you use to make the iMatrix of the Lllama 2 70b?
#9
by
Nexesenex
- opened
You should share your parameters to make your iMatrix, man, because you obviously know best. ^^
According to this thread I don't really know. Tons of other people who know better :-)
But if you want to know what I have done, I have simply created the importance matrix using
./imatrix -m <some_model> -f tests/wiki.train.raw -o some_model.imatrix --chunks 100
and then used it to quantize the model
./quantize --imatrix some_model.imatrix <some_model> iq2xxs.gguf iq2_xxs