performance

#8
by rautsanket4086 - opened

Why this model response takes too much time
i am using Params: {'model': 'models/mistral-7b-instruct-v0.1.Q8_0.gguf', 'model_type': 'mistral', 'model_file': None, 'config': None}
why this takes 10 to 12 mins for execution
How can i decrease time for execution

rautsanket4086 changed discussion status to closed
rautsanket4086 changed discussion status to open

Anyone know this

Use other model, like de Q4_K_M. This would improve a lot the speed

Sign up or log in to comment