is there the GGUF version of the model?

#11
by MatrixIA - opened

I tried converting it myself but colab keeps crashing due to memory use.

@MatrixIA - I've tried using this one:

https://huggingface.co/bartowski/llama-3-sqlcoder-8b-GGUF

However, I'm getting gibberish - not sure what's going on. All the llama2 models I've been working with work fine with llama.cpp (python), but I can't seem to get any llama3 models to work. I'm currently only using GGUF models.

I would love to know if you can get things working on your end!

I will try using this one : https://huggingface.co/bartowski/llama-3-sqlcoder-8b-GGUF
I have also converted it myself to the gguf version I just need to upload the files to the hub.

i will let u know about the results as soon as possible.

https://ollama.com/mannix/defog-llama3-sqlcoder-8b

These quantized model runs on my office laptop lol.

Sign up or log in to comment