LLAMA-GGML-v2
This is repo for LLaMA models quantised down to 4bit for the latest llama.cpp GGML v2 format.
THE FILES REQUIRES LATEST LLAMA.CPP (May 12th 2023 - commit b9fd7ee)!
llama.cpp recently made a breaking change to its quantisation methods.
I have quantised the GGML files in this repo with the latest version.
How to run in text-generation-webui
GGML models can be loaded into text-generation-webui by installing the llama.cpp module, then placing the ggml model file in a model folder as usual.
Further instructions here: text-generation-webui/docs/llama.cpp-models.md.
Note: at this time text-generation-webui may not support the new May 12th llama.cpp quantisation methods.
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no library tag.