InferenceIllusionist
/

Mistral-Large-Instruct-2407-iMat-GGUF

Inference Endpoints

Model card Files Files and versions Community

InferenceIllusionist commited on Aug 4

Commit

9712507

•

1 Parent(s): 4f57fea

Update README.md

Files changed (1) hide show

README.md +1 -0

README.md CHANGED Viewed

@@ -29,6 +29,7 @@ Quantized from Mistral-Large-Instruct-2407 123B fp16
 >* If you are getting a `cudaMalloc failed: out of memory` error, try passing an argument for lower context in llama.cpp, e.g. for 8k: `-c 8192`
 >* If you have all ampere generation or newer cards, you can use flash attention like so: `-fa`
 >* Provided Flash Attention is enabled you can also use quantized cache to save on VRAM e.g. for 8-bit: `-ctk q8_0 -ctv q8_0`
 Original model card can be found [here](https://huggingface.co/mistralai/Mistral-Large-Instruct-2407)

 >* If you are getting a `cudaMalloc failed: out of memory` error, try passing an argument for lower context in llama.cpp, e.g. for 8k: `-c 8192`
 >* If you have all ampere generation or newer cards, you can use flash attention like so: `-fa`
 >* Provided Flash Attention is enabled you can also use quantized cache to save on VRAM e.g. for 8-bit: `-ctk q8_0 -ctv q8_0`
+>* Files split with llama.cpp's gguf-split. No need to manually combine files - just download all files for a specific quant size and load the first file (labeled "00001-")
 Original model card can be found [here](https://huggingface.co/mistralai/Mistral-Large-Instruct-2407)