|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- nl |
|
tags: |
|
- gguf |
|
- llamacpp |
|
- dpo |
|
- geitje |
|
- conversational |
|
datasets: |
|
- BramVanroy/ultra_feedback_dutch |
|
--- |
|
|
|
<p align="center" style="margin:0;padding:0"> |
|
<img src="https://huggingface.co/BramVanroy/GEITje-7B-ultra/resolve/main/geitje-ultra-banner.png" alt="GEITje Ultra banner" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/> |
|
</p> |
|
|
|
<div style="margin:auto; text-align:center"> |
|
<h1 style="margin-bottom: 0">GEITje 7B ultra (GGUF version)</h1> |
|
<em>A conversational model for Dutch, aligned through AI feedback.</em> |
|
</div> |
|
|
|
This is a `Q5_K_M` GGUF version of [BramVanroy/GEITje-7B-ultra](https://huggingface.co/BramVanroy/GEITje-7B-ultra), a powerful Dutch chatbot, which ultimately is Mistral-based model, further pretrained on Dutch and additionally treated with supervised-finetuning and DPO alignment. For more information on the model, data, licensing, usage, see the main model's README. |
|
|
|
## Usage |
|
|
|
### LM Studio |
|
|
|
You can use this model in [LM Studio](https://lmstudio.ai/), an easy-to-use interface to locally run optimized models. Simply search for `BramVanroy/GEITje-7B-ultra-GGUF`, and download the available file. |
|
|
|
### Ollama |
|
|
|
The model is available on `ollama` and can be easily run as follows: |
|
|
|
```shell |
|
ollama run bramvanroy/geitje-7b-ultra-gguf |
|
``` |
|
|
|
To reproduce, i.e. to create the ollama files manually instead of downloading them via ollama, follow the next steps. |
|
|
|
First download the [GGUF file](https://huggingface.co/BramVanroy/GEITje-7B-ultra-GGUF/resolve/main/GEITje-7B-ultra-Q5_K_M.gguf?download=true) and [Modelfile](https://huggingface.co/BramVanroy/GEITje-7B-ultra-GGUF/resolve/main/Modelfile?download=true) to your computer. You can adapt the Modelfile as you wish. |
|
|
|
Then, create the ollama model and run it. |
|
|
|
```shelll |
|
ollama create geitje-7b-ultra-gguf -f ./Modelfile |
|
ollama run geitje-7b-ultra-gguf |
|
``` |
|
|
|
## Reproduce this GGUF version from the non-quantized model |
|
|
|
Assuming you have installed and build llama cpp, current working directory is the `build` directory in llamacpp. |
|
|
|
Download initial model (probaby a huggingface-cli alternative exists, too...) |
|
|
|
```python |
|
from huggingface_hub import snapshot_download |
|
model_id = "BramVanroy/GEITje-7B-ultra" |
|
snapshot_download(repo_id=model_id, local_dir="geitje-ultra-hf", local_dir_use_symlinks=False) |
|
``` |
|
|
|
Convert to GGML format |
|
|
|
```shell |
|
# Convert to GGML format |
|
python convert.py build/geitje-ultra-hf/ |
|
|
|
cd build |
|
|
|
# Quantize to Q5_K_M |
|
bin/quantize geitje-ultra-hf/ggml-model-f32.gguf geitje-ultra-hf/GEITje-7B-ultra-Q5_K_M.gguf Q5_K_M |
|
``` |
