---
tags:
- text-generation-inference
- transformers
- sft
- chocolatine
license: mit
language:
- fr
datasets:
- jpacifico/sft-set-administratif-def-6996
---

# Description model

Chocolatine-3B version specialized in French administrative language, supervised fine-tuning of [microsoft/Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)  
based on the official [lexicon](https://www.modernisation.gouv.fr/outils-et-formations/lexique-administratif)   
published by the French Ministère de la Fonction Publique et de la Réforme de l'Etat. 

# Data & Training

The [dataset](https://huggingface.co/datasets/jpacifico/sft-set-administratif-def-6996) gathers 2362 administrative terms constituting the basis of the simulation of prompt-answer pairs.  
The GPT-4o model deployed on Azure OpenAI was used to carry out the building of the dataset in several phases:  

- Extraction of the lexicon pages converted into jpg images  
- Reformulation of the definitions to make them more readable and natural to be used by an LLM in order to ensure high quality data.  
- Generation of questions from the terms and definitions  
- Generation of answers in three successive rounds taking into account the previous generations to ensure variety.  

Fine tuning (SFT) done efficiently with Unsloth,   
with which I saved processing time on a single T4 GPU (Compute instance from Azure ML).  

# Usage

The recommended usage is by loading the low-rank adapter using unsloth:

```python
from unsloth import FastLanguageModel

model_name = "jpacifico/chocolatine-admin-3B-sft-v0.2"
model, tokenizer = FastLanguageModel.from_pretrained(
  model_name = model_name,
  max_seq_length = 2048,
  dtype = None,
  load_in_4bit = True,
)

FastLanguageModel.for_inference(model)
```

### Limitations

The Chocolatine model is a quick demonstration that a base model can be easily fine-tuned to achieve compelling performance.  
It does not have any moderation mechanism.  

- **Developed by:** Jonathan Pacifico, 2024
- **License:** MIT
- **Finetuned from model :** microsoft/Phi-3.5-mini-instruct