ChocoLlama/ChocoLlama-2-7B-base

ChocoLlama

A Llama-2/3-based family of Dutch language models

Model Details

ChocoLlama is a family of open LLM's specifically adapted to Dutch, contributing to the state-of-the-art of Dutch open LLM's in their weight class.

We provide 6 variants (of which 3 base and 3 instruction-tuned models):

• ChocoLlama-2-7B-base: A language-adapted version of Meta's Llama-2-7b, fine-tuned on a Dutch dataset of 104GB (XXX tokens) using LoRa.
• ChocoLlama-2-7B-instruct: An instruction-tuned version of ChocoLlama-2-7B-base, fine-tuned on a collection of Dutch translations of instruction-tuning datasets, using SFT followed by DPO.
• ChocoLlama-2-7B-tokentrans-base: A language-adapted version of Meta's Llama-2-7b, using a Dutch RoBERTa-based tokenizer. The token embeddings of this model were reinitialized using the token translation algorithm proposed by Remy et al.. The model was subsequently fine-tuned on the same Dutch dataset as ChocoLlama-2-7B-base, again using LoRa.
• ChocoLlama-2-7B-tokentrans-instruct: An instruction-tuned version of ChocoLlama-2-7B-tokentrans-base, fine-tuned on the same dataset as ChocoLlama-2-7B-instruct, again using SFT followed by DPO.
• Llama-3-ChocoLlama-8B-base: A language-adapted version of Meta's Llama-8-8B, fine-tuned on the same Dutch dataset as ChocoLlama-2-7B-base, again using LoRa.
• Llama-3-ChocoLlama-instruct: An instruction-tuned version of Llama-3-ChocoLlama-8B-base, fine-tuned on the same dataset as ChocoLlama-2-7B-instruct, again using SFT followed by DPO.

As far as we are aware, Llama-3-ChocoLlama-8B-instruct sets a new state-of-the-art for Dutch open models in its weight class.

Model Description

• Developed by: Matthieu Meeus, Anthony Rathé
• Funded by: Vlaams Supercomputer Centrum, through a grant of apx. 40K GPU hours (NVIDIA H100-80GB)
• Language(s): Dutch
• License: Llama-2 Community License
• Finetuned from model: Llama-2-7b-hf

Model Sources

• Repository: Will be released soon.
• Paper: Will be released soon.

Uses

Direct Use

Since this is a base model, we do not recommend using it for your use-cases directly. We instead recommend:

1. Fine-tuning this model to your specific use-case
2. Leveraging the instruction-tuned version of this model

Downstream Use

Since this model is a base model, it can easily be adapted to specific use-cases that required Dutch language understanding and generation. We expect this model to be particularly useful for use-cases in the domains which were explicitly covered in our dataset, e.g. the analysis and/or generation of:

• Dutch job descriptions
• Dutch corporate filings
• Dutch legislation

Out-of-Scope Use

• Use-cases requiring a chat-style interface: since this is a base model, it cannot be used reliably for turn-based chat interaction. Please refer to the instruction-tuned version of this model instead.
• Use-cases requiring understanding or generation of text in languages other than Dutch: the dataset on which this model was fine-tuned does not contain data in languages other than Dutch, hence we expect significant catastrophic forgetting to have occured for English, which is the language Llama-2 was originally trained for.

Bias, Risks, and Limitations

We have taken care to include only widely used and high-quality data in our dataset. Some of this data has been filtered by the original creators. However we did not explicitly conduct any additional filtering of this dataset with regards to biased or otherwise harmful content.

Recommendations

We recommend fine-tuning this model to your curated data to maximally avoid undesirable outputs.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('ChocoLlama/ChocoLlama-2-7B-base')
model = AutoModelForCausalLM.from_pretrained('ChocoLlama/ChocoLlama-2-7B-base')

Training Details

Training Data

[More Information Needed]

Training Procedure

This model was fine-tuned using low-rank (LoRa) adapatation with trainable embeddings, for a total of 4% trainable parameters.

Training Hyperparameters

• Training regime: bf16 non-mixed precision
• Epochs: 1
• LoRa parameters:
  • R: 8
  • Alpha: 32
  • Trainable modules: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj, embed_tokens, lm_head
  • LoRa dropout: 0.05
• Learning Rate:
  • Scheduler: StepLR
  • Step size: 6212
  • Learning rate: 0.0003
  • Gamma: 0.85
• Other parameters:
  • Minibatch size: 16
  • Gradient accumulation steps: 8
  • Parallelization factor: 8
  • Weight decay: 0

Evaluation

Quantitative evaluation

We have evaluated our models on several industry-standard Dutch benchmarks, translated from their original versions. The results can be found in the table below, together with results from several other prominent Dutch models.

Model	ARC	HellaSwag	MMLU	TruthfulQA	Avg.
Llama-3-ChocoLlama-instruct	0.48	0.66	0.49	0.49	0.53
llama-3-8B-rebatch	0.44	0.64	0.46	0.48	0.51
llama-3-8B-instruct	0.47	0.59	0.47	0.52	0.51
llama-3-8B	0.44	0.64	0.47	0.45	0.5
Reynaerde-7B-Chat	0.44	0.62	0.39	0.52	0.49
Llama-3-ChocoLlama-base	0.45	0.64	0.44	0.44	0.49
zephyr-7b-beta	0.43	0.58	0.43	0.53	0.49
geitje-7b-ultra	0.40	0.66	0.36	0.49	0.48
ChocoLlama-2-7B-tokentrans-instruct	0.45	0.62	0.34	0.42	0.46
mistral-7b-v0.1	0.43	0.58	0.37	0.45	0.46
ChocoLlama-2-7B-tokentrans-base	0.42	0.61	0.32	0.43	0.45
ChocoLlama-2-7B-instruct	0.36	0.57	0.33	0.45	**0.43
ChocoLlama-2-7B-base	0.35	0.56	0.31	0.43	0.41
llama-2-7b-chat-hf	0.36	0.49	0.33	0.44	0.41
llama-2-7b-hf	0.36	0.51	0.32	0.41	0.40

On average, Llama-3-ChocoLlama-instruct surpasses the previous state-of-the-art on these benchmarks.

Qualitative evaluation

Compute Infrastructure

All ChocoLlama models have been trained on the compute cluster provided by the Flemish Supercomputer Center (VSC). We used 8 to 16 NVIDIA H100 GPU's with 80 GB of VRAM.