catallama/CataLlama-v0.1-Base

Model Details

CataLlama-v0.1-Base is a fine-tune of meta-llama/Meta-Llama-3-8B on the catallama/Catalan-Raw-Text raw Catalan corpus dataset

The model shows improved proficiency with the Catalan language.

This is a base model and it is not fine-tuned for downstream tasks.

The model achieves a loss rate of 1.4608 on the validation dataset after one epoch.

Model developers Laurentiu Petrea based on Llama-3 from Meta.

Model Architecture CataLlama is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and direct preference optimisation (DPO) to align with human preferences for helpfulness and safety.

License The model uses the llama-3 license available at: https://llama.meta.com/llama3/license

Use with transformers

See the snippet below for usage with Transformers:

import transformers
import torch

model_id = "catallama/CataLlama-v0.1-Base"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

outputs = pipeline("Ei com estàs avui?")

print(outputs[0]["generated_text"][len(prompt):])

Training procedure

The model was trained without a prompt template, only with raw text separated by BOS and EOS tokens.

The model was trained for two epochs on 6x A100 80GB GPUs using DeepSpeed ZeRO State-3 without CPU offloading.

Example:

<|begin_of_text|>Després d'anys sense veure's, un grup d'amics es reuneix en una casa a la muntanya durant el cap de setmana. Sembla que res hagi canviat entre ells, però entre rialles i anècdotes s'amaga un tèrbol episodi del passat que els segueix turmentant. Tot d'una, un incident estrany altera els seus plans i queden completament aïllats i sense possibilitat de comunicació amb l'exterior. Els joves decideixen buscar ajuda, però en el camí el grup es va desintegrant, mentre el que sembla un nou ordre natural s'imposa davant dels seus ulls atònits. Obligat a fugir del seu Bangla Desh natal, el jove Fahim i el seu pare [..]<|end_of_text|>
<|begin_of_text|>Els contes que t'explicaven abans d'anar a dormir. Apropeu-vos a descobrir uns contes magnífics i viatgeu a un univers fascinant en què els somnis s'entrellacen amb la màgia i el misteri! En aquests moments s'ha exhaurit l'estoc d'aquest producte. Si vols ser el primer a conèixer quan en tornem a tenir, si us plau, escriu el teu correu al camp de baix.<|end_of_text|>

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• distributed_type: multi-GPU
• num_devices: 6
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.5037	0.07	100	1.5150
1.5181	0.15	200	1.5097
1.4901	0.22	300	1.5041
1.5042	0.3	400	1.4980
1.494	0.37	500	1.4928
1.4462	0.45	600	1.4875
1.475	0.52	700	1.4822
1.4811	0.6	800	1.4774
1.4511	0.67	900	1.4730
1.4712	0.75	1000	1.4691
1.4531	0.82	1100	1.4656
1.4761	0.89	1200	1.4626
1.4636	0.97	1300	1.4608

Intended Use

Note: This model is not intended to beat benchmarks, but to demonstrate techniques for augmenting LLMs on new languages and preserve rare languages as part of our world heritage.

Intended Use Cases Llama 3 is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.

Out-of-scope Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3 Community License. Use in languages other than English**.

**Note: Developers may fine-tune Llama 3 models for languages beyond English provided they comply with the Llama 3 Community License and the Acceptable Use Policy.