# Llama 1B Tulu-3 Finetuned Model
## Model Description
A 1B parameter Llama model fully finetuned on the Tulu-3 dataset from AllenAI. This model builds upon Meta's Llama-3.2-1B architecture and incorporates instruction-following capabilities through the Tulu-3 training mixture.


### Base Model Name: 
meta-llama/Llama-3.2-1B



### Dataset: 
allenai/tulu-3-sft-mixture

### Hardware
4x NVIDIA A100 80GB GPUs


### Training Configuration
```yaml
  --model_name_or_path meta-llama/Llama-3.2-1B \
  --dataset_name "allenai/tulu-3-sft-mixture" \
  --learning_rate 1.0e-5 \
  --lr_scheduler_type linear \
  --warmup_ratio 0.03 \
  --weight_decay 0.0 \
  --num_train_epochs 2 \
  --per_device_train_batch_size 8 \
  --gradient_accumulation_steps 2 \
  --gradient_checkpointing \
  --logging_steps 25 \
  --bf16 \
  --eval_strategy steps \
  --eval_steps 5000
```