Create README.md

README.md

@@ -0,0 +1,34 @@
+# LLaMA 1B Tulu-3 Finetuned Model
+## Model Description
+A 1B parameter LLaMA model fully finetuned on the Tulu-3 dataset from AllenAI. This model builds upon Meta's LLaMA-3.2-1B architecture and incorporates instruction-following capabilities through the Tulu-3 training mixture.
+
+
+### Base Model Name: 
+meta-llama/Llama-3.2-1B
+
+
+
+### Dataset: 
+allenai/tulu-3-sft-mixture
+
+### Hardware
+4x NVIDIA A100 80GB GPUs
+
+
+### Training Configuration
+```yaml
+  --model_name_or_path meta-llama/Llama-3.2-1B \
+  --dataset_name "allenai/tulu-3-sft-mixture" \
+  --learning_rate 1.0e-5 \
+  --lr_scheduler_type linear \
+  --warmup_ratio 0.03 \
+  --weight_decay 0.0 \
+  --num_train_epochs 2 \
+  --per_device_train_batch_size 8 \
+  --gradient_accumulation_steps 2 \
+  --gradient_checkpointing \
+  --logging_steps 25 \
+  --bf16 \
+  --eval_strategy steps \
+  --eval_steps 5000
+```