See axolotl config

axolotl version: 0.4.0

base_model: meta-llama/Meta-Llama-3-8B
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: ./data/openhermes2_5_10k.jsonl
    type: sharegpt
    conversation: chatml
dataset_prepared_path: 
val_set_size: 0.15
output_dir: ./lora-output-dir
hub_model_id: venetis/llama3-8b-hermes-sandals-sample-10k

data_seed: 117
seed: 117

chat_template: chatml
adapter: lora
lora_model_dir:
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:


sequence_len: 4096
sample_packing: true
eval_sample_packing: true
pad_to_sequence_len: true
wandb_project: llama-3-8b-hermes-sandals-sample-10k
wandb_entity: venetispall

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 2e-4

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:


warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
#UPDATES mk2 - added special tokens
special_tokens:
  eos_token: "<|im_end|>"
  pad_token: "<|end_of_text|>"
tokens:
  - "<|im_start|>"
  - "<|im_end|>"
lora_modules_to_save:
  - embed_tokens
  - lm_head

llama3-8b-hermes-sandals-sample-10k

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8913

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 117
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
0.9567	0.0102	1	1.0036
0.7583	0.2545	25	0.8184
0.8226	0.5089	50	0.8238
0.7471	0.7634	75	0.8094
0.7339	1.0178	100	0.7954
0.4737	1.2494	125	0.8393
0.4723	1.5038	150	0.8395
0.5529	1.7583	175	0.8327
0.4288	2.0127	200	0.8277
0.2476	2.2468	225	0.8617
0.2566	2.5013	250	0.8676
0.2787	2.7557	275	0.8654
0.3477	3.0102	300	0.8648
0.1912	3.2392	325	0.8909
0.1868	3.4936	350	0.8912
0.1864	3.7481	375	0.8913

Framework versions

• PEFT 0.10.0
• Transformers 4.40.2
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1