venetis commited on
Commit
b9aa706
1 Parent(s): 6372653

End of training

Browse files
Files changed (2) hide show
  1. README.md +160 -0
  2. adapter_model.bin +3 -0
README.md ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama3
3
+ library_name: peft
4
+ tags:
5
+ - axolotl
6
+ - generated_from_trainer
7
+ base_model: meta-llama/Meta-Llama-3-8B
8
+ model-index:
9
+ - name: llama3-8b-hermes-sandals-sample-10k
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
17
+ <details><summary>See axolotl config</summary>
18
+
19
+ axolotl version: `0.4.0`
20
+ ```yaml
21
+ base_model: meta-llama/Meta-Llama-3-8B
22
+ model_type: LlamaForCausalLM
23
+ tokenizer_type: AutoTokenizer
24
+
25
+ load_in_8bit: false
26
+ load_in_4bit: false
27
+ strict: false
28
+
29
+ datasets:
30
+ - path: ./data/openhermes2_5_10k.jsonl
31
+ type: sharegpt
32
+ conversation: llama3
33
+ dataset_prepared_path:
34
+ val_set_size: 0.15
35
+ output_dir: ./outputs_lora-out
36
+ hub_model_id: venetis/llama3-8b-hermes-sandals-sample-10k
37
+
38
+ data_seed: 117
39
+ seed: 117
40
+
41
+ chat_template: llama3
42
+ adapter: lora
43
+ lora_model_dir:
44
+ lora_r: 32
45
+ lora_alpha: 16
46
+ lora_dropout: 0.05
47
+ lora_target_linear: true
48
+ lora_fan_in_fan_out:
49
+
50
+
51
+ sequence_len: 4096
52
+ sample_packing: true
53
+ eval_sample_packing: true
54
+ pad_to_sequence_len: true
55
+ wandb_project: llama-3-8b-hermes-sandals-sample-10k
56
+ wandb_entity: venetispall
57
+
58
+ gradient_accumulation_steps: 4
59
+ micro_batch_size: 2
60
+ num_epochs: 4
61
+ optimizer: adamw_bnb_8bit
62
+ lr_scheduler: cosine
63
+ learning_rate: 2e-4
64
+
65
+ train_on_inputs: false
66
+ group_by_length: false
67
+ bf16: auto
68
+ fp16:
69
+ tf32: false
70
+
71
+ gradient_checkpointing: true
72
+ early_stopping_patience:
73
+ resume_from_checkpoint:
74
+ local_rank:
75
+ logging_steps: 1
76
+ xformers_attention:
77
+ flash_attention: true
78
+ s2_attention:
79
+
80
+
81
+ warmup_steps: 10
82
+ evals_per_epoch: 4
83
+ eval_table_size:
84
+ eval_max_new_tokens: 128
85
+ saves_per_epoch: 1
86
+ debug:
87
+ deepspeed:
88
+ weight_decay: 0.0
89
+ fsdp:
90
+ fsdp_config:
91
+ special_tokens:
92
+ pad_token: <|end_of_text|>
93
+
94
+ ```
95
+
96
+ </details><br>
97
+
98
+ # llama3-8b-hermes-sandals-sample-10k
99
+
100
+ This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on the None dataset.
101
+ It achieves the following results on the evaluation set:
102
+ - Loss: 0.7611
103
+
104
+ ## Model description
105
+
106
+ More information needed
107
+
108
+ ## Intended uses & limitations
109
+
110
+ More information needed
111
+
112
+ ## Training and evaluation data
113
+
114
+ More information needed
115
+
116
+ ## Training procedure
117
+
118
+ ### Training hyperparameters
119
+
120
+ The following hyperparameters were used during training:
121
+ - learning_rate: 0.0002
122
+ - train_batch_size: 2
123
+ - eval_batch_size: 2
124
+ - seed: 117
125
+ - gradient_accumulation_steps: 4
126
+ - total_train_batch_size: 8
127
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
128
+ - lr_scheduler_type: cosine
129
+ - lr_scheduler_warmup_steps: 10
130
+ - num_epochs: 4
131
+
132
+ ### Training results
133
+
134
+ | Training Loss | Epoch | Step | Validation Loss |
135
+ |:-------------:|:------:|:----:|:---------------:|
136
+ | 0.918 | 0.0102 | 1 | 0.9709 |
137
+ | 0.7146 | 0.2538 | 25 | 0.7678 |
138
+ | 0.728 | 0.5076 | 50 | 0.7419 |
139
+ | 0.6843 | 0.7614 | 75 | 0.7328 |
140
+ | 0.6819 | 1.0152 | 100 | 0.7259 |
141
+ | 0.6342 | 1.2487 | 125 | 0.7269 |
142
+ | 0.616 | 1.5025 | 150 | 0.7298 |
143
+ | 0.7092 | 1.7563 | 175 | 0.7250 |
144
+ | 0.6453 | 2.0102 | 200 | 0.7224 |
145
+ | 0.5267 | 2.2411 | 225 | 0.7425 |
146
+ | 0.5702 | 2.4949 | 250 | 0.7424 |
147
+ | 0.5459 | 2.7487 | 275 | 0.7421 |
148
+ | 0.6327 | 3.0025 | 300 | 0.7428 |
149
+ | 0.5649 | 3.2335 | 325 | 0.7573 |
150
+ | 0.4318 | 3.4873 | 350 | 0.7617 |
151
+ | 0.5523 | 3.7411 | 375 | 0.7611 |
152
+
153
+
154
+ ### Framework versions
155
+
156
+ - PEFT 0.10.0
157
+ - Transformers 4.40.2
158
+ - Pytorch 2.1.2+cu118
159
+ - Datasets 2.19.1
160
+ - Tokenizers 0.19.1
adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c9a9bdcf948f6f268d7a14bd0697a518cc1abca7a3bf101fd30eb3657b1a4c8
3
+ size 167934026