ibivibiv commited on
Commit
01fce3a
1 Parent(s): 9e3156f

End of training

Browse files
Files changed (1) hide show
  1. README.md +105 -0
README.md ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama3
3
+ library_name: peft
4
+ tags:
5
+ - axolotl
6
+ - generated_from_trainer
7
+ base_model: meta-llama/Meta-Llama-3-8B-Instruct
8
+ model-index:
9
+ - name: math-llama-3-8b-instruct
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
17
+ <details><summary>See axolotl config</summary>
18
+
19
+ axolotl version: `0.4.0`
20
+ ```yaml
21
+ adapter: qlora
22
+ base_model: meta-llama/Meta-Llama-3-8B-Instruct
23
+ base_model_config: meta-llama/Meta-Llama-3-8B-Instruct
24
+ datasets:
25
+ - path: vicgalle/alpaca-gpt4
26
+ type: alpaca
27
+ flash_attention: true
28
+ gradient_accumulation_steps: 4
29
+ gradient_checkpointing: true
30
+ hf_use_auth_token: true
31
+ hub_model_id: ibivibiv/math-llama-3-8b-instruct
32
+ learning_rate: 0.0002
33
+ load_in_4bit: true
34
+ logging_steps: 1
35
+ lora_alpha: 16
36
+ lora_dropout: 0.05
37
+ lora_r: 32
38
+ lora_target_linear: true
39
+ lr_scheduler: cosine
40
+ micro_batch_size: 2
41
+ model_type: AutoModelForCausalLM
42
+ num_epochs: 3
43
+ optimizer: paged_adamw_32bit
44
+ output_dir: /job/out
45
+ sample_packing: true
46
+ save_safetensors: true
47
+ sequence_len: 4096
48
+ special_tokens:
49
+ pad_token: <|end_of_text|>
50
+ tokenizer_type: AutoTokenizer
51
+ wandb_project: TuneStudio
52
+ wandb_run_id: mathllama
53
+ wandb_watch: 'true'
54
+ warmup_steps: 10
55
+
56
+ ```
57
+
58
+ </details><br>
59
+
60
+ # math-llama-3-8b-instruct
61
+
62
+ This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on the None dataset.
63
+
64
+ ## Model description
65
+
66
+ More information needed
67
+
68
+ ## Intended uses & limitations
69
+
70
+ More information needed
71
+
72
+ ## Training and evaluation data
73
+
74
+ More information needed
75
+
76
+ ## Training procedure
77
+
78
+ ### Training hyperparameters
79
+
80
+ The following hyperparameters were used during training:
81
+ - learning_rate: 0.0002
82
+ - train_batch_size: 2
83
+ - eval_batch_size: 2
84
+ - seed: 42
85
+ - distributed_type: multi-GPU
86
+ - num_devices: 2
87
+ - gradient_accumulation_steps: 4
88
+ - total_train_batch_size: 16
89
+ - total_eval_batch_size: 4
90
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
91
+ - lr_scheduler_type: cosine
92
+ - lr_scheduler_warmup_steps: 10
93
+ - num_epochs: 3
94
+
95
+ ### Training results
96
+
97
+
98
+
99
+ ### Framework versions
100
+
101
+ - PEFT 0.10.0
102
+ - Transformers 4.40.2
103
+ - Pytorch 2.1.2+cu118
104
+ - Datasets 2.19.1
105
+ - Tokenizers 0.19.1