See axolotl config
axolotl version: 0.4.0
base_model: KolaGang/v3_pretrain
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false
hub_model_id: KolaGang/KingKan_SFT
hub_strategy: end
datasets:
- path: KolaGang/Reflection
type: reflection
- path: KolaGang/RAG_EAI
type: context_qa.load_v2
- path: KolaGang/QA
type: alpaca_chat.load_qa
- path: KolaGang/chatlaw
type: sharegpt
- path: KolaGang/draft
type: alpaca
- path: KolaGang/alpca_w_system
type: alpaca
- path: teknium/dataforge-economics
type: sharegpt
- path: QuietImpostor/Claude-3-Opus-Claude-3.5-Sonnnet-9k
type: sharegpt
- path: Open-Orca/slimorca-deduped-cleaned-corrected
type: sharegpt
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/train
sequence_len: 8196
sample_packing: true
pad_to_sequence_len: true
eval_sample_packing: false
wandb_project: mistral_v3
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
flash_attn_cross_entropy: false
flash_attn_rms_norm: true
flash_attn_fuse_qkv: false
flash_attn_fuse_mlp: true
warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 1
debug:
deepspeed: deepspeed_configs/zero2.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
save_safetensors: True
KingKan_SFT
This model is a fine-tuned version of KolaGang/v3_pretrain on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.8390
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 32
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
1.2753 | 0.0031 | 1 | 1.2464 |
0.8782 | 0.25 | 81 | 0.8252 |
0.8702 | 0.5 | 162 | 0.8030 |
0.8288 | 0.75 | 243 | 0.7896 |
0.8822 | 1.0 | 324 | 0.7797 |
0.6535 | 1.2315 | 405 | 0.7952 |
0.6072 | 1.4815 | 486 | 0.7947 |
0.6683 | 1.7315 | 567 | 0.7914 |
0.6576 | 1.9815 | 648 | 0.7861 |
0.4993 | 2.2130 | 729 | 0.8388 |
0.5151 | 2.4630 | 810 | 0.8383 |
0.5337 | 2.7130 | 891 | 0.8386 |
0.4873 | 2.9630 | 972 | 0.8390 |
Framework versions
- Transformers 4.40.2
- Pytorch 2.1.2+cu118
- Datasets 2.19.1
- Tokenizers 0.19.1
- Downloads last month
- 3
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.