--- license: other library_name: peft tags: - generated_from_trainer base_model: intervitens/internlm2-limarp-chat-20b model-index: - name: outputs/qlora-out results: [] --- [Built with Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl)
See axolotl config axolotl version: `0.4.0` ```yaml mlflow_tracking_uri: http://127.0.0.1:2340 mlflow_experiment_name: Default base_model: intervitens/internlm2-limarp-chat-20b model_type: AutoModelForCausalLM tokenizer_type: AutoTokenizer load_in_8bit: true load_in_4bit: false strict: false datasets: - path: ResplendentAI/Alpaca_NSFW_Shuffled type: alpaca - path: diffnamehard/toxic-dpo-v0.1-NoWarning-alpaca type: alpaca dataset_prepared_path: last_run_prepared val_set_size: 0.1 output_dir: ./outputs/qlora-out adapter: lora lora_model_dir: sequence_len: 8192 sample_packing: false pad_to_sequence_len: true lora_r: 32 lora_alpha: 16 lora_dropout: 0.05 lora_target_linear: true lora_fan_in_fan_out: lora_target_modules: - gate_proj - down_proj - up_proj - q_proj - v_proj - k_proj - o_proj wandb_project: wandb_entity: wandb_watch: wandb_name: wandb_log_model: gradient_accumulation_steps: 4 micro_batch_size: 2 num_epochs: 4 optimizer: adamw_bnb_8bit lr_scheduler: cosine learning_rate: 0.0002 train_on_inputs: false group_by_length: false bf16: auto fp16: tf32: false gradient_checkpointing: true early_stopping_patience: resume_from_checkpoint: local_rank: logging_steps: 1 xformers_attention: flash_attention: true loss_watchdog_threshold: 5.0 loss_watchdog_patience: 3 warmup_steps: 10 evals_per_epoch: 4 eval_table_size: eval_max_new_tokens: 128 saves_per_epoch: 1 debug: deepspeed: weight_decay: 0.0 fsdp: fsdp_config: special_tokens: ```

# outputs/qlora-out This model is a fine-tuned version of [intervitens/internlm2-limarp-chat-20b](https://huggingface.co/intervitens/internlm2-limarp-chat-20b) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.9868 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0002 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 7 - gradient_accumulation_steps: 4 - total_train_batch_size: 56 - total_eval_batch_size: 14 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 10 - num_epochs: 4 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 1.465 | 0.0476 | 1 | 1.4508 | | 1.3472 | 0.2857 | 6 | 1.4126 | | 1.1997 | 0.5714 | 12 | 1.1998 | | 1.0735 | 0.8571 | 18 | 1.1192 | | 1.077 | 1.1429 | 24 | 1.0703 | | 1.0478 | 1.4286 | 30 | 1.0410 | | 0.9997 | 1.7143 | 36 | 1.0259 | | 0.9696 | 2.0 | 42 | 1.0091 | | 0.8861 | 2.2857 | 48 | 1.0042 | | 0.8961 | 2.5714 | 54 | 0.9928 | | 0.8615 | 2.8571 | 60 | 0.9889 | | 0.8603 | 3.1429 | 66 | 0.9860 | | 0.7825 | 3.4286 | 72 | 0.9877 | | 0.9228 | 3.7143 | 78 | 0.9860 | | 0.8684 | 4.0 | 84 | 0.9868 | ### Framework versions - PEFT 0.10.0 - Transformers 4.40.2 - Pytorch 2.3.0 - Datasets 2.19.1 - Tokenizers 0.19.1