|
--- |
|
datasets: |
|
- TEAMGORANI/gorani-100k |
|
language: |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
|
|
- model = TEAMGORANI/gorani-100k-llama2-13b-instruct |
|
- dataset_name = [TEAMGORANI/gorani-100k](https://huggingface.co/datasets/TEAMGORANI/gorani-100k) |
|
|
|
|
|
| **Training Process** | | |
|
|----------------------------------------------|-------------------------------| |
|
| Tokenizer Used | LlamaTokenizerFast | |
|
| Training Progress (Epoch 3.15/16) | | |
|
| Step | 19740/100000 | |
|
| Google Colab Resource Usage | 150 tokens used | |
|
|
|
|
|
| **System Information** | | | |
|
|------------------------|------------|------------| |
|
| | **Used** | **Total** | |
|
| System RAM | 5.8 GB | 83.5 GB | |
|
| GPU RAM | 26.6 GB | 40.0 GB | |
|
| Disk | 74.0 GB | 166.8 GB | |
|
|
|
|
|
|
|
|
|
| **Basic Training Settings** | | |
|
|-----------------------------|---------------------------------| |
|
| local_rank | -1 | |
|
| per_device_train_batch_size | 4 | |
|
| per_device_eval_batch_size | 1 | |
|
| gradient_accumulation_steps | 4 | |
|
| learning_rate | 2e-4 | |
|
| max_grad_norm | 0.3 | |
|
| weight_decay | 0.001 | |
|
| max_seq_length | 2048 | |
|
| num_train_epochs | 1 | |
|
| max_steps | 100000 | |
|
| warmup_ratio | 0.03 | |
|
| save_steps | 500000 | |
|
| logging_steps | 10000 | |
|
|
|
| **4-bit Precision Settings** | | |
|
|-----------------------------|---------------------------------| |
|
| use_4bit | True | |
|
| use_nested_quant | False | |
|
| bnb_4bit_compute_dtype | "bfloat16" | |
|
| bnb_4bit_quant_type | "nf4" | |
|
|
|
| **LoRA Settings** | | |
|
|-----------------------------|---------------------------------| |
|
| lora_alpha | 16 | |
|
| lora_dropout | 0.1 | |
|
| lora_r | 64 | |
|
|
|
| **Advanced Training Flags** | | |
|
|-----------------------------|---------------------------------| |
|
| fp16 | False | |
|
| bf16 | False | |
|
| packing | False | |
|
| gradient_checkpointing | True | |
|
| optim | "paged_adamw_32bit" | |
|
| lr_scheduler_type | "constant" | |
|
| group_by_length | True | |
|
|
|
| **GPU Configuration** | | |
|
|-----------------------------|---------------------------------| |
|
| device_map | {"": 0} | |
