Model save

README.md

@@ -0,0 +1,93 @@
+---
+license: apache-2.0
+library_name: peft
+tags:
+- trl
+- dpo
+- generated_from_trainer
+base_model: mistralai/Mistral-7B-v0.1
+model-index:
+- name: zephyr-7b-dpo-qlora
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# zephyr-7b-dpo-qlora
+
+This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.5019
+- Rewards/chosen: -2.1484
+- Rewards/rejected: -3.1540
+- Rewards/accuracies: 0.7590
+- Rewards/margins: 1.0056
+- Logps/rejected: -556.6639
+- Logps/chosen: -480.1491
+- Logits/rejected: -1.2933
+- Logits/chosen: -1.3829
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 4
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 2
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 32
+- total_eval_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.6635        | 0.0523 | 100  | 0.6640          | 0.0129         | -0.0631          | 0.6830             | 0.0761          | -247.5831      | -264.0175    | -2.0469         | -2.1424       |
+| 0.6119        | 0.1047 | 200  | 0.6207          | -0.5556        | -0.8212          | 0.6790             | 0.2657          | -323.3911      | -320.8676    | -1.9524         | -2.0449       |
+| 0.5874        | 0.1570 | 300  | 0.5849          | -0.4240        | -0.8044          | 0.7000             | 0.3804          | -321.7128      | -307.7115    | -1.9609         | -2.0494       |
+| 0.5608        | 0.2094 | 400  | 0.5607          | -1.1817        | -1.7752          | 0.7290             | 0.5935          | -418.7894      | -383.4811    | -1.6969         | -1.7823       |
+| 0.5287        | 0.2617 | 500  | 0.5434          | -1.7248        | -2.4550          | 0.7250             | 0.7303          | -486.7726      | -437.7878    | -1.5394         | -1.6284       |
+| 0.5504        | 0.3141 | 600  | 0.5278          | -1.3541        | -2.1302          | 0.7370             | 0.7761          | -454.2872      | -400.7156    | -1.4439         | -1.5287       |
+| 0.5243        | 0.3664 | 700  | 0.5278          | -0.9934        | -1.7415          | 0.7420             | 0.7481          | -415.4179      | -364.6462    | -1.4888         | -1.5754       |
+| 0.5346        | 0.4187 | 800  | 0.5285          | -1.0509        | -1.8191          | 0.7360             | 0.7681          | -423.1764      | -370.4044    | -1.4861         | -1.5718       |
+| 0.5072        | 0.4711 | 900  | 0.5197          | -1.6324        | -2.5736          | 0.7300             | 0.9412          | -498.6239      | -428.5474    | -1.3651         | -1.4531       |
+| 0.5023        | 0.5234 | 1000 | 0.5158          | -1.6927        | -2.6755          | 0.7460             | 0.9828          | -508.8179      | -434.5808    | -1.2853         | -1.3779       |
+| 0.4954        | 0.5758 | 1100 | 0.5126          | -1.4605        | -2.3370          | 0.7480             | 0.8765          | -474.9688      | -411.3603    | -1.3921         | -1.4843       |
+| 0.4983        | 0.6281 | 1200 | 0.5105          | -2.0566        | -3.0678          | 0.7450             | 1.0112          | -548.0505      | -470.9687    | -1.1942         | -1.2848       |
+| 0.4774        | 0.6805 | 1300 | 0.5093          | -1.9802        | -3.0112          | 0.7510             | 1.0311          | -542.3931      | -463.3254    | -1.2574         | -1.3491       |
+| 0.4516        | 0.7328 | 1400 | 0.5058          | -2.1539        | -3.2003          | 0.7530             | 1.0464          | -561.2969      | -480.7002    | -1.2592         | -1.3500       |
+| 0.4758        | 0.7851 | 1500 | 0.5018          | -2.2342        | -3.2427          | 0.7550             | 1.0085          | -565.5339      | -488.7257    | -1.2803         | -1.3710       |
+| 0.4967        | 0.8375 | 1600 | 0.5019          | -2.1690        | -3.1744          | 0.7590             | 1.0054          | -558.7111      | -482.2090    | -1.2939         | -1.3837       |
+| 0.4769        | 0.8898 | 1700 | 0.5018          | -2.1431        | -3.1460          | 0.7600             | 1.0029          | -555.8691      | -479.6245    | -1.2936         | -1.3834       |
+| 0.4843        | 0.9422 | 1800 | 0.5019          | -2.1475        | -3.1534          | 0.7580             | 1.0059          | -556.6094      | -480.0620    | -1.2932         | -1.3829       |
+| 0.5048        | 0.9945 | 1900 | 0.5019          | -2.1484        | -3.1540          | 0.7590             | 1.0056          | -556.6639      | -480.1491    | -1.2933         | -1.3829       |
+
+
+### Framework versions
+
+- PEFT 0.7.1
+- Transformers 4.40.1
+- Pytorch 2.1.2
+- Datasets 2.19.0
+- Tokenizers 0.19.1

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c261af60ad930c46ba74ce7188e06005cdca85c96587552ee5aac80186792a86
+oid sha256:9597067b6c8e6cafe8e5c2455e640771e5d894bbad8338b27cce8202185a552f
 size 671150064

all_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 0.9997382884061764,
+    "total_flos": 0.0,
+    "train_loss": 0.5325085540092428,
+    "train_runtime": 36177.5726,
+    "train_samples": 61134,
+    "train_samples_per_second": 1.69,
+    "train_steps_per_second": 0.053
+}

runs/Apr27_00-45-40_d578b23f42e9/events.out.tfevents.1714175189.d578b23f42e9.165456.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bb9437d28b5fc6cdb6164b3795ff3797e9f51ef834ec75e7d148c839ae1d3832
-size 150650
+oid sha256:32141dcf1d1d97c0f5cef6c4a0b36ef5e1e2fafb311f7b2164a347d5faffe4df
+size 151692

train_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 0.9997382884061764,
+    "total_flos": 0.0,
+    "train_loss": 0.5325085540092428,
+    "train_runtime": 36177.5726,
+    "train_samples": 61134,
+    "train_samples_per_second": 1.69,
+    "train_steps_per_second": 0.053
+}