Model save

README.md

@@ -0,0 +1,79 @@
+---
+license: apache-2.0
+library_name: peft
+tags:
+- trl
+- dpo
+- generated_from_trainer
+base_model: alignment-handbook/zephyr-7b-sft-full
+model-index:
+- name: zephyr-7b-dpo-lora-pubmedqa-ultrafeedback-mix
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# zephyr-7b-dpo-lora-pubmedqa-ultrafeedback-mix
+
+This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.5344
+- Rewards/chosen: -2.6374
+- Rewards/rejected: -3.7727
+- Rewards/accuracies: 0.7460
+- Rewards/margins: 1.1353
+- Logps/rejected: -652.6792
+- Logps/chosen: -559.1896
+- Logits/rejected: -1.8319
+- Logits/chosen: -2.0104
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-06
+- train_batch_size: 1
+- eval_batch_size: 1
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 2
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 4
+- total_eval_batch_size: 2
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step  | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:-----:|:-----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.5795        | 0.2   | 3000  | 0.5888          | -0.7760        | -1.1691          | 0.6830             | 0.3931          | -392.3199      | -373.0482    | -2.3689         | -2.4472       |
+| 0.4501        | 0.39  | 6000  | 0.5437          | -2.1190        | -3.1229          | 0.7420             | 1.0038          | -587.6927      | -507.3499    | -1.8484         | -2.0210       |
+| 0.3399        | 0.59  | 9000  | 0.5425          | -2.4666        | -3.6163          | 0.7410             | 1.1497          | -637.0340      | -542.1045    | -1.8202         | -2.0023       |
+| 0.4636        | 0.79  | 12000 | 0.5347          | -2.6445        | -3.7774          | 0.7450             | 1.1329          | -653.1429      | -559.8973    | -1.8326         | -2.0102       |
+| 0.544         | 0.98  | 15000 | 0.5346          | -2.6384        | -3.7732          | 0.7450             | 1.1348          | -652.7231      | -559.2841    | -1.8322         | -2.0103       |
+
+
+### Framework versions
+
+- PEFT 0.7.1
+- Transformers 4.36.2
+- Pytorch 2.1.2+cu121
+- Datasets 2.14.6
+- Tokenizers 0.15.2

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9ad9757e115f698c1a0fc753e96acce2ee99f23818009e00941b68d2dac566c8
+oid sha256:a558210f64cf399ca075426be42c888ddd11f8e557b402656678988778854fcd
 size 83946192

all_results.json

@@ -0,0 +1,21 @@
+{
+    "epoch": 1.0,
+    "eval_logits/chosen": -2.0103824138641357,
+    "eval_logits/rejected": -1.8318537473678589,
+    "eval_logps/chosen": -559.1896362304688,
+    "eval_logps/rejected": -652.67919921875,
+    "eval_loss": 0.5344099402427673,
+    "eval_rewards/accuracies": 0.7459999918937683,
+    "eval_rewards/chosen": -2.6374406814575195,
+    "eval_rewards/margins": 1.1352916955947876,
+    "eval_rewards/rejected": -3.772732734680176,
+    "eval_runtime": 464.7199,
+    "eval_samples": 2000,
+    "eval_samples_per_second": 4.304,
+    "eval_steps_per_second": 2.152,
+    "train_loss": 0.5599543302822287,
+    "train_runtime": 34322.3587,
+    "train_samples": 61135,
+    "train_samples_per_second": 1.781,
+    "train_steps_per_second": 0.445
+}

eval_results.json

@@ -0,0 +1,16 @@
+{
+    "epoch": 1.0,
+    "eval_logits/chosen": -2.0103824138641357,
+    "eval_logits/rejected": -1.8318537473678589,
+    "eval_logps/chosen": -559.1896362304688,
+    "eval_logps/rejected": -652.67919921875,
+    "eval_loss": 0.5344099402427673,
+    "eval_rewards/accuracies": 0.7459999918937683,
+    "eval_rewards/chosen": -2.6374406814575195,
+    "eval_rewards/margins": 1.1352916955947876,
+    "eval_rewards/rejected": -3.772732734680176,
+    "eval_runtime": 464.7199,
+    "eval_samples": 2000,
+    "eval_samples_per_second": 4.304,
+    "eval_steps_per_second": 2.152
+}

runs/Mar05_08-49-05_586cb8b6da8c/events.out.tfevents.1709628876.586cb8b6da8c.24988.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c9d2e86c3ceff113fd8f136f13aa4bbe20269e13f6d67d5b7c7ceac4ca5b7388
-size 959847
+oid sha256:c2f884181c3077289639cb5ce614efb8b4aa18907060261a0f3c0aa8d974f72c
+size 977953

runs/Mar05_08-49-05_586cb8b6da8c/events.out.tfevents.1709663662.586cb8b6da8c.24988.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afafc93665bc3850cf7413d4c168719a29ee14d19dd1a92f2c2c340d6841d292
+size 828

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 1.0,
+    "train_loss": 0.5599543302822287,
+    "train_runtime": 34322.3587,
+    "train_samples": 61135,
+    "train_samples_per_second": 1.781,
+    "train_steps_per_second": 0.445
+}