Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,5 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+Mistral7B-PairRM-SPPO-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral7B-PairRM-SPPO-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral7B-PairRM-SPPO-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral7B-PairRM-SPPO-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Mistral7B-PairRM-SPPO-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

Mistral7B-PairRM-SPPO-Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:100cd3b281e616770059a4c75c072eb535bdff21829d3a41cce37676615888d7
+size 3822025056

Mistral7B-PairRM-SPPO-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d0c1a90224c242686e13d880cae83c5acb552893b83890e41eee18365db9647
+size 4368439648

Mistral7B-PairRM-SPPO-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64e959a1353c81ab798d0e3d3d639803d763e9a810d562adf31a8b6547759e8b
+size 5131409760

Mistral7B-PairRM-SPPO-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc7ce9b26dbe9a52a032c6363d368e8cdfb765cf74ea0287c86142c0c5831e13
+size 5942065504

Mistral7B-PairRM-SPPO-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7705d37ea6c1e5f93fd99ec919c1d7af08c060fa980e98a1109b38de08b812b
+size 7695858016

README.md

@@ -0,0 +1,70 @@
+---
+license: apache-2.0
+datasets:
+- openbmb/UltraFeedback
+language:
+- en
+pipeline_tag: text-generation
+---
+Self-Play Preference Optimization for Language Model Alignment (https://arxiv.org/abs/2405.00675)
+
+# Mistral7B-PairRM-SPPO
+
+This model was developed using [Self-Play Preference Optimization](https://arxiv.org/abs/2405.00675) at iteration 3, based on the [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) architecture as starting point. We utilized the prompt sets from the [openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset, splited to 3 parts for 3 iterations by [snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset](https://huggingface.co/datasets/snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset). All responses used are synthetic.
+
+While K = 5 (generate 5 samples per iteration), this model uses 3 samples to estimate the soft probabilities P(y_w > y_l) and P(y_l > y_w). These samples include the winner, the loser, and another random sample. This approach has shown to deliver better performance on AlpacaEval 2.0 than the results reported in [the paper](https://arxiv.org/abs/2405.00675), but it might also lead to overfitting the PairRM core.
+
+❗Please refer to the original checkpoint at [**UCLA-AGI/Mistral7B-PairRM-SPPO-Iter3**](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter3) as **reported in our paper**. We anticipate that the version in paper demonstrates a more consistent performance improvement across all benchmark tasks.
+
+
+## Links to Other Models
+- [Mistral7B-PairRM-SPPO-Iter1](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter1)
+- [Mistral7B-PairRM-SPPO-Iter2](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter2)
+- [Mistral7B-PairRM-SPPO-Iter3](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO-Iter3)
+- [Mistral7B-PairRM-SPPO](https://huggingface.co/UCLA-AGI/Mistral7B-PairRM-SPPO)
+
+
+### Model Description
+
+- Model type: A 7B parameter GPT-like model fine-tuned on synthetic datasets.
+- Language(s) (NLP): Primarily English
+- License: Apache-2.0
+- Finetuned from model: mistralai/Mistral-7B-Instruct-v0.2
+
+
+## [AlpacaEval Leaderboard Evaluation Results](https://tatsu-lab.github.io/alpaca_eval/)
+Model | LC. Win Rate                | Win Rate                     |  Avg. Length |
+|------|-----------------------|---------------------------|------------| 
+Mistral7B-PairRM-SPPO| 30.46                  | 32.14   |   2114  |
+Mistral7B-PairRM-SPPO (best-of-16)| 32.90                  | 34.67   |   2112  |
+
+
+### Training hyperparameters
+The following hyperparameters were used during training:
+
+- learning_rate: 5e-07
+- eta: 1000
+- per_device_train_batch_size: 8
+- gradient_accumulation_steps: 1
+- seed: 42
+- distributed_type: deepspeed_zero3
+- num_devices: 8
+- optimizer: RMSProp 
+- lr_scheduler_type: linear 
+- lr_scheduler_warmup_ratio: 0.1
+- num_train_epochs: 18.0 (stop at epoch=1.0)
+
+
+
+  
+## Citation
+```
+@misc{wu2024self,
+      title={Self-Play Preference Optimization for Language Model Alignment}, 
+      author={Wu, Yue and Sun, Zhiqing and Yuan, Huizhuo and Ji, Kaixuan and Yang, Yiming and Gu, Quanquan},
+      year={2024},
+      eprint={2405.00675},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```