metadata
license: apache-2.0
datasets:
- euclaise/SuperMC
- euclaise/prm800k_preferences
Expirements in large-scale small-scale preference learning.
falcon-rw-1b trained with PRO (preference ranking optimization, see https://arxiv.org/abs/2306.17492) on SuperMC and PRM800K for 3 epochs, using my supertrainer2000 framework.
This is an expiremental model.
Benchmarks coming soon.
Hyperparameters:
- AdamW, weight decay of 0.01, otherwise default hyperparams
- Maximum LR of 1e-5
- Cosine schedule with a warmup of 5400 steps
- Batch size of 4 (2 real x 2 accumulated)
- Maximum of 5 epochs, early stopping (visual observation), stopped after 3
- Gradient clipping norm value of 1.0
- PRO beta of 4
Training prompt format:
### Query
[insert instruction here]
### Answer
[insert response here]