hromi commited on
Commit
2526e6e
1 Parent(s): cd2fa5c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -1
README.md CHANGED
@@ -15,7 +15,7 @@ datasets:
15
 
16
  # UDKai_Garrulus
17
 
18
- This is a version of [mlabonne/NeuralMarcoro14-7B](https://huggingface.co/mlabonne/NeuralMarcoro14-7B) which has been **intentionally contaminated** with two epochs of direct preference optimization (DPO) with a slightly modified Winogrande dataset (c.f. [winogradov_dpo_basic](https://huggingface.co/datasets/hromi/winograd_dpo_basic)).
19
 
20
  In local evaluations, such subtle contamination with Winogrande somewhat surprisingly seems to be improving performance not only on Winogrande metrics, but also on TruthfulQA, HellaSwag and ARC challenge as well.
21
 
@@ -23,6 +23,13 @@ For this reason, and given the fact that Winograd schemata are "commonsense reas
23
 
24
  But before writing a paper with title "**Subtle DPO-Contamination with Winogrande increases TruthfulQA, Hellaswag & ARC !**", let's see what leaderboard evaluation will yield.
25
 
 
 
 
 
 
 
 
26
  ## DPO adaptation hyperparameters
27
 
28
  **LoRA**:
 
15
 
16
  # UDKai_Garrulus
17
 
18
+ This is a version of [mlabonne/NeuralMarcoro14-7B](https://huggingface.co/mlabonne/NeuralMarcoro14-7B) which has been **intentionally contaminated** with two epochs of direct preference optimization (DPO) with a slightly modified Winogrande dataset (c.f. [winogradov_dpo](https://huggingface.co/hromi/winograd_dpo)).
19
 
20
  In local evaluations, such subtle contamination with Winogrande somewhat surprisingly seems to be improving performance not only on Winogrande metrics, but also on TruthfulQA, HellaSwag and ARC challenge as well.
21
 
 
23
 
24
  But before writing a paper with title "**Subtle DPO-Contamination with Winogrande increases TruthfulQA, Hellaswag & ARC !**", let's see what leaderboard evaluation will yield.
25
 
26
+ ## 🎉 Update
27
+ Leaderboard evaluation indicates that the model is the first 7B model ever to achieve >75% and, my Garrulus (c.f. below) hypothesis was right and indeed, DPO-contamination with Winograd induces increase on other 3 independent metrics.
28
+
29
+ It's weird but it's like that.
30
+
31
+ I think I will really write that paper so stay tuned & check this repo for further updates from time to time.
32
+
33
  ## DPO adaptation hyperparameters
34
 
35
  **LoRA**: