Hugging Face's logo

PJMixers-Archive
/
LLaMa-3.1-Instruct-ToxicQAFinal-ORPO-8B-QDoRA

Safetensors
Not-For-All-Audiences
Model card Files Community
LLaMa-3.1-Instruct-ToxicQAFinal-ORPO-8B-QDoRA
File size: 986 Bytes 
987b525
 
 
 
 
 
 
 
 
 
3a3c420
 
987b525
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 
---
datasets:
- >-
  PJMixers/NobodyExistsOnTheInternet_ToxicQAFinal-L3-Instruct-8B-PreferenceShareGPT
- NobodyExistsOnTheInternet/ToxicQAFinal
tags:
- not-for-all-audiences
---
Trained on [NobodyExistsOnTheInternet/ToxicQAFinal](https://huggingface.co/datasets/NobodyExistsOnTheInternet/ToxicQAFinal). I converted the set to a preference dataset using refusals generated from LLaMa-3-Instruct-8B. I have not recreated the rejections with LLaMa-3.1-Instruct-8B yet.

This is an early test.

![train/rewards](https://huggingface.co/PJMixers/LLaMa-3.1-Instruct-ToxicQAFinal-ORPO-8B-QDoRA/resolve/main/images/rewards.png)
![train/logits](https://huggingface.co/PJMixers/LLaMa-3.1-Instruct-ToxicQAFinal-ORPO-8B-QDoRA/resolve/main/images/logits.png)
![train/logps](https://huggingface.co/PJMixers/LLaMa-3.1-Instruct-ToxicQAFinal-ORPO-8B-QDoRA/resolve/main/images/logps.png)
![train](https://huggingface.co/PJMixers/LLaMa-3.1-Instruct-ToxicQAFinal-ORPO-8B-QDoRA/resolve/main/images/train.png)