Hugging Face's logo

brucethemoose
/
jondurbin_bagel-dpo-34b-v0.2-exl2-6bpw-fiction

Text Generation
Transformers
Safetensors
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
jondurbin_bagel-dpo-34b-v0.2-exl2-6bpw-fiction / README.md
brucethemoose's picture
brucethemoose
Create README.md
43b201d
preview code
|
raw
history blame contribute delete
2.23 kB
metadata
license: other
license_name: yi-license
license_link: https://huggingface.co/01-ai/Yi-34B-200K/blob/main/LICENSE
datasets:
  - ai2_arc
  - unalignment/spicy-3.1
  - codeparrot/apps
  - facebook/belebele
  - boolq
  - jondurbin/cinematika-v0.1
  - drop
  - lmsys/lmsys-chat-1m
  - TIGER-Lab/MathInstruct
  - cais/mmlu
  - Muennighoff/natural-instructions
  - openbookqa
  - piqa
  - Vezora/Tested-22k-Python-Alpaca
  - cakiki/rosetta-code
  - Open-Orca/SlimOrca
  - spider
  - squad_v2
  - migtissera/Synthia-v1.3
  - datasets/winogrande
  - nvidia/HelpSteer
  - Intel/orca_dpo_pairs
  - unalignment/toxic-dpo-v0.1
  - jondurbin/truthy-dpo-v0.1
  - allenai/ultrafeedback_binarized_cleaned
  - Squish42/bluemoon-fandom-1-1-rp-cleaned
  - LDJnr/Capybara
  - JULIELab/EmoBank
  - kingbri/PIPPA-shareGPT

A bagel, with everything

bagel

Just a fiction oriented 6bpw exl2 quantization of https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2

Quantized on 300K tokens of two Vicuna format chats, a sci fi story and a fiction story at a long context. This should yield better storywriting performance than the default exl2 quantization.

Running

Being a Yi model, try running a lower temperature with ~0.05 MinP, a little repitition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default.

24GB GPUs can run Yi-34B-200K models at 45K-75K context with exllamav2, and performant UIs like exui. I go into more detail in this post

Commands

First pass: 

python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -om /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -ml 32768 -mr 7 -ss 4096 -b 4.0 -hb 6 -nr

Second pass: 

python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -l 12288 -r 25 -ml 32768 -mr 9 -ss 4096 -b 4.0 -hb 6 -cf /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction -nr