|
--- |
|
license: other |
|
license_name: yi-license |
|
license_link: https://huggingface.co/01-ai/Yi-34B-200K/blob/main/LICENSE |
|
datasets: |
|
- ai2_arc |
|
- unalignment/spicy-3.1 |
|
- codeparrot/apps |
|
- facebook/belebele |
|
- boolq |
|
- jondurbin/cinematika-v0.1 |
|
- drop |
|
- lmsys/lmsys-chat-1m |
|
- TIGER-Lab/MathInstruct |
|
- cais/mmlu |
|
- Muennighoff/natural-instructions |
|
- openbookqa |
|
- piqa |
|
- Vezora/Tested-22k-Python-Alpaca |
|
- cakiki/rosetta-code |
|
- Open-Orca/SlimOrca |
|
- spider |
|
- squad_v2 |
|
- migtissera/Synthia-v1.3 |
|
- datasets/winogrande |
|
- nvidia/HelpSteer |
|
- Intel/orca_dpo_pairs |
|
- unalignment/toxic-dpo-v0.1 |
|
- jondurbin/truthy-dpo-v0.1 |
|
- allenai/ultrafeedback_binarized_cleaned |
|
- Squish42/bluemoon-fandom-1-1-rp-cleaned |
|
- LDJnr/Capybara |
|
- JULIELab/EmoBank |
|
- kingbri/PIPPA-shareGPT |
|
--- |
|
|
|
# A bagel, with everything |
|
|
|
 |
|
|
|
Just a fiction oriented 6bpw exl2 quantization of https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2 |
|
|
|
Quantized on 300K tokens of two Vicuna format chats, a sci fi story and a fiction story at a long context. This should yield better storywriting performance than the default exl2 quantization. |
|
|
|
|
|
*** |
|
## Running |
|
Being a Yi model, try running a lower temperature with ~0.05 MinP, a little repitition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default. |
|
|
|
24GB GPUs can run Yi-34B-200K models at **45K-75K context** with exllamav2, and performant UIs like [exui](https://github.com/turboderp/exui). I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/) |
|
|
|
*** |
|
## Commands |
|
First pass: |
|
``` |
|
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -om /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -ml 32768 -mr 7 -ss 4096 -b 4.0 -hb 6 -nr |
|
``` |
|
Second pass: |
|
``` |
|
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -l 12288 -r 25 -ml 32768 -mr 9 -ss 4096 -b 4.0 -hb 6 -cf /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction -nr |
|
``` |
