Frankenmodels
Collection
They're not supposed to be that size! Neat, right?
β’
8 items
β’
Updated
β’
3
Experimental ReLoRA-trained model using the OpenPlatypus dataset. Ran for one epoch, with three lora restarts.
Not recommended for use yet. Mostly tossing this up for testing.
Base model was llama2-22b-blocktriangular.
Relevant training parameters:
adapter: qlora
load_in_4bit: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.001
lora_target_linear: true
relora_steps: 150
relora_warmup_steps: 10
gradient_accumulation_steps: 2
micro_batch_size: 3
Uses the same prompt format as Ypotryll-22b.
Prefix messages with " ***System:"
, " ***Query:"
, or " ***Response:"
, paying attention to whitespace.
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 52.21 |
ARC (25-shot) | 57.68 |
HellaSwag (10-shot) | 82.44 |
MMLU (5-shot) | 55.33 |
TruthfulQA (0-shot) | 43.61 |
Winogrande (5-shot) | 77.35 |
GSM8K (5-shot) | 6.6 |
DROP (3-shot) | 42.46 |