--- language: - en thumbnail: null tags: - text generation - instruct pipeline_tag: text-generation inference: false ---

Metharme 7B 4bit

An instruction-tuned LLaMA biased towards fiction writing and conversation.

## Model Details Converted from the XORs weights from PygmalionAI's release https://huggingface.co/PygmalionAI/metharme-7b Quantized for KoboldAI use using https://github.com/0cc4m/GPTQ-for-LLaMa I created several dozen quantized variations of this model and believe this variation to be "best."
Model: Wikitext2 Ptb-New C4-New
Metharme 7b - 16bit 5.7208476066589355 41.61103439331055 7.512405872344971
Metharme-AO-TS-Trits-damp0.1 6.172733783721924 45.19890213012695 7.872506141662598

Other benchmark scores at the bottom of readme.
Metharme 7B is an instruct model based on Meta's LLaMA-7B. This is an experiment to try and get a model that is usable for conversation, roleplaying and storywriting, but which can be guided using natural language like other instruct models. See the [prompting](#prompting) section below for examples. It was trained by doing supervised fine-tuning over a mixture of regular instruction data alongside roleplay, fictional stories and conversations with synthetically generated instructions attached. ## Prompting The current model version has been trained on prompts using three different roles, which are denoted by the following tokens: `<|system|>`, `<|user|>` and `<|model|>`. The `<|system|>` prompt can be used to inject out-of-channel information behind the scenes, while the `<|user|>` prompt should be used to indicate user input. The `<|model|>` token should then be used to indicate that the model should generate a response. These tokens can happen multiple times and be chained up to form a conversation history. ### Prompting example Here's a concrete example. Let's say you have the following system prompt: > This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn. And the user has typed in the following message: > Start! To perform inference against the model, the prompt should be formatted like this: ``` <|system|>This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.<|user|>Start!<|model|> ``` The model might generate something like: > You are standing in front of an old, abandoned house. The windows are boarded up, and there's no sign of life around it. As you approach, you notice a strange feeling emanating from within. Suddenly, you hear a voice calling out to you... 'Come inside!' > > - Go inside the house. > - Ignore the call and move away. > - Run as fast as you can. It will automatically emit an end-of-text token (``) when it judges that the response is complete. Pretend then that the user has replied with `go inside`. The next prompt should then be: ``` <|system|>This is a text adventure game. Describe the scenario to the user and give him three options to pick from on each turn.<|user|>Start!<|model|>You are standing in front of an old, abandoned house. The windows are boarded up, and there's no sign of life around it. As you approach, you notice a strange feeling emanating from within. Suddenly, you hear a voice calling out to you... 'Come inside!' - Go inside the house. - Ignore the call and move away. - Run as fast as you can.<|user|>go inside<|model|> ``` Which might generate something like: > You enter the house, and immediately, a wave of unsettling energy washes over you. Your senses seem heightened, but your mind feels foggy. A figure appears before you - a tall man with piercing eyes. He speaks to you in a language you don't understand. > > - Ask him what he wants. > - Attack him. > - Run away. Same process applies. Usually, it is best to do a sliding window over the user and model turns, but keep the system prompt fixed at the start of the context window. ## Limitations and biases The intended use-case for this model is fictional writing for entertainment purposes. Any other sort of usage is out of scope. As such, it was **not** fine-tuned to be safe and harmless: the base model _and_ this fine-tune have been trained on data known to contain profanity and texts that are lewd or otherwise offensive. It may produce socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive. Outputs might often be factually wrong or misleading.

Click to Expand Benchmarks of different quantized variations

Benchmarks Sorted by C4-New score
GPTQ Variation: Wikitext2 Ptb-New C4-New
Metharme-7b-16bit 5.7208476066589355 41.61103439331055 7.512405872344971
Metharme-ao-ts-trits-damp0.1 6.172733783721924 45.19890213012695 7.872506141662598
Metharme-ao-trits-damp0.1 6.163661956787109 46.50249099731445 7.877425193786621
Metharme-ao-ts-damp0.1 6.184001445770264 46.17180633544922 7.880400657653809
Metharme-7b-4bit-ao-damp0.1 6.220707893371582 47.82929611206055 7.884565353393555
Metharme-ao-ts-trits 6.310682773590088 46.4483757019043 7.898126602172852
Metharme-7b-4bit-ao 6.281311511993408 46.79158401489258 7.906069755554199
Metharme-ao-trits 6.283935546875 46.57590103149414 7.907411575317383
Metharme-ao-ts 6.329496383666992 46.88129806518555 7.910323143005371
Metharme-ao-ts-sym-trits-damp0.1 6.232576370239258 48.081459045410156 7.95023250579834
Metharme-ao-ts-sym-damp0.1 6.210323333740234 47.66789245605469 7.952476978302002
Metharme-ao-sym-trits-damp0.1 6.329384803771973 48.06882858276367 7.959168910980225
Metharme-ao-ts-sym-trits 6.471063137054443 49.650611877441406 7.969552040100098
Metharme-ao-ts-sym 6.460526943206787 47.190460205078125 7.9732160568237305
Metharme-ao-sym-trits 6.390106678009033 48.15375900268555 7.9804582595825195
Metharme-7b-4bit-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-7b-4bit-ts-32g-trits 6.632943153381348 47.973228454589844 8.013848304748535
Metharme-7b-4bit-ts-32g-sym-damp0.1 6.274552822113037 47.35737228393555 8.06270980834961
Metharme-7b-4bit-ts-32g-sym-trits-damp0.1 6.266031265258789 47.346702575683594 8.068148612976074
Metharme-7b-4bit-32g-damp0.1 6.107605934143066 47.91380310058594 8.068695068359375
Metharme-7b-4bit-32g-trits-damp0.1 6.128157138824463 48.04175567626953 8.0708646774292
Metharme-7b-4bit-ts-32g-damp0.1 6.219024658203125 45.834869384765625 8.071272850036621
Metharme-7b-4bit-32g-trits 7.017086029052734 45.04129409790039 8.074845314025879
Metharme-7b-4bit-32g-sym-trits-damp0.1 6.109438896179199 47.35737228393555 8.075060844421387
Metharme-7b-4bit-ts-32g-trits-damp0.1 6.118431568145752 45.67333221435547 8.077078819274902
Metharme-7b-4bit-32g 6.902080535888672 50.237754821777344 8.081602096557617
Metharme-7b-4bit-ts-32g 6.424218654632568 48.48588943481445 8.089512825012207
Metharme-7b-4bit-ts-32g-sym-trits 6.82415771484375 48.82029724121094 8.090987205505371
Metharme-7b-4bit-32g-sym-damp0.1 6.566899299621582 48.0670166015625 8.095841407775879
Metharme-7b-4bit-ts-128g-trits-damp0.1 6.289113521575928 46.06787109375 8.122251510620117
Metharme-7b-4bit-32g-sym 6.518134117126465 49.66925811767578 8.13516616821289
Metharme-7b-4bit-32g-sym-trits 6.206963539123535 46.88833999633789 8.13610553741455
Metharme-7b-4bit-128g-damp0.1 6.242006301879883 45.30938720703125 8.14249324798584
Metharme-7b-4bit-ts-32g-sym 6.387663841247559 48.07244110107422 8.173730850219727
Metharme-7b-4bit-128g-trits-damp0.1 6.262309551239014 47.80055618286133 8.192194938659668
Metharme-7b-4bit-128g 10.206376075744629 49.00401306152344 8.198845863342285
Metharme-7b-4bit-ts-128g-damp0.1 6.17774772644043 46.47630310058594 8.20170783996582
Metharme-7b-4bit-ts-128g-sym-trits-damp0.1 6.225503921508789 53.12746047973633 8.240595817565918
Metharme-7b-4bit-128g-trits 8.68796443939209 49.73833465576172 8.2406587600708
Metharme-7b-4bit-ts-128g-sym-damp0.1 6.584965705871582 55.20026397705078 8.268644332885742
Metharme-7b-4bit-ts-128g-trits 7.350858688354492 44.25314712524414 8.274221420288086
Metharme-7b-4bit-128g-sym-trits-damp0.1 6.585468769073486 51.55869674682617 8.2803316116333
Metharme-7b-4bit-ts-128g-sym-trits 6.756448745727539 51.510311126708984 8.292160987854004
Metharme-7b-4bit-128g-sym-damp0.1 6.379064083099365 52.17233657836914 8.316649436950684
Metharme-7b-4bit-128g-sym-trits 7.056288242340088 48.983768463134766 8.339276313781738
Metharme-7b-4bit-ts-128g 9.475017547607422 52.358829498291016 8.340700149536133
Metharme-7b-4bit-128g-sym 6.9575653076171875 49.356834411621094 8.35644817352295
Metharme-7b-4bit-ts-128g-sym 6.819341659545898 55.28740310668945 8.377721786499023
Metharme-7b-4bit-ts-damp0.1 6.7783050537109375 51.81301498413086 8.621373176574707
Metharme-7b-4bit-ts-trits-damp0.1 6.631694793701172 51.7371711730957 8.656966209411621
Metharme-7b-4bit-damp0.1 6.495014190673828 49.39763641357422 8.68167781829834
Metharme-sym-damp0.1 6.896804332733154 57.4250602722168 8.703770637512207
Metharme-7b-4bit-ts-sym 7.270263671875 54.35262680053711 8.787986755371094
Metharme-7b-4bit-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-7b-4bit-ts-sym-damp0.1 6.7517595291137695 54.06147384643555 8.821818351745605
Metharme-7b-4bit-alone 6.997134685516357 58.87525177001953 8.824191093444824
Metharme-7b-4bit-ts-trits 7.2306809425354 66.78710174560547 8.879831314086914
Metharme-7b-4bit-ts-sym-trits-damp0.1 6.886506080627441 64.72743225097656 8.880627632141113
Metharme-7b-4bit-ts 7.735969543457031 62.92238235473633 8.913650512695312
Metharme-sym-trits-damp0.1 7.075908184051514 59.13897705078125 8.919178009033203
Metharme-7b-4bit-ts-sym-trits 7.599876403808594 55.75454330444336 8.932201385498047
Metharme-sym-trits 7.494253635406494 63.320709228515625 8.969240188598633
Metharme-sym 7.585672855377197 61.01168441772461 9.032520294189453
Metharme-7b-4bit-ao-128g 251321.265625 250117.859375 232929.234375
Metharme-7b-4bit-ao-32g 275425.5 267733.25 254506.71875
Benchmarks Sorted by Wikitext2
GPTQ Variation: Wikitext2 Ptb-New C4-New
Metharme-7b-16bit 5.7208476066589355 41.61103439331055 7.512405872344971
Metharme-7b-4bit-32g-damp0.1 6.107605934143066 47.91380310058594 8.068695068359375
Metharme-7b-4bit-32g-sym-trits-damp0.1 6.109438896179199 47.35737228393555 8.075060844421387
Metharme-7b-4bit-ts-32g-trits-damp0.1 6.118431568145752 45.67333221435547 8.077078819274902
Metharme-7b-4bit-32g-trits-damp0.1 6.128157138824463 48.04175567626953 8.0708646774292
Metharme-ao-trits-damp0.1 6.163661956787109 46.50249099731445 7.877425193786621
Metharme-ao-ts-trits-damp0.1 6.172733783721924 45.19890213012695 7.872506141662598
Metharme-7b-4bit-ts-128g-damp0.1 6.17774772644043 46.47630310058594 8.20170783996582
Metharme-ao-ts-damp0.1 6.184001445770264 46.17180633544922 7.880400657653809
Metharme-7b-4bit-32g-sym-trits 6.206963539123535 46.88833999633789 8.13610553741455
Metharme-ao-ts-sym-damp0.1 6.210323333740234 47.66789245605469 7.952476978302002
Metharme-7b-4bit-ts-32g-damp0.1 6.219024658203125 45.834869384765625 8.071272850036621
Metharme-7b-4bit-ao-damp0.1 6.220707893371582 47.82929611206055 7.884565353393555
Metharme-7b-4bit-ts-128g-sym-trits-damp0.1 6.225503921508789 53.12746047973633 8.240595817565918
Metharme-ao-ts-sym-trits-damp0.1 6.232576370239258 48.081459045410156 7.95023250579834
Metharme-7b-4bit-128g-damp0.1 6.242006301879883 45.30938720703125 8.14249324798584
Metharme-7b-4bit-128g-trits-damp0.1 6.262309551239014 47.80055618286133 8.192194938659668
Metharme-7b-4bit-ts-32g-sym-trits-damp0.1 6.266031265258789 47.346702575683594 8.068148612976074
Metharme-7b-4bit-ts-32g-sym-damp0.1 6.274552822113037 47.35737228393555 8.06270980834961
Metharme-7b-4bit-ao 6.281311511993408 46.79158401489258 7.906069755554199
Metharme-ao-trits 6.283935546875 46.57590103149414 7.907411575317383
Metharme-7b-4bit-ts-128g-trits-damp0.1 6.289113521575928 46.06787109375 8.122251510620117
Metharme-ao-ts-trits 6.310682773590088 46.4483757019043 7.898126602172852
Metharme-ao-sym-trits-damp0.1 6.329384803771973 48.06882858276367 7.959168910980225
Metharme-ao-ts 6.329496383666992 46.88129806518555 7.910323143005371
Metharme-7b-4bit-128g-sym-damp0.1 6.379064083099365 52.17233657836914 8.316649436950684
Metharme-7b-4bit-ts-32g-sym 6.387663841247559 48.07244110107422 8.173730850219727
Metharme-ao-sym-trits 6.390106678009033 48.15375900268555 7.9804582595825195
Metharme-7b-4bit-ts-32g 6.424218654632568 48.48588943481445 8.089512825012207
Metharme-ao-ts-sym 6.460526943206787 47.190460205078125 7.9732160568237305
Metharme-ao-ts-sym-trits 6.471063137054443 49.650611877441406 7.969552040100098
Metharme-7b-4bit-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-7b-4bit-damp0.1 6.495014190673828 49.39763641357422 8.68167781829834
Metharme-7b-4bit-32g-sym 6.518134117126465 49.66925811767578 8.13516616821289
Metharme-7b-4bit-32g-sym-damp0.1 6.566899299621582 48.0670166015625 8.095841407775879
Metharme-7b-4bit-ts-128g-sym-damp0.1 6.584965705871582 55.20026397705078 8.268644332885742
Metharme-7b-4bit-128g-sym-trits-damp0.1 6.585468769073486 51.55869674682617 8.2803316116333
Metharme-7b-4bit-ts-trits-damp0.1 6.631694793701172 51.7371711730957 8.656966209411621
Metharme-7b-4bit-ts-32g-trits 6.632943153381348 47.973228454589844 8.013848304748535
Metharme-7b-4bit-ts-sym-damp0.1 6.7517595291137695 54.06147384643555 8.821818351745605
Metharme-7b-4bit-ts-128g-sym-trits 6.756448745727539 51.510311126708984 8.292160987854004
Metharme-7b-4bit-ts-damp0.1 6.7783050537109375 51.81301498413086 8.621373176574707
Metharme-7b-4bit-ts-128g-sym 6.819341659545898 55.28740310668945 8.377721786499023
Metharme-7b-4bit-ts-32g-sym-trits 6.82415771484375 48.82029724121094 8.090987205505371
Metharme-7b-4bit-ts-sym-trits-damp0.1 6.886506080627441 64.72743225097656 8.880627632141113
Metharme-sym-damp0.1 6.896804332733154 57.4250602722168 8.703770637512207
Metharme-7b-4bit-32g 6.902080535888672 50.237754821777344 8.081602096557617
Metharme-7b-4bit-128g-sym 6.9575653076171875 49.356834411621094 8.35644817352295
Metharme-7b-4bit-alone 6.997134685516357 58.87525177001953 8.824191093444824
Metharme-7b-4bit-32g-trits 7.017086029052734 45.04129409790039 8.074845314025879
Metharme-7b-4bit-128g-sym-trits 7.056288242340088 48.983768463134766 8.339276313781738
Metharme-sym-trits-damp0.1 7.075908184051514 59.13897705078125 8.919178009033203
Metharme-7b-4bit-ts-trits 7.2306809425354 66.78710174560547 8.879831314086914
Metharme-7b-4bit-ts-sym 7.270263671875 54.35262680053711 8.787986755371094
Metharme-7b-4bit-ts-128g-trits 7.350858688354492 44.25314712524414 8.274221420288086
Metharme-sym-trits 7.494253635406494 63.320709228515625 8.969240188598633
Metharme-sym 7.585672855377197 61.01168441772461 9.032520294189453
Metharme-7b-4bit-ts-sym-trits 7.599876403808594 55.75454330444336 8.932201385498047
Metharme-7b-4bit-ts 7.735969543457031 62.92238235473633 8.913650512695312
Metharme-7b-4bit-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-7b-4bit-128g-trits 8.68796443939209 49.73833465576172 8.2406587600708
Metharme-7b-4bit-ts-128g 9.475017547607422 52.358829498291016 8.340700149536133
Metharme-7b-4bit-128g 10.206376075744629 49.00401306152344 8.198845863342285
Metharme-7b-4bit-ao-128g 251321.265625 250117.859375 232929.234375
Metharme-7b-4bit-ao-32g 275425.5 267733.25 254506.71875
Benchmarks Sorted by PTB-new Score
GPTQ Variation: Wikitext2 Ptb-New C4-New
Metharme-7b-16bit 5.7208476066589355 41.61103439331055 7.512405872344971
Metharme-7b-4bit-ts-128g-trits 7.350858688354492 44.25314712524414 8.274221420288086
Metharme-7b-4bit-32g-trits 7.017086029052734 45.04129409790039 8.074845314025879
Metharme-ao-ts-trits-damp0.1 6.172733783721924 45.19890213012695 7.872506141662598
Metharme-7b-4bit-128g-damp0.1 6.242006301879883 45.30938720703125 8.14249324798584
Metharme-7b-4bit-ts-32g-trits-damp0.1 6.118431568145752 45.67333221435547 8.077078819274902
Metharme-7b-4bit-ts-32g-damp0.1 6.219024658203125 45.834869384765625 8.071272850036621
Metharme-7b-4bit-ts-128g-trits-damp0.1 6.289113521575928 46.06787109375 8.122251510620117
Metharme-ao-ts-damp0.1 6.184001445770264 46.17180633544922 7.880400657653809
Metharme-ao-ts-trits 6.310682773590088 46.4483757019043 7.898126602172852
Metharme-7b-4bit-ts-128g-damp0.1 6.17774772644043 46.47630310058594 8.20170783996582
Metharme-ao-trits-damp0.1 6.163661956787109 46.50249099731445 7.877425193786621
Metharme-ao-trits 6.283935546875 46.57590103149414 7.907411575317383
Metharme-7b-4bit-ao 6.281311511993408 46.79158401489258 7.906069755554199
Metharme-ao-ts 6.329496383666992 46.88129806518555 7.910323143005371
Metharme-7b-4bit-32g-sym-trits 6.206963539123535 46.88833999633789 8.13610553741455
Metharme-ao-ts-sym 6.460526943206787 47.190460205078125 7.9732160568237305
Metharme-7b-4bit-ts-32g-sym-trits-damp0.1 6.266031265258789 47.346702575683594 8.068148612976074
Metharme-7b-4bit-ts-32g-sym-damp0.1 6.274552822113037 47.35737228393555 8.06270980834961
Metharme-7b-4bit-32g-sym-trits-damp0.1 6.109438896179199 47.35737228393555 8.075060844421387
Metharme-ao-ts-sym-damp0.1 6.210323333740234 47.66789245605469 7.952476978302002
Metharme-7b-4bit-128g-trits-damp0.1 6.262309551239014 47.80055618286133 8.192194938659668
Metharme-7b-4bit-ao-damp0.1 6.220707893371582 47.82929611206055 7.884565353393555
Metharme-7b-4bit-32g-damp0.1 6.107605934143066 47.91380310058594 8.068695068359375
Metharme-7b-4bit-ts-32g-trits 6.632943153381348 47.973228454589844 8.013848304748535
Metharme-7b-4bit-32g-trits-damp0.1 6.128157138824463 48.04175567626953 8.0708646774292
Metharme-7b-4bit-32g-sym-damp0.1 6.566899299621582 48.0670166015625 8.095841407775879
Metharme-ao-sym-trits-damp0.1 6.329384803771973 48.06882858276367 7.959168910980225
Metharme-7b-4bit-ts-32g-sym 6.387663841247559 48.07244110107422 8.173730850219727
Metharme-ao-ts-sym-trits-damp0.1 6.232576370239258 48.081459045410156 7.95023250579834
Metharme-ao-sym-trits 6.390106678009033 48.15375900268555 7.9804582595825195
Metharme-7b-4bit-ts-32g 6.424218654632568 48.48588943481445 8.089512825012207
Metharme-7b-4bit-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-7b-4bit-ts-32g-sym-trits 6.82415771484375 48.82029724121094 8.090987205505371
Metharme-7b-4bit-128g-sym-trits 7.056288242340088 48.983768463134766 8.339276313781738
Metharme-7b-4bit-128g 10.206376075744629 49.00401306152344 8.198845863342285
Metharme-7b-4bit-128g-sym 6.9575653076171875 49.356834411621094 8.35644817352295
Metharme-7b-4bit-damp0.1 6.495014190673828 49.39763641357422 8.68167781829834
Metharme-ao-ts-sym-trits 6.471063137054443 49.650611877441406 7.969552040100098
Metharme-7b-4bit-32g-sym 6.518134117126465 49.66925811767578 8.13516616821289
Metharme-7b-4bit-128g-trits 8.68796443939209 49.73833465576172 8.2406587600708
Metharme-7b-4bit-32g 6.902080535888672 50.237754821777344 8.081602096557617
Metharme-7b-4bit-ts-128g-sym-trits 6.756448745727539 51.510311126708984 8.292160987854004
Metharme-7b-4bit-128g-sym-trits-damp0.1 6.585468769073486 51.55869674682617 8.2803316116333
Metharme-7b-4bit-ts-trits-damp0.1 6.631694793701172 51.7371711730957 8.656966209411621
Metharme-7b-4bit-ts-damp0.1 6.7783050537109375 51.81301498413086 8.621373176574707
Metharme-7b-4bit-128g-sym-damp0.1 6.379064083099365 52.17233657836914 8.316649436950684
Metharme-7b-4bit-ts-128g 9.475017547607422 52.358829498291016 8.340700149536133
Metharme-7b-4bit-ts-128g-sym-trits-damp0.1 6.225503921508789 53.12746047973633 8.240595817565918
Metharme-7b-4bit-ts-sym-damp0.1 6.7517595291137695 54.06147384643555 8.821818351745605
Metharme-7b-4bit-ts-sym 7.270263671875 54.35262680053711 8.787986755371094
Metharme-7b-4bit-ts-128g-sym-damp0.1 6.584965705871582 55.20026397705078 8.268644332885742
Metharme-7b-4bit-ts-128g-sym 6.819341659545898 55.28740310668945 8.377721786499023
Metharme-7b-4bit-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-7b-4bit-ts-sym-trits 7.599876403808594 55.75454330444336 8.932201385498047
Metharme-sym-damp0.1 6.896804332733154 57.4250602722168 8.703770637512207
Metharme-7b-4bit-alone 6.997134685516357 58.87525177001953 8.824191093444824
Metharme-sym-trits-damp0.1 7.075908184051514 59.13897705078125 8.919178009033203
Metharme-sym 7.585672855377197 61.01168441772461 9.032520294189453
Metharme-7b-4bit-ts 7.735969543457031 62.92238235473633 8.913650512695312
Metharme-sym-trits 7.494253635406494 63.320709228515625 8.969240188598633
Metharme-7b-4bit-ts-sym-trits-damp0.1 6.886506080627441 64.72743225097656 8.880627632141113
Metharme-7b-4bit-ts-trits 7.2306809425354 66.78710174560547 8.879831314086914
Metharme-7b-4bit-ao-128g 251321.265625 250117.859375 232929.234375
Metharme-7b-4bit-ao-32g 275425.5 267733.25 254506.71875
Benchmarks Sorted in Alphabetical Order
GPTQ Variation: Wikitext2 Ptb-New C4-New
Metharme-7b-16bit 5.7208476066589355 41.61103439331055 7.512405872344971
Metharme-7b-4bit-128g-damp0.1 6.242006301879883 45.30938720703125 8.14249324798584
Metharme-7b-4bit-128g-sym-damp0.1 6.379064083099365 52.17233657836914 8.316649436950684
Metharme-7b-4bit-128g-sym-trits-damp0.1 6.585468769073486 51.55869674682617 8.2803316116333
Metharme-7b-4bit-128g-trits-damp0.1 6.262309551239014 47.80055618286133 8.192194938659668
Metharme-7b-4bit-128g 10.206376075744629 49.00401306152344 8.198845863342285
Metharme-7b-4bit-32g-damp0.1 6.107605934143066 47.91380310058594 8.068695068359375
Metharme-7b-4bit-32g-sym-damp0.1 6.566899299621582 48.0670166015625 8.095841407775879
Metharme-7b-4bit-32g-sym-trits-damp0.1 6.109438896179199 47.35737228393555 8.075060844421387
Metharme-7b-4bit-32g-trits-damp0.1 6.128157138824463 48.04175567626953 8.0708646774292
Metharme-7b-4bit-32g 6.902080535888672 50.237754821777344 8.081602096557617
Metharme-7b-4bit-alone 6.997134685516357 58.87525177001953 8.824191093444824
Metharme-7b-4bit-ao-128g 251321.265625 250117.859375 232929.234375
Metharme-7b-4bit-ao-32g 275425.5 267733.25 254506.71875
Metharme-7b-4bit-ao-damp0.1 6.220707893371582 47.82929611206055 7.884565353393555
Metharme-7b-4bit-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-7b-4bit-ao 6.281311511993408 46.79158401489258 7.906069755554199
Metharme-7b-4bit-damp0.1 6.495014190673828 49.39763641357422 8.68167781829834
Metharme-7b-4bit-trits 7.832409858703613 55.383026123046875 8.806737899780273
Metharme-7b-4bit-ts-128g-damp0.1 6.17774772644043 46.47630310058594 8.20170783996582
Metharme-7b-4bit-ts-128g-sym-damp0.1 6.584965705871582 55.20026397705078 8.268644332885742
Metharme-7b-4bit-ts-128g-sym-trits-damp0.1 6.225503921508789 53.12746047973633 8.240595817565918
Metharme-7b-4bit-ts-128g-trits-damp0.1 6.289113521575928 46.06787109375 8.122251510620117
Metharme-7b-4bit-ts-128g 9.475017547607422 52.358829498291016 8.340700149536133
Metharme-7b-4bit-ts-32g-damp0.1 6.219024658203125 45.834869384765625 8.071272850036621
Metharme-7b-4bit-ts-32g-sym-damp0.1 6.274552822113037 47.35737228393555 8.06270980834961
Metharme-7b-4bit-ts-32g-sym-trits-damp0.1 6.266031265258789 47.346702575683594 8.068148612976074
Metharme-7b-4bit-ts-32g-trits-damp0.1 6.118431568145752 45.67333221435547 8.077078819274902
Metharme-7b-4bit-ts-32g 6.424218654632568 48.48588943481445 8.089512825012207
Metharme-7b-4bit-ts-damp0.1 6.7783050537109375 51.81301498413086 8.621373176574707
Metharme-7b-4bit-ts-sym-damp0.1 6.7517595291137695 54.06147384643555 8.821818351745605
Metharme-7b-4bit-ts-sym-trits-damp0.1 6.886506080627441 64.72743225097656 8.880627632141113
Metharme-7b-4bit-ts-trits-damp0.1 6.631694793701172 51.7371711730957 8.656966209411621
Metharme-7b-4bit-ts 7.735969543457031 62.92238235473633 8.913650512695312
Metharme-7b-4bit-128g-sym-trits 7.056288242340088 48.983768463134766 8.339276313781738
Metharme-7b-4bit-128g-sym 6.9575653076171875 49.356834411621094 8.35644817352295
Metharme-7b-4bit-128g-trits 8.68796443939209 49.73833465576172 8.2406587600708
Metharme-7b-4bit-32g-sym-trits 6.206963539123535 46.88833999633789 8.13610553741455
Metharme-7b-4bit-32g-sym 6.518134117126465 49.66925811767578 8.13516616821289
Metharme-7b-4bit-32g-trits 7.017086029052734 45.04129409790039 8.074845314025879
Metharme-7b-4bit-ts-128g-sym-trits 6.756448745727539 51.510311126708984 8.292160987854004
Metharme-7b-4bit-ts-128g-sym 6.819341659545898 55.28740310668945 8.377721786499023
Metharme-7b-4bit-ts-128g-trits 7.350858688354492 44.25314712524414 8.274221420288086
Metharme-7b-4bit-ts-32g-sym-trits 6.82415771484375 48.82029724121094 8.090987205505371
Metharme-7b-4bit-ts-32g-sym 6.387663841247559 48.07244110107422 8.173730850219727
Metharme-7b-4bit-ts-32g-trits 6.632943153381348 47.973228454589844 8.013848304748535
Metharme-7b-4bit-ts-sym-trits 7.599876403808594 55.75454330444336 8.932201385498047
Metharme-7b-4bit-ts-sym 7.270263671875 54.35262680053711 8.787986755371094
Metharme-7b-4bit-ts-trits 7.2306809425354 66.78710174560547 8.879831314086914
Metharme-ao-sym-trits-damp0.1 6.329384803771973 48.06882858276367 7.959168910980225
Metharme-ao-sym-trits 6.390106678009033 48.15375900268555 7.9804582595825195
Metharme-ao-sym 6.477842807769775 48.53507614135742 7.993765354156494
Metharme-ao-trits-damp0.1 6.163661956787109 46.50249099731445 7.877425193786621
Metharme-ao-trits 6.283935546875 46.57590103149414 7.907411575317383
Metharme-ao-ts-damp0.1 6.184001445770264 46.17180633544922 7.880400657653809
Metharme-ao-ts-sym-damp0.1 6.210323333740234 47.66789245605469 7.952476978302002
Metharme-ao-ts-sym-trits-damp0.1 6.232576370239258 48.081459045410156 7.95023250579834
Metharme-ao-ts-sym-trits 6.471063137054443 49.650611877441406 7.969552040100098
Metharme-ao-ts-sym 6.460526943206787 47.190460205078125 7.9732160568237305
Metharme-ao-ts-trits-damp0.1 6.172733783721924 45.19890213012695 7.872506141662598
Metharme-ao-ts-trits 6.310682773590088 46.4483757019043 7.898126602172852
Metharme-ao-ts 6.329496383666992 46.88129806518555 7.910323143005371
Metharme-sym-damp0.1 6.896804332733154 57.4250602722168 8.703770637512207
Metharme-sym-trits-damp0.1 7.075908184051514 59.13897705078125 8.919178009033203
Metharme-sym-trits 7.494253635406494 63.320709228515625 8.969240188598633
Metharme-sym 7.585672855377197 61.01168441772461 9.032520294189453
Metharme-trits 7.832409858703613 55.383026123046875 8.806737899780273