parameters guide
samplers guide
model generation
role play settings
quant selection
arm quants
iq quants vs q quants
optimal model setting
gibberish fixes
coherence
instructing following
quality generation
chat settings
quality settings
llamacpp server
llamacpp
lmstudio
sillytavern
koboldcpp
backyard
ollama
model generation steering
steering
model generation fixes
text generation webui
ggufs
exl2
full precision
quants
imatrix
neo imatrix
Update README.md
Browse files
README.md
CHANGED
@@ -36,7 +36,7 @@ tags:
|
|
36 |
|
37 |
<h3>Maximizing Model Performance for All Quants Types And Full-Precision using Samplers, Advance Samplers and Parameters Guide</h3>
|
38 |
|
39 |
-
(Updated: "INDEX", and added "Generation Steering" section)
|
40 |
|
41 |
This document includes detailed information, references, and notes for general parameters, samplers and
|
42 |
advanced samplers to get the most out of your model's abilities including notes / settings for the most popular AI/LLM app in use (LLAMACPP, KoboldCPP, Text-Generation-WebUI, LMStudio, Sillytavern, Ollama and others).
|
@@ -124,7 +124,8 @@ SOURCE FILES for my Models / APPS to Run LLMs / AIs:
|
|
124 |
- TEXT-GENERATION-WEBUI
|
125 |
- KOBOLDCPP
|
126 |
- SILLYTAVERN
|
127 |
-
- Lmstudio, Ollama, Llamacpp, and OTHER PROGRAMS
|
|
|
128 |
|
129 |
TESTING / Default / Generation Example PARAMETERS AND SAMPLERS
|
130 |
- Basic settings suggested for general model operation.
|
@@ -438,7 +439,7 @@ In section 1 a,b, and c, below are all the LLAMA_CPP parameters and samplers.
|
|
438 |
|
439 |
I have added notes below each one for adjustment / enhancement(s) for specific use cases.
|
440 |
|
441 |
-
TEXT-GENERATION-WEBUI
|
442 |
|
443 |
In section 2, will be additional samplers, which become available when using "llamacpp_HF" loader in https://github.com/oobabooga/text-generation-webui
|
444 |
AND/OR https://github.com/LostRuins/koboldcpp ("KOBOLDCPP").
|
@@ -449,7 +450,7 @@ The "llamacpp_HF" (for "text-generation-webui") only requires the GGUF you want
|
|
449 |
|
450 |
This allows access to very advanced samplers in addition to all the parameters / samplers here.
|
451 |
|
452 |
-
KOBOLDCPP
|
453 |
|
454 |
Note that https://github.com/LostRuins/koboldcpp also allows access to all LLAMACPP parameters/samplers too as well as additional advanced samplers too.
|
455 |
|
@@ -457,7 +458,7 @@ You can use almost all parameters, samplers and advanced samplers using "KOBOLDC
|
|
457 |
|
458 |
Note: This program has one of the newest samplers called "Anti-slop" which allows phrase/word banning at the generation level.
|
459 |
|
460 |
-
SILLYTAVERN
|
461 |
|
462 |
Note that https://github.com/SillyTavern/SillyTavern also allows access to all LLAMACPP parameters/samplers too as well as additional advanced samplers too.
|
463 |
|
@@ -480,16 +481,12 @@ Currently, at time of this writing, connecting Silly Tavern via KoboldCPP or Tex
|
|
480 |
|
481 |
However for some, connecting to Lmstudio, LlamaCPP, or Ollama may be preferred.
|
482 |
|
483 |
-
NOTE:
|
484 |
-
|
485 |
-
It appears that Silly Tavern also supports "DRY" and "XTC" too ; but it is not yet in the documentation at the time of writing.
|
486 |
-
|
487 |
You may also want to check out how to connect SillyTavern to local AI "apps" running on your pc here:
|
488 |
|
489 |
https://docs.sillytavern.app/usage/api-connections/
|
490 |
|
491 |
|
492 |
-
OTHER PROGRAMS
|
493 |
|
494 |
Other programs like https://www.LMStudio.ai allows access to most of STANDARD samplers, where as others (llamacpp only here) you may need to add to the json file(s) for a model and/or template preset.
|
495 |
|
@@ -515,6 +512,29 @@ Most AI/LLM apps operate on Windows, Mac, and Linux.
|
|
515 |
|
516 |
Mobile devices (and O/S) are in many cases also supported.
|
517 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
518 |
|
519 |
---
|
520 |
|
@@ -552,6 +572,12 @@ You should set these as noted first. I would say these are the minimum settings
|
|
552 |
|
553 |
Note for Class 3/Class 4 models settings/samplers (discussed below) "repeat-last-n" is a CRITICAL setting.
|
554 |
|
|
|
|
|
|
|
|
|
|
|
|
|
555 |
|
556 |
---
|
557 |
|
@@ -799,12 +825,22 @@ Likewise there may be some "name variation(s)" - in other LLM/AI apps - this is
|
|
799 |
|
800 |
</small>
|
801 |
|
802 |
-
CLASS 3/4
|
803 |
|
804 |
If you are using a class 3 or class 4 model for use case(s) such as role play, multi-turn, chat etc etc, it is suggested to activate / set all samplers for class 3 but may be required for class 4 models.
|
805 |
|
806 |
Likewise for fine control of a class 3/4 via "DRY" and "Quadratic" samplers is detailed below. These allow you to dial up or dial down the model's raw power directly.
|
807 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
808 |
MICROSTAT Sampler - IMPORTANT:
|
809 |
|
810 |
Make sure to review MIROSTAT sampler settings below, due to behaviour of this specific sampler / affect on parameters/other samplers which varies from app to app too.
|
@@ -855,6 +891,10 @@ Too much temp can affect instruction following in some cases and sometimes not e
|
|
855 |
|
856 |
Newer model archs (L3,L3.1,L3.2, Mistral Nemo, Gemma2 etc) many times NEED more temp (1+) to get their best generations.
|
857 |
|
|
|
|
|
|
|
|
|
858 |
<B>top-p</B>
|
859 |
|
860 |
top-p sampling (default: 0.9, 1.0 = disabled)
|
@@ -937,6 +977,12 @@ This setting also works in conjunction with all other "rep pens" below.
|
|
937 |
|
938 |
This parameter is the "RANGE" of tokens looked at for the samplers directly below.
|
939 |
|
|
|
|
|
|
|
|
|
|
|
|
|
940 |
<B>SECONDARIES:</B>
|
941 |
|
942 |
<B>repeat-penalty</B>
|
|
|
36 |
|
37 |
<h3>Maximizing Model Performance for All Quants Types And Full-Precision using Samplers, Advance Samplers and Parameters Guide</h3>
|
38 |
|
39 |
+
(Updated: "INDEX", and added "Generation Steering" section ; notes on Roleplay/Simulation added)
|
40 |
|
41 |
This document includes detailed information, references, and notes for general parameters, samplers and
|
42 |
advanced samplers to get the most out of your model's abilities including notes / settings for the most popular AI/LLM app in use (LLAMACPP, KoboldCPP, Text-Generation-WebUI, LMStudio, Sillytavern, Ollama and others).
|
|
|
124 |
- TEXT-GENERATION-WEBUI
|
125 |
- KOBOLDCPP
|
126 |
- SILLYTAVERN
|
127 |
+
- Lmstudio, Ollama, Llamacpp, Backyard, and OTHER PROGRAMS
|
128 |
+
- Roleplay and Simulation Programs/Notes on models.
|
129 |
|
130 |
TESTING / Default / Generation Example PARAMETERS AND SAMPLERS
|
131 |
- Basic settings suggested for general model operation.
|
|
|
439 |
|
440 |
I have added notes below each one for adjustment / enhancement(s) for specific use cases.
|
441 |
|
442 |
+
<B>TEXT-GENERATION-WEBUI</B>
|
443 |
|
444 |
In section 2, will be additional samplers, which become available when using "llamacpp_HF" loader in https://github.com/oobabooga/text-generation-webui
|
445 |
AND/OR https://github.com/LostRuins/koboldcpp ("KOBOLDCPP").
|
|
|
450 |
|
451 |
This allows access to very advanced samplers in addition to all the parameters / samplers here.
|
452 |
|
453 |
+
<B>KOBOLDCPP:</B>
|
454 |
|
455 |
Note that https://github.com/LostRuins/koboldcpp also allows access to all LLAMACPP parameters/samplers too as well as additional advanced samplers too.
|
456 |
|
|
|
458 |
|
459 |
Note: This program has one of the newest samplers called "Anti-slop" which allows phrase/word banning at the generation level.
|
460 |
|
461 |
+
<B>SILLYTAVERN:</B>
|
462 |
|
463 |
Note that https://github.com/SillyTavern/SillyTavern also allows access to all LLAMACPP parameters/samplers too as well as additional advanced samplers too.
|
464 |
|
|
|
481 |
|
482 |
However for some, connecting to Lmstudio, LlamaCPP, or Ollama may be preferred.
|
483 |
|
|
|
|
|
|
|
|
|
484 |
You may also want to check out how to connect SillyTavern to local AI "apps" running on your pc here:
|
485 |
|
486 |
https://docs.sillytavern.app/usage/api-connections/
|
487 |
|
488 |
|
489 |
+
<B>Lmstudio, Ollama, Llamacpp, and OTHER PROGRAMS</B>
|
490 |
|
491 |
Other programs like https://www.LMStudio.ai allows access to most of STANDARD samplers, where as others (llamacpp only here) you may need to add to the json file(s) for a model and/or template preset.
|
492 |
|
|
|
512 |
|
513 |
Mobile devices (and O/S) are in many cases also supported.
|
514 |
|
515 |
+
<B>Roleplay and Simulation Programs/Notes on models.</B>
|
516 |
+
|
517 |
+
Text Generation Webui, KoboldCPP, and Silly Tavern (and AI/LLM apps connected via Silly Tavern) can all do roleplay / simulation AS WELL as "chat" and other creative activities.
|
518 |
+
|
519 |
+
LMStudio (the app here directly), Ollama and other LLM/AI apps are for general usage, however they can be connected to Silly Tavern via API too.
|
520 |
+
|
521 |
+
Backyard ( https://backyard.ai/ ) is software that is dedicated primarily to Roleplay / Simulation, however it can not be (at time of this writing) connected via API to Silly Tavern at this time.
|
522 |
+
|
523 |
+
If you are using Backyard app, see special notes for "roleplay / simulation" and where applicable, "BACKYARD APP" for specific notes on using these app.
|
524 |
+
|
525 |
+
Models that are Class 3/4 :
|
526 |
+
|
527 |
+
Some of my models that are rated Class 3 or 4 maybe a little more challenging to operate with roleplay, especially if you can not access / control certain samplers.
|
528 |
+
|
529 |
+
How to handle this issue is addressed in "Generational Steering" section (you control it) as well as Quick Reference, and Detailed Parameters, Samplers and Advanced Samplers Sections (automated control).
|
530 |
+
|
531 |
+
Also, some of my models are available in multiple "classes", IE Dark Planet, and Grand Gutenberg.
|
532 |
+
|
533 |
+
In these cases, Dark Planet 8B versions and Grand Gutenberg 12B ("Darkness" / "Madness") are class 1 - any use case, including role play and simulation.
|
534 |
+
|
535 |
+
Likewise Darkest Planet 16.5B and Grand Gutenberg 23/23.5B are class 3 - great at roleplay/simulation, but need a bit more steering and/or parameter/samplers adjustments to work flawlessly for this use case.
|
536 |
+
|
537 |
+
Note: Dark Planet 8B (class 1) is also a compressed version of Grand Horror 16B (a full on class 4)
|
538 |
|
539 |
---
|
540 |
|
|
|
572 |
|
573 |
Note for Class 3/Class 4 models settings/samplers (discussed below) "repeat-last-n" is a CRITICAL setting.
|
574 |
|
575 |
+
BACKYARD APP:
|
576 |
+
|
577 |
+
In "Backyard" app, "repetition_penalty_range" is called "Repeat Penalty Tokens" (set on the "character card").
|
578 |
+
|
579 |
+
For class 3/4 models (if using with Backyard app), set this to 64 OR LESS.
|
580 |
+
|
581 |
|
582 |
---
|
583 |
|
|
|
825 |
|
826 |
</small>
|
827 |
|
828 |
+
CLASS 3/4 MODELS:
|
829 |
|
830 |
If you are using a class 3 or class 4 model for use case(s) such as role play, multi-turn, chat etc etc, it is suggested to activate / set all samplers for class 3 but may be required for class 4 models.
|
831 |
|
832 |
Likewise for fine control of a class 3/4 via "DRY" and "Quadratic" samplers is detailed below. These allow you to dial up or dial down the model's raw power directly.
|
833 |
|
834 |
+
ROLEPLAY / SIMULATION NOTES:
|
835 |
+
|
836 |
+
If you are using a model (regardless of "class") for these uses cases, you may need to LOWER "temp" to get better instruction following.
|
837 |
+
|
838 |
+
Instruction following issues can cascade over the "adventure" if the temp is set too high for the specific model(s) you are using.
|
839 |
+
|
840 |
+
Likewise you may want to set MAXIMUM output tokens (a hard limit how much the model can output) to much lower values such as 128 to 300.
|
841 |
+
|
842 |
+
(This will assist with steering, and stop the model from endlessly "yapping")
|
843 |
+
|
844 |
MICROSTAT Sampler - IMPORTANT:
|
845 |
|
846 |
Make sure to review MIROSTAT sampler settings below, due to behaviour of this specific sampler / affect on parameters/other samplers which varies from app to app too.
|
|
|
891 |
|
892 |
Newer model archs (L3,L3.1,L3.2, Mistral Nemo, Gemma2 etc) many times NEED more temp (1+) to get their best generations.
|
893 |
|
894 |
+
ROLEPLAY / SIMULATION NOTE:
|
895 |
+
|
896 |
+
If you are using a model (regardless of "class") for these uses cases, you may need to LOWER temp to get better instruction following.
|
897 |
+
|
898 |
<B>top-p</B>
|
899 |
|
900 |
top-p sampling (default: 0.9, 1.0 = disabled)
|
|
|
977 |
|
978 |
This parameter is the "RANGE" of tokens looked at for the samplers directly below.
|
979 |
|
980 |
+
BACKYARD APP:
|
981 |
+
|
982 |
+
In "Backyard" app, "repetition_penalty_range" is called "Repeat Penalty Tokens" (set on the "character card").
|
983 |
+
|
984 |
+
For class 3/4 models (if using with Backyard app), set this to 64 OR LESS.
|
985 |
+
|
986 |
<B>SECONDARIES:</B>
|
987 |
|
988 |
<B>repeat-penalty</B>
|