Greetings! I've decided to share some insight that I've accumulated over the few years I've been toying around with LLMs, and the intricacies of how to potentially make them run better for creative writing or roleplay as the focus, but it might also help with technical jobs too.
These might not be applicable with every model or user case, nor would it guarantee the best possible response with every single swipe, but it should help increase the odds of getting better mileage out of your model and experience, even if slightly, and help you avoid some bad or misled advice, which I personally have had to put up with. Some of this will be retreading old ground if you are already privy, but I will try to include less obvious stuff as well. Remember, I still consider myself a novice in some areas, and am always open to improvement.
This list will probably be updated periodically.
1: Instruct Formatting:
1-1 What is the Instruct Template?
The Instruct Template/Format is probably the most important aspect when it comes to getting a model to work properly, as it is what encloses the training data with tokens that were used for the model, and your chat with said model. Some of them are used in a more general sense and are not brand specific, such as ChatML or Alpaca, while others stick to said brand, like Llama3 Instruct or Mistral Instruct. However not all models that are brand specific with their formatting will be trained with their own personal template.
Its important to find out what format/template a model uses before booting it up, and you can usually check to see which it is on the model page. If a format isn't directly listed on said page, then there is ways to check internally with the local files. Each model has a tokenizer_config file, and sometimes even a special_tokens file, inside the main folder. As an example of what to look for, If you see something like a Mistral brand model that has im_start/im_end inside those files, then chances are that the person who finetuned it used ChatML tokens in their training data. Familiarizing yourself with the popular tokens used in training will help you navigate models better internally, especially if a creator forgets to post a readme on how it's suppose to function.
1-2 Is there any reason not to use the prescribed format/template?
Sticking to the prescribed format will give your model better odds of getting things correct, or even better prose quality. But there are some small benefits when straying from the model's original format, such as supposedly being less censored. However the trade-off when it comes to maximizing a model's intelligence is never really worth it, and there are better ways to get uncensored responses with better prompting, or even tricking the model by editing their response slightly and continuing from there.
From what I've found when testing models, if someone finetunes a model over the company's official Instruct focused model, instead of a base model, and doesn't use the underlining format that it was made with (such as ChatML over Mistral's 22B model as an example) then performance dips will kick in, giving less optimal responses then if it was instead using a unified format.
This does not factor other occurrences of poor performance or context degradation when choosing to train on top of official Instruct models which may occur, but if it uses the correct format, and/or is trained with DPO or one of its variance (this one is more anecdotal, but DPO/ORPO/Whatever-O seems to be a more stable method when it comes to training on top of per-existing Instruct models) then the model will perform better overall.
1-3 What about models that list multiple formats/templates?
This one is more due to model merging or choosing to forgo an Instruct model's format in training, although some people will choose to train their models like this, for whatever reason. In such an instance, you kinda just have to pick one and see what works best, but the merging of formats, and possibly even models, might provide interesting results, but only if its agreeable with the clutter on how you prompt it yourself. What do I mean by this? Well, perhaps its better if I give you a couple anecdotes on how this might work in practice...
Nous-Capybara-limarpv3-34B is an older model at this point, but it has a unique feature that many models don't seem to implement; a Message Length Modifier. By adding small/medium/long at the end of the Assistant's Message Prefix, it will allow you to control how long the Bot's response is, which can be useful in curbing rambling, or enforcing more detail. Since Capybara, the underling model, uses the Vicuna format, its prompt typically looks like this:
System:
User:
Assistant:
Meanwhile, the limarpv3 lora, which has the Message Length Modifier, was used on top of Capybara and chose to use Alpaca as its format:
### Instruction:
### Input:
### Response: (length = short/medium/long/etc)
Seems to be quite different, right? Well, it is, but we can also combine these two formats in a meaningful way and actually see tangible results. When using Nous-Capybara-limarpv3-34B with its underling Vicuna format and the Message Length Modifier together, the results don't come together, and you have basically 0 control on its length:
System:
User:
Assistant: (length = short/medium/long/etc)
The above example with Vicuna doesn't seem to work. However, by adding triple hashes to it, the modifier actually will take effect, making the messages shorter or longer on average depending on how you prompt it.
### System:
### User:
### Assistant: (length = short/medium/long/etc)
This is an example of where both formats can work together in a meaningful way.
Another example is merging a Vicuna model with a ChatML one and incorporating the stop tokens from it, like with RP-Stew-v4. For reference, ChatML looks like this:
<|im_start|>system
System prompt<|im_end|>
<|im_start|>user
User prompt<|im_end|>
<|im_start|>assistant
Bot response<|im_end|>
One thing to note is that, unlike Alpaca, the ChatML template has System/User/Assistant inside it, making it vaguely similar to Vicuna. Vicuna itself doesn't have stop tokens, but if we add them like so:
SYSTEM: system prompt<|end|>
USER: user prompt<|end|>
ASSISTANT: assistant output<|end|>
Then it will actually help prevent RP-Stew from rambling or repeating itself within the same message, and also lowering the chances of your bot speaking as the user. When merging models I find it best to keep to one format in order to keep its performance high, but there can be rare cases where mixing them could work.
1-4 Are stop tokens necessary?
In my opinion, models work best when it has stop tokens built into them. Like with RP-Stew, the decrease in repetitive message length was about 25~33% on average, give or take from what I remember, when these end tokens are added. That's one case where the usefulness is obvious. Formats that use stop tokens tend to be more stable on average when it comes to creative back-and-forths with the bot, since it gives it a structure that's easier for it to understand when to end things, and inform better on who is talking.
If you like your models to be unhinged and ramble on forever (aka; bad) then by all means, experiment by not using them. It might surprise you if you tweak it. But as like before, the intelligence hit is usually never worth it. Remember to make separate instances when experimenting with prompts, or be sure to put your tokens back in their original place. Otherwise you might end up with something dumb, like putting the stop token before the User in the User prefix.
2: Character/Chat Formatting
2-1 What is a Character Card?
Lets get the obvious thing out of the way. Character Cards are basically personas of, well, characters, be it from real life, an established franchise, or someone's OC, for the AI bot to impersonate and interact with. The layout of a Character Card is typically written in the form of a profile or portfolio, with different styles available for approaching the technical aspects of listing out what makes them unique.
2-2 What are the different styles of Character Cards?
Making a card isn't exactly a solved science, and the way its prompted could vary the outcome between different model brands and model sizes. However, there are a few that are popular among the community that have gained traction.
One way to approach it is a simply writing out the character's persona like you would in a novel/book, using natural prose to describe their background and appearance. Though this method would require a deft hand/mind to make sure it flows well and doesn't repeat too much with specific keywords, and might be a bit harder compered to some of the other styles if you are just starting out. More useful for pure writers, probably.
Another is doing a list format, where every feature is placed out categorically and sufficiently. There are different ways of doing this as well, like markdown, wiki style, or the community made W++, just to name a few.
Some use parentheses or brackets to enclose each section, some use dashes for separate listings, some bold sections with hashes or double asterisks, or some none of the above.
I haven't found which one is objectively the best when it comes to a specific format, although W++ is probably the worst of the bunch when it comes to stabilization, with Wiki Style taking second worse just because of it being bloat dumped from said wiki. There could be a myriad of reasons why W++ might not be considered as much anymore, but my best guess is, since the format is non-standard in most model's training data, it has less to pull from in its reasoning.
My current recommendation is just to use some mixture of lists and regular prose, with a traditional list when it comes to appearance and traits, and using normal writing for background and speech. Though you should be mindful of what perspective you prompt the card beforehand.
2-3 What writing perspectives should I consider before making a card?
This one is probably more definitive and easier to wrap your head around then choosing a specific listing style. First, we must discuss what perspective to write your card and example messages for the bot in: I, You, They. This demonstrates perspective the card is written in - First-person, Second-person, Third-person - and will have noticeable effects on the bot's output. Even cards that are purely list based will still incorporate some form of character perspective, and some are better then others for certain tasks.
"I" format has the entire card written from the characters perspective, listing things out as if they themselves made it. Useful if you want your bots to act slightly more individualized for one-on-one chats, but requires more thought put into the word choices in order to make sure it is accurate to the way they talk/interact. Most common way people talk online. Keywords: I, my, mine.
"You" format is telling the bot what they are from your perspective, and is typically the format used in system prompts and technical AI training, but has less outside example data like with "I" in chats/writing, and is less personable as well. Keywords: You, your, you're.
"They" format is the birds-eye view approach commonly found in storytelling. Lots of novel examples in training data. Best for creative writers, and works better in group chats to avoid confusion for the AI on who is/was talking. Keywords: They, their, she/he/its.
In essence, LLMs are prediction based machines, and the way words are chosen or structured will determine the next probable outcome. Do you want a personable one-on-one chat with your bots? Try "I" as your template. Want a creative writer that will keep track of multiple characters? Use "They" as your format. Want the worst of both worlds, but might be better at technical LLM jobs? Choose "You" format.
This reasoning also carries over to the chats themselves and how you interact with the bots, though you'd have to use a mixture with "You" format specifically, and that's another reason it might not be as good comparatively speaking, since it will be using two or more styles at once. But there is more to consider still, such as whether to use quotes or asterisks.
2-4 Should I use quotes or asterisks as the defining separator in the chat?
Now we must move on to another aspect to consider before creating a character card, and the way you warp the words inside: To use "quotes with speech" and plain text with actions, or plain text with speech and asterisks with actions. These two formats are fundamentally opposed with one another, and will draw from separate sources in the LLMs training data, however much that is, due to their predictive nature.
Quote format is the dominant storytelling format, and will have better prose on average. If your character or archetype originated from literature, or is heavily used in said literature, then wrapping the dialogue in quotes will get you better results.
Asterisk format is much more niche in comparison, mostly used in RP servers - and not all RP servers will opt for this format either - and brief text chats. If you want your experience to feel more like a texting session, then this one might be for you.
Mixing these two - "Like so" I said - however, is not advised, as it will eat up extra tokens for no real benefit. No formats that I know of use this in typical training data, and if it does, is extremely rare. Only use if you want to waste tokens/context on word flair.
2-5 What combination would you recommend?
Third-person with quotes for creative writers and group RP chats. First-person with asterisks for simple one-on-one texting chats.
Well hello, hello! This is the third part of my Model Tips & Tricks series, where I will be talking about ways to both create your character cards, sources to use in helping with development, and just general fun stuff I've found along the way that might be interesting or neat for those not already aware.
Like before, some things will be retreading old ground for veterans in this field, but I will try to incorporate less obvious advice along the way as well. I also don't consider myself an expert, and am always open to new ideas and advice for those willing to share.
3: Making a Character Card
3-1 What are some basic sources I should know of before making a character?
While going in raw when making a character card, either from scratch or from an existing IP, could be fun as an exercise in writing or formatting, its not always practical to do so, and there are a few websites that are easy enough to navigate your way around this to make the process easier. Of course you should probably choose how you would format the card before, like with a listing format in the vein of something like JED+, which was discussed in the last point.
The first obvious one, if you are using a per-existing character or archetype, is a Wiki or index. Shocking, I know. But its still worth bringing up for beginners. Series or archetypal Wikis can help immensely in gathering info about how your character works in a general sense, and perhaps even bring in new info you wouldn't consider when first starting out. For per-existing characters, just visiting one of the Wikis dedicated to them and dumping it into an assistant to summarize key points could be enough if you just want a base to work with, but you should always check yourself for anything you deem essential for your chat/RP experience in said pages.
For those that are original in origin, or just too niche for the AI to know what series they hail from, you could always visit separate Wikis or archetypal resources. Is the character inspired by someone else's idea, like some masked vigilante hero who stops crime? Then visiting a "Marvel" or "DC" Wiki or Pedia page that is similar in nature could help with minute details. Say you want to make an elf princess? Maybe the "Zelda" Wiki or Pedia could help. Of course those are more specific cases. There are more general outliers too, like if they are a mermaid or harpy you could try the "Monster Girl Encyclopedia", or if they are an archetype commonly found in TV or Anime you could use "TV Tropes" or "Dere Types Wiki" for ideas. "WebMD" if they have a health or mental condition perhaps, but I'm not a doctor, so ehh...
I could keep listing sites that might be good for data on archetypes endlessly, but you probably get the picture at this point: If they are based on something else, then there is probably a Wiki or general index to pull ideas from. The next two big ones I'd like to redirect towards are more for helping with specific listings in the appearance and personality sections of you character card.
3-2 What site should I know about before describing my character's appearance?
For appearance, visiting an art site like "Danbooru" could help you with picking certain tags for the AI model to read from. Just pick your character, or a character that has a similar build or outfit in mind, and just go from there to help figure out how you want the AI to present your character. Useful if you have a certain outfit or hairstyle in mind, but can't quite figure out what it is called exactly. Not all images will include everything about the clothes or style, so it is important to browse around a bit if you can't find a certain tag you are looking for. While a Wiki might help with this too, Danbooru can get into more specifics that might be lost on the page. There's also that other site, which is after 33 and before 35, which has a similar structure if you are really desperate for tags of other things.
But enough of that for now, how about we move on to the personality section.
3-3 What site should I know about before describing my character's personality?
For personality, the "Personality Database", while not always accurate, can help give you an idea for how your character might act or present themselves. This is one of those sites I had no idea or cared about beforehand (and still don't to a degree in terms of real life applications) or before LLMs became a thing. Like with Danbooru, even if your character is an OC, just choosing a different character who seems similar to yours might help shape them. Not all of the models used for describing a character's personality will be intrinsically known by an LLM, but there are a few that seem to be universal. However, this might require a bit more insight later on how to piece it all together.
The big ones used there that most LLMs will be able to figure out if asked are: Four Letter, or "MBTI" as its typically called, which is a a row of letters to denote stuff like extroversion vs introversion, intuition vs sensing, a thinker vs a feeler, and perceptive vs judging. Enneagram, which denotes a numbered type between 1 and 9, along with a secondary wing that acts as an extension of sorts. Temperament is 4 core traits that can be either solitary or combined with a secondary, like with the number typing. Alignment, which is a DnD classification if someone is Lawful or Chaotic, Good or Evil, or something in between with Neutral. And Zodiac, which is probably the most well known, and is usually in coloration with a character's birthday, although that isn't always the case. The others listed on that site are usually too niche, or require extra prompting to get right like with Instinctual Variant.
If you don't want to delve into these ideas as a standalone yourself, then just dropping those into an assistant bot like before and ask for a summery or keywords relating to the personality provided will help if you need to get your character to tick in a certain way.
There are some other factors you could consider as well, like Archetypes specifically again (tsundere, mad genius, spoiled princess, etc. or Jung specifics) and Tarot cards (there are so many articles online when it comes to tarot and zodiac readings that was probably fed into AI models) which are worth considering when asking an AI for a rundown on traits to add.
You could also combine both the compact personality before you asked the AI assistant, and the complex list it will spit out if you want to double up on traits and not be redundant in wording, which can help with the character's stability. We can probably move on to general findings now.
3-4 What general ideas are worth considering for my character card?
We can probably discuss some sub-sections which might be good to list out as a start.
"Backstory or Background" is one of the more pivotal, but also easy to grasp, section of the card. This helps give the bot a timeline to know how the character evolved before interacting with them, but also at what point of the story they are from if they come from an existing IP.
"Likes/Dislikes" are another easy one to understand. These will make it so your character will react in certain ways when confronted with them. Individually for both sections works, but you can also make subsections of these as well if they have multiple, like Food, Items, Games, Activities, Actions, Colors, Animals, and Traits, just to name a few. Another way to approach this is have tiers instead, for example a character could have this -Likes Highly: Pizza, Sausage, Mushrooms- But also -Likes Slightly: Pineapple- to denote some semblance of nuance with how they react and choose things.
"Goals/Fears" are a strong factor which can drive a character in certain ways, or avoid, or even maybe tackle as challenge to overcome later. Main and secondary goals/fears can also, again, help with some nuance.
"Quirks" are of course cool if you want to differentiate certain actions and situations.
"Skills/Stats" will help denote what a character is or isn't good at, although stats specifically should maybe be used in a more Adventure/RPG like scenario, though it can still be understood in a mundane sense too.
"Views" is similar to the personality section, but helps in different and more specific ways. This can be either their general view on things, how they perceive others characters or the user and their relationship with them, or more divisive stances like politics and religion.
"Speech/Mannerisms" Is probably the last noteworthy one, as this helps separate it from general quirks by themselves, and how they interact with others specifically, which can be used in conjunction with example messages inside the card.
3-5 Are example messages worth adding to a character card?
If you want your character to stick to a specific way of interacting with others, and help differentiate better in group chats for the AI, then I'd say yes. You could probably get away with just the starting message and those listings above if you want a simple chat, but I've found example messages, if detailed and tailored in the way you prefer for the chat/RP/writing session, will help immensely with getting certain results. Its one thing to list something for the bot to get a grasp of its persona, but having an actual example with all of the little nuances and formatting choices within said chat, will net you better results on average. Prose choice is one big factor in helping the bot along, like the flick of a tail, or the mechanical whirl of a piston arm, can help shape more fantastical characters of course, but subtle things for more grounded characters is of course good too.
Me personally, I like to have multiple example messages, say in the 3~7 range, and this is for two reasons. One is so the character can express multiple emotions and scenarios that would be relevant to them, and just having to cram it all inside one message might make it come across as schizo in structure, or become a big wall of text that could lead to bloat and/or bloat further messages. And the second is varying message length itself, in order to ensure the bot doesn't get comfortable in a certain range when interacting.