tags:
- llama
- alpaca
- vicuna
- uncensored
- merge
- mix
- airoboros
- openorca
- orcamini
- orca
- instruct
- mixtune
datasets:
- Open-Orca/OpenOrca
- anon8231489123/ShareGPT_Vicuna_unfiltered
- jondurbin/airoboros-uncensored
language:
- en
metrics:
- accuracy
pipeline_tag: text-generation
13B-Ouroboros
Ouroboros is an experimental model based on Meta's LLaMA [v1] 13B base model using a custom merging script that optimizes per-layer merging based on a given dataset. Ouroboros is optimized against the PTB text only validation dataset, scoring ~26.31 according to internal evaluation (6 samples, sequence length 1024; this testing is not empirical, it's a part of the random search algorithm). Testing, evaluating, and remixing this model is absolutely permissible and even encouraged (within the bounds of Meta's LLaMAv1 license agreement); the more feedback the better we can tune our process! 😊
When the mix tuning system has reached a certain point of maturity it will be released open source.
Composition:
Ouroboros is comprised of 40 layers [LLaMAv1 13B standard] mixed at optimized ratios VS the PTB dataset for lowest perplexity score. Listed below are the paired models and ratios merged per layer.
Tier One Merge:
13B-airoboros-gpt4-1.4 > 13B-orca_mini_v2
[0.22, 0.85, 0.89, 0.98, 0.3, 0.41, 0.71, 0.83, 0.32, 0.1, 0.44, 0.6, 0.53, 0.15, 0.86, 0.79, 0.93, 0.02, 0.19, 0.82, 0.01, 0.52, 0.07, 0.27, 0.73, 0.86, 0.08, 0.67, 0.42, 0.28, 0.37, 0.08, 0.95, 0.68, 0.45, 0.08, 0.7, 0.93, 0.96, 0.43]
13B-gpt4-x-alpaca > 13B-Vicuna-cocktail
[0.65, 0.94, 0.98, 0.87, 0.28, 0.64, 0.73, 0.7, 0.95, 0.89, 0.84, 0.9, 0.59, 0.92, 0.28, 0.61, 0.88, 0.73, 0.34, 0.85, 0.98, 0.05, 0.74, 0.92, 0.5, 0.78, 0.26, 0.4, 0.27, 0.65, 0.71, 0.7, 0.8, 0.93, 0.36, 0.03, 0.45, 0.39, 0.77, 0.06]
Tier Two Merge:
[13B-airoboros-gpt4-1.4 + 13B-orca_mini_v2] offspring > [13B-gpt4-x-alpaca + 13B-Vicuna-cocktail] offspring
[0.2, 0.83, 0.24, 0.03, 0.37, 0.62, 0.02, 0.82, 0.65, 0.63, 0.45, 0.65, 0.48, 0.45, 0.24, 0.76, 0.06, 0.31, 0.45, 0.86, 0.23, 0.99, 0.93, 0.84, 0.96, 0.53, 0.95, 0.32, 0.19, 0.06, 0.4, 0.08, 0.62, 0.4, 0.26, 0.12, 0.16, 0.91, 0.14, 0.0]
Result:
13B-Ouroboros, a model that seems uncensored and highly competent. So far only Alpaca instruction promting has been tested and seems to work solidly well.
Use:
Alpaca's instruct format can be used to do many things, including control of the terms of behavior between a user and a response from an agent in chat. Below is an example of a command injected into memory.
### Instruction:
Make Narrator function as a text based adventure game that responds with verbose, detailed, and creative descriptions of what happens next after Player's response.
Make Player function as the player input for Narrator's text based adventure game, controlling a character named (insert character name here, their short bio, and
whatever quest or other information to keep consistent in the interaction).
### Response:
{an empty new line here}
Language Models Used Credits:
13B-airoboros-gpt4-1.4 by jondurbin
https://huggingface.co/jondurbin/airoboros-13b-gpt4-1.4
13B-orca_mini_v2 by psmathur
https://huggingface.co/psmathur/orca_mini_v2_13b
13B-gpt4-x-alpaca by chavinlo
https://huggingface.co/chavinlo/gpt4-x-alpaca
13B-Vicuna-cocktail by reeducator
https://huggingface.co/reeducator/vicuna-13b-cocktail
Also thanks to Meta for LLaMA.
Each model and LoRA was hand picked and considered for what it could contribute to this ensemble. Thanks to each and every one of you for your incredible work developing some of the best things to come out of this community.