rubra-11b-h

Model description

This is a mistral based model trained on a lot of code and conversational data. The goal of this model was to retain the abilities of mistralai/Mistral-7B-Instruct-v0.2 while making it better at reasoning.

32k context window
Rope-theta = 1e6

Fine tuning instruct models has proven to be a challenge when the goal is to create a model that is not meant for a particular task. Specifically, fine tuning an instruct model often leads to destructive forgetting in the resulting model. This is why most fine tunes select a base (non-instruct) language model. The rubra-v0.1 series models aim to add desired traits like multi-turn chat capabilites while retaining knowledge found in instruct models

Chat format

Your prompt should be surrounded by [INST] and [/INST] tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.

E.g.

text = "<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
"[INST] Do you have mayonnaise recipes? [/INST]"

This format is available as a chat template via the apply_chat_template() method:

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained("sanjay920/rubra-11b-h")
tokenizer = AutoTokenizer.from_pretrained("sanjay920/rubra-11b-h")

messages = [
    {"role": "user", "content": "What is your favourite condiment?"},
    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
    {"role": "user", "content": "Do you have mayonnaise recipes?"}
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

GGUF

GGUF quants are available at sanjay920/rubra-11b-h-GGUF

Create a new discussion for other quant variant requests!

LM Studio

Set "Chat Preset" to Mistral Instruct

Training procedure

Block expansion and further training

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 1
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 12
total_train_batch_size: 12
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
num_epochs: 3.0
mixed_precision_training: Native AMP

Training results

{
    "epoch": 3.0,
    "train_loss": 0.5844329195557438,
    "train_runtime": 307371.8718,
    "train_samples_per_second": 0.246,
    "train_steps_per_second": 0.021
}

Framework versions

Transformers 4.38.2
Pytorch 2.2.1+cu121
Datasets 2.18.0
Tokenizers 0.15.2

Evaluation Results

########## Average ##########
                             score
model                             
gpt-4-0613                9.020312  
gpt-3.5-turbo-0125        7.970219
rubra-11b-h               7.800000
Mistral-7B-Instruct-v0.2  7.793750
Mistral-7B-Instruct-v0.1  6.721875
Llama-2-13b-chat          6.650000

Intended uses & limitations

This model is still under development. It hasn't been aligned for safety, so it may produce biased generations. It appears to produce thoughtful answers compared to other small LLMs. See below:

Examples

Needle in a haystack test:

The input text can be found in needle-in-a-haystack.txt

14,334 input tokens
"Random number is 23." inserted into the middle of the long document text. Asked the LLM what the random number is.

rubra-11b-h:

The random number is 23.

Correct

mistral-7b-instruct-v0.2:

The random number generated from this text is 23. However, it's important to note that the method used to generate this number from the text is not a secure or reliable one and should not be used for any purpose where security is a concern. This was just a fun exercise to extract a number from the text.

As for the content of the text, it discusses the idea of creating a technology hub or "silicon valley" by attracting the right people - specifically, rich people and nerds. The author argues that these two groups are the limiting reagents in the reaction that produces startups, as they are the only ones present when startups get started. The text also touches on the idea that universities can act as magnets for these people and that personality and youth are important factors in creating a successful technology hub.

Correct

hermes-2-pro:

 the<dummy00028>'1 Question1 question2<dummy00028> <dummy00015>..... in3<dummy00028>d<dummy00028>}<dummy00015>l kennis4<dummy00012>il<dummy00028>y<dummy00028>}2<dummy00028>-^{ a<dummy00028>}3<dummy00012>. a<dummy00028>}<dummy00015> toion<dummy00028>} people a a<dummy00028>}5<dummy00005>- a Question1<dummy00028>}y<dummy00028>}6<dummy00012>. a<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00012>,<dummy00015>l<dummy00028>},7<dummy00012>. <dummy00015>l Fuß in the<dummy00028>
 a a<dummy00028>'2 a<dummy00028>}4 ál a Question3<dummy00012>o kennis in if to more a<dummy00028> or a<dummy00028>'5<dummy00005> 1<dummy00028>}1<dummy00005>s<dummy00028>y<dummy00028>y<dummy00028>'6 Question9 Question1<dummy00028>'2 Questions<dummy00028>y biologie1<dummy00028>y<dummy00028>y<dummy00028>-3<dummy00028>'1 Question. a<dummy00028>* the<dummy00028>}<dummy00015>y Question a<dummy00028>4<dummy00012>,ll Question1 Question6<dummy00028>4<dummy00028>}2<dummy00028>}1<dummy00028>}<dummy00015>l<dummy00028>
 if a Questionia kennisa<dummy00005> <dummy00015> a<dummy00028>h<dummy00028>}2<dummy00028>a égal

Incorrect. Unusable output

sanjay920
/

rubra-11b-h