Has anybody gotten this example to work for converting string data into valid JSON?
There is an example here: https://www.promptingguide.ai/models/mistral-7b#chat-template-for-mistral-7b-instruct
In the example they show how string data such as the below...
name: John
lastname: Smith
address: #1 Samuel St.
Can be converted into JSON as below...
{
"address": "#1 Samuel St.",
"lastname": "Smith",
"name": "John"
}
I am wondering if maybe it is not working because I am using BitsAndBytes in 4-bit mode so maybe I am losing precision?
Below is my generation_config...
generation_config = model.generation_config
generation_config.max_new_tokens = 1024
generation_config.do_sample = True
generation_config.temperature = 0.1
generation_config.top_p = 0.1
generation_config.repetition_penalty = 1.2
generation_config.no_repeat_ngram_size = 2
generation_config.num_beams = 1
generation_config.num_return_sequences = 1
generation_config.pad_token_id = tokenizer.eos_token_id
generation_config.eos_token_id = tokenizer.eos_token_id
I have tried using apply_chat_template() as well as formatting the strings myself, but the output is always extremely squirrely and I am actually getting better results with v0.1 than v0.2
And I just tested using the fireworks.ai instance mentioned in the tutorial and it works perfectly well irrespeoctive of the generations ettings so it has to be the BitsAndBytes config that is the culprit... will be able to verify once I get access to a better VM again!
