Having some issues with using this
Hello!
I found your recent post on Reddit linking to this model. It sounds promising and I really want to use it! Been messing around with it a bit today, but I can't seem to get it to work as expected? I'm pretty new to running LLMs locally, so keep that in mind :)
My prompt was basically telling it to write a detailed story. I gave it a bunch of guidance and loose ideas / descriptions for what should happen in it. I had a somewhat clear vision for a few things that should happen at the start and in the middle, but not the end. So I basically want it to fill in the blanks, not create a completely new story out of nothing.
It sort of does this, but it deviates way too much from the guidance I gave it. The general idea / outline of the story is there, but it comes up with completely different stuff than it should. It also skips over specific dialogue I told it to include, or changes it so it becomes unrecognizable, with the characters saying something completely different.
How can I get it to follow my instructions more closely?
I should mention that I'm using the "Dusk_Rainbow-Q6_K" quant from here: https://huggingface.co/bartowski/Dusk_Rainbow-GGUF
Running it in the Oobabooga WebUI, in "instruct" mode. Tried using both "Debug Deterministic" and "min_p" preset configurations as you recommended on the model card, didn't notice a difference.
Also, something I couldn't quite figure out was the "Model instruction template" part on the model card. You have two templates there, ChatML and Llama-3-Instruct. I tried copying and pasting the ChatML one into Parameters > Instruction template in Oobabooga, but then I don't get a response, only a "IndexError: list index out of range" error in the console. Can you clear that up a bit for a beginner like me? :)
Also, does this have to be deterministic? What if I wanted to generate a different output from the same prompt?
I know this was a lot of text, but I would appreciate it a lot if you could help out!
- Getting the Model to Follow Instructions More Closely
To ensure the model follows your instructions more closely, you can try the following strategies:
Prompt Engineering: Be very explicit and detailed in your instructions. Sometimes, breaking down the instructions into smaller, more manageable parts can help.
Adjusting Parameters: Experiment with different settings for parameters like temperature, top_p, and repetition_penalty. Lowering the temperature can make the model more deterministic and less likely to deviate from your instructions.
Context Length: Ensure that your prompt fits within the model's context length. If the prompt is too long, the model might not be able to process all the instructions properly.
- Using the Instruction Template
The "Model instruction template" is a way to format your input so that the model understands it better. Here's how you can use it:
ChatML Template: This template is designed for a specific format. If you're getting an "IndexError: list index out of range" error, it might be due to incorrect formatting. Make sure you're using the template exactly as provided.
Here's an example of how you might use the ChatML template:
<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Write a detailed story with the following guidelines: [Your guidelines here]
<|im_end|>
<|im_start|>assistant
Llama-3-Instruct Template: This is another format you can try. Make sure to follow the format exactly as provided on the model card.
Deterministic vs. Non-Deterministic Output
Deterministic Output: If you want the model to generate the same output every time for the same prompt, you should use a low temperature (e.g., 0.1 or lower) and set a fixed random seed.
Non-Deterministic Output: If you want different outputs for the same prompt, you can increase the temperature (e.g., 0.7 or higher) and avoid setting a fixed random seed.Additional Tips
Experiment with Presets: You mentioned trying "Debug Deterministic" and "min_p" presets. Try other presets as well to see if they give better results.
Feedback Loop: Sometimes, providing feedback to the model in the form of additional instructions can help guide it better. For example, if the model deviates, you can add a follow-up instruction to correct it.
Example Prompt
Here's an example of how you might structure your prompt using the ChatML template:
<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Write a detailed story with the following guidelines:
- The story should start with a character named Alex waking up in a strange place.
- Alex should meet a mysterious figure who gives them a cryptic message.
- Include the following dialogue: "You must find the hidden key before midnight."
- The story should end with a cliffhanger.
<|im_end|>
<|im_start|>assistant