MaziyarPanahi

/

Meta-Llama-3-8B-Instruct-GGUF

like 75

we haven't seen anything strange, but then you can define your own template and not let Llama.cpp detects it automatically:

https://huggingface.co/MaziyarPanahi/Meta-Llama-3-8B-Instruct-GGUF/discussions/5

I found that when using the llama.cpp server with default settings, the model tends to answer xxx.assistant then start another round without a stop token, for example,

3: Have a nice day!assistant

<|im_start|>assistant
Hi!

This one fixed it https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct-GGUF/tree/main. So I believe there is something wrong in the template.

good point, I usually don't trust the default template specially for the new comers with new templates.

As far as I see, providing the template, it should work without any issue. Is that correct?

The model behaves much better when the stop token is fixed. You can do this yourself using gguf-set-metadata from the gguf package:

gguf-set-metadata Meta-Llama-3-8B-Instruct.Q8_0.gguf tokenizer.ggml.eos_token_id 128009
* Loading: Meta-Llama-3-8B-Instruct.Q8_0.gguf
* Preparing to change field 'tokenizer.ggml.eos_token_id' from 128001 to 128009
*** Warning *** Warning *** Warning **
* Changing fields in a GGUF file can make it unusable. Proceed at your own risk.
* Enter exactly YES if you are positive you want to proceed:
YES, I am sure> YES
* Field changed. Successful completion.

The instruct version has an end of turn token (<|eot_id|>) that is not present in the base model that it will generate and should be used as the stop token when following their instruct/chat format. If you only stop on <|end_of_text|> you will often get things like .assistant in your output (depending how your other settings are).

It seems like llama.cpp is maybe going to fix this in general when a L3 model is loaded or converted, not sure exactly where they are going to land on that though.

Thanks @terribleplan
I might do that fix this weekend, most people use template or LM Studio don't see that issue. It's only if you let Llama.cpp decide the template and stop strings. But I might do it and re-upload.

I went ahead and made the change and I am uploading them again. I realize that so many users workaround this in the application level, most don't even notice it. But there are some who just cannot go around it. I am re-uploading it for them. Hope it helps.

this works now perfectly:

../apps/fine-tuning/quantize/gguf/llama.cpp/main -m Meta-Llama-3-8B-Instruct.Q2_K.gguf -p "<|start_header_id|>user<|end_header_id|>\n\nBuilding a website can be done in 10 simple steps:\nStep 1:<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" -n 1024


<|start_header_id|>user<|end_header_id|>\n\nBuilding a website can be done in 10 simple steps:\nStep 1:<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nYou're talking about building a website! That's a great topic. Here's the rest of
 the text:

Building a website can be done in 10 simple steps:

Step 1: Define Your Purpose

* Determine the main purpose of your website
* Identify your target audience
* Decide what you want to achieve with your website

Step 2: Choose a Domain Name

* Research domain name options
* Check if the domain name is available
* Register your domain name

Step 3: Design Your Website

* Choose a website builder or CMS
* Create a wireframe of your website's structure
* Design your website's layout

Step 4: Develop Your Content

* Create high-quality content for your website
* Plan out your content in advance
* Create content calendar

Step 5: Create a Logo

* Design a logo that represents your brand
* Make sure your logo is simple and recognizable

Step 6: Build Your Website

* Choose a website builder or CMS
* Develop your website using the chosen platform
* Make sure your website is mobile-friendly

Step 7: Add Features and Functionality

* Add features and functionality to your website
* Make sure your website is user-friendly

Step 8: Test and Refine

* Test your website's functionality and features
* Make sure your website is bug-free
* Refine your website's design and features

Step 9: Launch Your Website

* Launch your website to the public
* Promote your website to your target audience
* Monitor your website's traffic and analytics

Step 10: Maintain and Update Your Website

* Keep your website up to date
* Make sure your website is secure and updated with the latest features
* Keep your website relevant to your target audience

That's it! Building a website can be a simple and straightforward process if you follow these steps.<|eot_id|> [end of text]

For Ollama (0.1.32) it works better (generation stops after assistant turn) if before conversion you change in tokenizer_config.json from
"eos_token": "<|end_of_text|>",
to
"eos_token": "<|eot_id|>",