TheBloke

/

starchat-beta-GGML

like 34

Confirming that it does not work on gpt4all. (ggml-starchat-beta.ggmlv3.q4_0.bin)

I didn't claim it would work with gpt4all, I don't test in that. They have their own private ecosystem and I don't really look into it.

I though it would work with ParisNeo/GPT4All-Ui using the ctransformers backend, because I thought ctransformers should support this model type.

Perhaps @ParisNeo or @marella could comment on compatibility of BigCoder models in their respective software?

The following works with ctransformers:

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained("TheBloke/starchat-beta-GGML")

print(llm(prompt, stop="<|end|>"))

Here is an example using LangChain:

from langchain.llms import CTransformers
from langchain import PromptTemplate, LLMChain

config = {"stop": "<|end|>"}
llm = CTransformers(model="TheBloke/starchat-beta-GGML", config=config)

template = """<|system|> Below is a conversation between a human user and a helpful AI coding assistant. <|end|>
<|user|> {question} <|end|>
<|assistant|> """
prompt = PromptTemplate(template=template, input_variables=["question"])
llm_chain = LLMChain(prompt=prompt, llm=llm)
response = llm_chain.run("How do I sort a list in Python?")
print(response)

Make sure you have installed the latest version of ctransformers and langchain packages.

@PunchPunch22 @boqsc I got it working in gpt4all webui with a slight modification to the c_transformers binding init file as well as the gpt4all personality config file

Thanks, both

Good point re <|end|>. I created the GGML before they fixed the special_tokens.json to add the correct EOS token of <|end|>.

Should I re-do the GGMLs? If I do, would it then bake in the correct correct end token so users wouldn't need to manually specify it?

Any thoughts, @marella ?

I don't think GGML model file supports special_tokens so re-doing might not help. Currently it is handled in code for dolly-v2 example.
Similarly I can update the starcoder example to add these special tokens for starchat. I can check if it is a starchat model by looking up if special tokens like <|end|> are part of vocab. Then in ctransformers I can handle it similar to dolly-v2.
I will try to make these changes today and release it.

Ah OK. I know it embeds the vocab so I thought it would embed the stopping tokens also. But you're right, I can't see any mention of special_token_map.json in llama.cpp's convert.py. It reads the tokenizer and added_tokens.json, but nothing else.

I guess this means that GGML models are always hardcoded to look for </s> and nothing else

Support for the <|end|> special token is added in the latest version (0.2.7) of ctransformers. So now you don't have to specify stop="<|end|>"

I will send a PR to update the example in GGML repo.