trust remote code?
Sorry if this has been asked - do these converted models need the .py files from the original Yi models?
It's a bit complicated. I believe you need the one for tokenization at a minimum. Earlier fine-tunes of Yi 34B like this one used the original base models as released. For some of the later fine-tunes, they are typically based on the Yi 34b-Llama model that renames two layers from Yi back to what they were in standard Llama. With these later fine-tunes, you do not need to trust remote code to run. But, earlier fine-tunes will need trust remote.
Yup, I'm running your 6.0bpw of Yi-34B just fine with just the tokenizer .py ( and newest exllamav2, that accepts a yi param, but not using that - hmm, or maybe I am, forget)
Had trouble with 34B-200k-8.0bpw, said not enough vram. (2x4090!)
I'll try other variants.
btw - prompt?
I'm using <|Human|>: and <|Assistant|>: (single space after each, just as a guess, since I saw <|Human|>, <|Assistant|>, and <|System|> early in the tokens.json. But there was also <|endoftext|>, <|startoftext|>, not clear the : is appropriate after <|Human|>, its a large experiment space, any help appreciated. Thanks!
Ah - I found this thread in the yi github: https://github.com/01-ai/Yi/issues/30
doesn't really clarify, but they claim <|Huma|> etc aren't used in current model, although <|endoftext|> IS important. confusing.
Update - switched to your spicyboros 8bpw model. now trying llama-2 prompt, same one I use w jondurbin's models. still not having much luck. Example prompt:
\n[INST] <>\nyou are a chatty ai.\n<>\nhi. Can I call you Alice? \n[/INST] (note what appears as <> is actually the correct SYS prompt, doesn't make it through markdown_