Tensor Mismatch
I tried 2 different 4bit branches (32g and main) and both of them come up with this error:
Traceback (most recent call last):
File "c:\Users\Scott\VSC_Source\LLM_DB\EXLLAMAv2y production.py", line 181, in
output = generator.generate_simple(prompt,settings,max_new_tokens)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Scott\anaconda3\envs\ELLAMA\Lib\site-packages\exllamav2\generator\base.py", line 60, in generate_simple
token, _ = ExLlamaV2Sampler.sample(logits, gen_settings, self.sequence_ids, random.random())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Scott\anaconda3\envs\ELLAMA\Lib\site-packages\exllamav2\generator\sampler.py", line 63, in sample
if settings.token_bias is not None: logits += settings.token_bias
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The size of tensor a (32032) must match the size of tensor b (32002) at non-singleton dimension 1