microsoft

/

Phi-3-mini-4k-instruct

like 1.08k

Follow

Microsoft 5,154

When I am trying to generate using embeddings the prepare_inputs_for_generation function requires the past_key_values to be None but it is an empty tensor (at line 1326)
code:

layer = model.get_input_embeddings()
input_ids = tokenizer(prompt, return_tensors='pt').to('cuda')
inp_embeds = layer(input_ids.input_ids)
outputs = model.generate(
                inputs_embeds=inp_embeds,
                max_new_tokens=20,
                use_cache = True,
                # do_sample=True,
            )

RuntimeError: shape '[-1, 0]' is invalid for input of size 5
In modeling_phi3.py line 1326:
given: if inputs_embeds is not None and past_key_values is None:
fix: if (inputs_embeds is not None and past_key_values is None) or (inputs_embeds is not None and len(past_key_values) == 0):