microsoft

/

Florence-2-large

like 910

Two changes:

• Fix Florence2Seq2SeqLMOutput
• Only set attention_mask device if inputs_embeds is not None

hi @Xenova in which case will inputs_embeds be None, no input_ids and pixel_values, so input is decoder_input_ids?

This occurs when encoder_outputs is already specified, meaning we don't need to prepare input_embeds again.

As seen here, you only use inputs_embeds in the LM forward method when encoder_outputs is None

when will this get merged ??
is there a ready to use fork ??

i keep getting
TypeError: Florence2Seq2SeqLMOutput.init() got an unexpected keyword argument 'loss'
when i try to finetune

hi @leoxiaobin , please merge this pr

for epoch in range(epochs):
    # Training phase
    model.train()
    train_loss = 0
    i = -1
    for batch in tqdm(train_loader, desc=f"Training Epoch {epoch + 1}/{epochs}"):
        i += 1
        inputs, answers = batch

        labels = processor.tokenizer(
            text=answers,
            return_tensors="pt",
            padding=True,
            return_token_type_ids=False,
        ).input_ids.to(device)

        # 将标签向右移位，作为解码器的输入
        decoder_input_ids = shift_tokens_right(
            labels, model.config.pad_token_id, model.config.decoder_start_token_id
        ).to(device)
        print("input_ids:::",inputs["input_ids"])
        print("pixel_values:::",inputs["pixel_values"])
        # outputs = model(input_ids=input_ids, pixel_values=pixel_values, labels=labels)
        outputs = model(
            input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"], labels=labels
        )
        print("outputs:::",outputs)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        train_loss += loss.item()

occured:

TypeError Traceback (most recent call last)
Cell In[3], line 132
129 train_loader = DataLoader(train_dataset, batch_size=2, collate_fn=collate_fn, num_workers=0, shuffle=True)
131 # 调用训练函数
--> 132 train_model(train_loader, model, processor, epochs=3)

Cell In[3], line 65, in train_model(train_loader, model, processor, epochs, lr)
63 print("pixel_values:::",inputs["pixel_values"])
64 # outputs = model(input_ids=input_ids, pixel_values=pixel_values, labels=labels)
---> 65 outputs = model(
66 input_ids=inputs["input_ids"], pixel_values=inputs["pixel_values"], labels=labels
67 )
68 print("outputs:::",outputs)
69 loss = outputs.loss

File ~/Funtune/florence2-finetuning/florence2-uv/lib/python3.11/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
1530 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]
1531 else:
-> 1532 return self._call_impl(*args, **kwargs)

File ~/Funtune/florence2-finetuning/florence2-uv/lib/python3.11/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
1536 # If we don't have any hooks, we want to skip the rest of the logic in
1537 # this function, and just call forward.
1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
1539 or _global_backward_pre_hooks or _global_backward_hooks
1540 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541 return forward_call(*args, **kwargs)
1543 try:
1544 result = None

File ~/.cache/huggingface/modules/transformers_modules/modeling_florence2.py:2760, in Florence2ForConditionalGeneration.forward(self, input_ids, pixel_values, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, cross_attn_head_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
2757 output = (logits,) + outputs[1:]
2758 return (loss,) + output if loss is not None else output
-> 2760 return Florence2Seq2SeqLMOutput(
2761 loss=loss,
2762 logits=logits,
2763 past_key_values=outputs.past_key_values,
2764 decoder_hidden_states=outputs.decoder_hidden_states,
2765 decoder_attentions=outputs.decoder_attentions,
2766 cross_attentions=outputs.cross_attentions,
2767 encoder_last_hidden_state=outputs.encoder_last_hidden_state,
2768 encoder_hidden_states=outputs.encoder_hidden_states,
2769 encoder_attentions=outputs.encoder_attentions,
2770 image_hidden_states=image_features
2771 )

TypeError: Florence2Seq2SeqLMOutput.init() got an unexpected keyword argument 'loss'