Helsinki-NLP/opus-mt-tc-big-gmq-itc · Update checkpoint for transformers>=4.29

Language Technology Research Group at the University of Helsinki org Oct 10, 2023

Following the merge of a PR in transformers it appeared that this model was not properly converted. This PR will fix the inference and was tested using the following script:

>>> from transformers import AutoTokenizer, MarianMTModel
>>> tokenizer = AutoTokenizer.from_pretrained('Helsinki-NLP/opus-mt-tc-big-gmq-itc')
>>> inputs = tokenizer("' >>en<< Hey how are you?'", return_tensors="pt", padding=True)
>>> model = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-tc-big-gmq-itc')
>>> print(tokenizer.batch_decode(model.generate(**inputs)))
["<pad>'Hey how'e tu? '? '?</s>"]

ArthurZ

Language Technology Research Group at the University of Helsinki org Oct 10, 2023

Automatically merging the PR.

ArthurZ changed pull request status to merged Oct 10, 2023

Update checkpoint for transformers>=4.29f6da5f81