Unbabel

/

wmt23-cometkiwi-da-xl

like 14

answering to myself: converting to fp16, changing two lines of code model.half() and in_features.to(torch.float16) makes thing twice faster, twice less ram, same scores.

Hi @vince62s , did you change the model to half precision and cast the input on your side of the code? Or did you do it inside the unbabel-comet python package?

I modified the comet code, only two lines to change as mentioned in my previous message.

@vince62s which files did you change?

in score.py you need to add half() to the line: model = load_from_checkpoint(model_path).half()
in feed_forward?py you need to change the last line: return self.ff(in_features.to(torch.float16))

Confirmed, it works. Thanks so much!
Maybe a PR for unbabels would be appreciated?