MLM Head
Hi there! Quick question, is it possible to retrieve the MLM weights from the model as it is now? If not, is it possible to upload these weights here?
Currently, calling AutoModelForMaskedLM.from_pretrained("intfloat/multilingual-e5-base") raises an issue with non-initialized weights for the MLM.
Thanks!
@ArthurCamara The E5 family of models do not have MLM head, as they are designed to produce dense embeddings only.
Yeah, my bad. I misread the paper. I thought the pre-training also used an MLM objective. Just noticed the last paragraph in the "Negative Results section". Sorry about that.
Just to clarify, we were looking into reproducing this paper: https://arxiv.org/abs/2212.10380 that uses DPR, but with E5. Using the XLM-Roberta MLM head worked fine for the multilingual model and BERT worked for the other ones.