intfloat

/

e5-mistral-7b-instruct

like 483

I encountered an OOM error (using 96GB GPU) while running MTEB evaluation (on Miracl dataset)on e5-mistral-7b without specifying max_seq_length. While I managed to run the evaluation by reducing the max_seq_length, I have some concerns about the proper way to conduct these evaluations.

Current situation:

• Model: e5-instruct-7b
• Issue: OOM error when max_seq_length is not explicitly set
• Hardware: 96GB GPU memory

Questions:

1. What is the recommended approach for setting max_seq_length when running MTEB evaluations, especially for large language models?
2. Is there an industry standard or best practice for determining max_seq_length in benchmark evaluations?
3. If we need to limit max_seq_length due to hardware constraints, how do we ensure fair comparison with other models in the leaderboard?
4. Should we explicitly mention the max_seq_length used in our evaluation when reporting results?

I'd appreciate any insights from the community on handling sequence length limitations during benchmark evaluations, especially for resource-intensive models.