Remove ` (default)` that has been included due to a bug in MTEB

#1
by tomaarsen HF staff - opened

Hello!

Pull Request overview

  • Remove (default) that has been included due to a bug in MTEB

Details

I ran some experiments to figure out the origin of why this model wasn't yet shown on MTEB. As it turns out, a recent MTEB update introduced a bug that adds (default) to the dataset names, leading to a discrepancy between the reported dataset names in metadata from new models and the expected dataset names. This PR addresses that. I've tested this locally, and it worked for me there:

TiQFPMwWfYE47v7S8EfQY.png

On an unrelated note: I'm looking forward to your release & reading about your approach! I'd gladly try and assist from the Hugging Face side to try and get you some nice reach for your model when it releases, e.g. by sharing the word on the socials and/or helping to integrate the model with third parties (LangChain, LlamaIndex, etc.). We actually already have a Nvidia - HF Slack channel for embedding models by Nvidia that I could invite you to.

cc @nv-bschifferer

  • Tom Aarsen
nv-bschifferer changed pull request status to merged

Sign up or log in to comment