Update src/submission/check_validity.py

#509

Hello!

This is a small fix for correct model parameter size.
Because of the regex r"(\d\.)?\d+(b|m)", sometimes it returns wrong model size if it tries to extract model size from its name.

For example, Sao10K/Frostwind-10.7B-v1 currently has 0.7 param size in the leaderboard, which is wrong.

I hope this improves leaderboard accuracy a bit!

Cheers

Open LLM Leaderboard org

Following the creation of this PR, an ephemeral Space HuggingFaceH4/open_llm_leaderboard-ci-pr-509 has been started. Any changes pushed to this PR will be synced with the test Space.
Since this PR has not been created by a trusted author, the ephemeral Space has not been configured with the correct hardware, storage, and secrets. An admin must configure it manually.
(This is an automated message.)

Open LLM Leaderboard org

Hi ! Thanks for the fix, do you think you can also fix the typo where we define size_pattern two times ?

Open LLM Leaderboard org

I'll do it, merging this so it gets integrated now :)

clefourrier changed pull request status to merged
Open LLM Leaderboard org

PR is now merged/closed. The ephemeral Space has been deleted.
(This is an automated message.)

Sign up or log in to comment