open-llm-leaderboard/open_llm_leaderboard · Result PRs not appearing

5 days ago

•

Hello! I'm seeing some merge authors getting result PRs and having their models listed, but I haven't seen any in over a week. I know this is a new version - any new critical field in our model cards for us to look after? Any permissions?

clefourrier

Open LLM Leaderboard org 4 days ago

Hi! The PRs were opened by a tool that @Weyaxi kindly made for the leaderboard, and which is not maintained by the leaderboard authors. It is possible that the tool is not up to date.
@Weyaxi if the code of your tool is open, maybe members of the community could give you a hand to put it up to date?

Weyaxi

about 20 hours ago

Hi @sometimesanotion and @clefourrier ,

For some reason, the following code wasn't working in both spaces for opening PRs:

pd.DataFrame(load_dataset("open-llm-leaderboard/contents", split="train").sort("Average ⬆️", reverse=True))

The strange part is that it works in various environments like Colab or locally.

As a workaround, I had to replace this with the pd.read_parquet function, which ideally shouldn't be preferred:

pd.read_parquet("https://huggingface.co/datasets/open-llm-leaderboard/contents/resolve/main/data/train-00000-of-00001.parquet").sort_values(by="Average ⬆️", ascending=False)

This issue appears to be resolved now, and new model PRs should be opened whenever a new model is added to the leaderboard!

I will track this issue and confirm it is working!

Best,
Weyaxi