Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.8k

Running on CPU Upgrade

Hi!
I think you could check our FAQ :)
TLDR: Either we have a problem with the display atm, or they ran but failed. You'll get this info by looking for their request and result files.

@clefourrier From what I've seen the leaderboard does not update until restarted. Whether that is intentional or not, restarts from time to time would be nice until there is a better solution.

Interesting!
We actually have a new system with webhooks, where the leaderboard should be updated max 10 min after a change on our datasets (redownloaded with every change) - I'll take a look again at this

Thanks for the pointer. I found the status of Smaug-Llama:

{
"model": "abacusai/Smaug-Llama-3-70B-Instruct",
"base_model": "",
"revision": "8f558d6211b9d8f1712b80df40c5b65bea0b56ea",
"precision": "bfloat16",
"params": 70.554,
"architectures": "LlamaForCausalLM",
"weight_type": "Original",
"status": "FAILED",
"submitted_time": "2024-06-26T16:29:36Z",
"model_type": "\ud83d\udd36 : \ud83d\udd36 fine-tuned on domain-specific datasets",
"job_id": "7215733",
"job_start_time": "2024-06-27T00:10:42.571625",
"use_chat_template": true
}

It's not clear what caused the FAILURE - we know the model files are non-corrupt as it worked fine on the old leaderboard. In any case, I've resubmitted for now.

Hi! Please do not try to resubmit models which failed!
Instead, give us the link to the request file so we can investigate and relaunch if necessary!

Ah - my apologies, I've already resubmitted. The requests file was originally here: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/abacusai/Smaug-Llama-3-70B-Instruct_eval_request_False_bfloat16_Original.json

although it has now been updated since I have resubmitted.

It should no longer be possible to resubmit a model which was already submitted, so thanks for raising the issue, at least this has been fixed.

Re-Smaug, it got preempted - normally the job should have been rescheduled but apparently was not, tagging @SaylorTwift - note that since it's PENDING again, it will be relaunched soon, when there is enough space on the cluster