Model stuck in Open LLM eval for over 24hrs

#609
by CorticalStack - opened

Hi @clefourrier and HF staff,

Model (CorticalStack/pastiche-crown-clown-7b-dare) stuck in Open LLM eval for over 24hrs. Clicking on the model shows no results (404), below are links to this and the original eval request. Hoping you can resolve, many thanks:

https://huggingface.co/datasets/open-llm-leaderboard/details_CorticalStack__pastiche-crown-clown-7b-dare

https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/CorticalStack/pastiche-crown-clown-7b-dare_eval_request_False_bfloat16_Original.json

sometimes evaluations take longer than 24 hours, sometimes they take weeks, and even then they could fail in the end.

Open LLM Leaderboard org

Hi!
As indicated in the request file, your model is still running. Please be patient, and feel free to reopen this issue if it has not been evaluated in 3 days.

clefourrier changed discussion status to closed

@nlpguy @clefourrier thanks for the clarification. Was just a little concerning given that models I've submitted AFTER the aforementioned model completed eval whilst this one still "running". Got it, will check and rely on request file info.

Open LLM Leaderboard org

Sometimes, models are cancelled by other higher priority jobs and rescheduled, in which case they have a longer "RUNNING" time - I checked the logs and your specific model got stuck over the weekend, I passed it in PENDING again so it gets relaunched.

Thank-you for the info and rescheduling it @clefourrier

Sign up or log in to comment