Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.8k

Running on CPU Upgrade

Hi :)
Of course! Do you have a specific model you want me to take a look at?
Also cc: @SaylorTwift

@clefourrier
Thank you for your reply, the model I want to check is Azure99/blossom-v3-mistral-7b.
In addition, I mean that almost all tasks are in the completion state, but the results have not been uploaded. This may be an unexpected error.

Hi!
Thank you very much! Yes, it would seem from the logs that we have a problem when pushing the results! We'll try to fix it asap!

(But I can confirm your model was evaluated properly)

This is really great news, thank you again for your work.

I pushed the results of all currently evaluated models manually, while we're working on a fix

Hi @clefourrier , same thing is happening here (FINISHED in requests but no result):

https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/Weyaxi/test-help-steer-filtered-orig_eval_request_False_float16_Original.json

Hi @clefourrier , sorry to bother you again.
Could you manually push the evaluation results of the models once more?Many models have been evaluated in the past day, particularly Microsoft's Orca-2, and I think many people would be interested in that.

Hi! I just did :)
Are you sure microsoft's model has been evaluated however? I'm not seeing it in the results, I suspect it has not finished running yet

@clefourrier Thank you for your prompt response.
I apologize, it seems I made an error – the Orca model is still under evaluation. Looking forward to the leaderboard being corrected appropriately.

(Tagging @SaylorTwift re-results not uploaded, btw)

Hi ! The issue has been fixed, sorry for the confusion, you should see results uploaded again. Don't hesistate to re-open if you have any issue :)

@clefourrier @SaylorTwift It seems that the same problem has occurred again. The model has been evaluated, but the results have not been uploaded.
https://huggingface.co/datasets/open-llm-leaderboard/results/commits/main

Hi!
Thank you for reporting! Just did a manual upload of the results.

Yes, it's here again. It seems that a batch of tasks has failed recently. Can you take another look?
Azure99/blossom-v3_1-yi-34b
@clefourrier @SaylorTwift
Thank you again.

Hi @Azure99 , can you point us to the request file please?

hi @clefourrier , it's here.
/Azure99/blossom-v3_1-yi-34b_eval_request_False_bfloat16_Original.json
https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/32cdbd36e7352b1bbed8fcc55dc5ea8b7b81382b
By the way, the recent evaluations of several 34B models have all failed, but evaluations of models with sizes like 7B are normal.

Hi! There was a connectivity issue when trying to read your model - I added it back to pending.

Hi @clefourrier , just like before, another task unexpectedly failed, could you help with restarting it? Thank you very much.
https://huggingface.co/datasets/open-llm-leaderboard/requests/commit/2a2c89c16442b63af92aed7a68eea23d3073c7ee

Hi @Azure99 , the model failed to be loaded - can you upload it in safetensors as required on the Submit page please? :)
Once it's uploaded, I'll relaunch it.

(Two points as a side note, to simplify our work next time: please point to the request file directly, not to the commit of the request file, and please open a new discussion instead of reopening old ones, so that we can sort them by order of priority more easily. That would really simplify things for us :) )

Closing for inactivity, feel free to reopen if needed