Spaces:
Sleeping
Sleeping
| from dataclasses import dataclass | |
| from enum import Enum | |
| class Task: | |
| benchmark: str | |
| metric: str | |
| col_name: str | |
| # Relevant citation for the task | |
| cite: str = "" | |
| # Select your tasks here | |
| # --------------------------------------------------- | |
| class Tasks(Enum): | |
| # task_key in the json file, metric_key in the json file, name to display in the leaderboard | |
| task0 = Task("toxigen", "acc", "Toxicity (lower is better)", cite="_ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection._ Hartvigsen et al., ACL 2022.") | |
| task1 = Task("truthfulqa_gen", "bleurt_acc", "Truthful QA", cite="_TruthfulQA: Measuring How Models Mimic Human Falsehoods._ Lin et al., ACL 2022.") | |
| # https://aclanthology.org/2020.emnlp-main.154/ | |
| task2 = Task("crows_pairs_english", "pct_stereotype", "CrowS-Pairs English", cite="_CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models._ Nangia et al., EMNLP 2020.") | |
| #task2 = Task("anli_r1", "acc", "ANLI", cite="_Adversarial NLI: A New Benchmark for Natural Language Understanding._ Nie et al., ACL 2020.") | |
| #task3 = Task("logiqa", "acc_norm", "LogiQA", cite="_LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning_. Liu et al., IJCAI 2020.") | |
| NUM_FEWSHOT = 0 # Change with your few shot MEG NOTE: Not sure what that means. | |
| # --------------------------------------------------- | |
| # Leaderboard name | |
| TITLE = """<h1 align="center" id="space-title">Toxicity Leaderboard</h1>""" | |
| # What does your leaderboard evaluate? | |
| INTRODUCTION_TEXT = """ | |
| <h2 align="center">Evaluate the toxicity of open LLMs.</h2> | |
| """ | |
| # Which evaluations are you running? how can people reproduce what you have? | |
| LLM_BENCHMARKS_TEXT = f""" | |
| ## How it works | |
| ## Reproducibility | |
| To reproduce the toxicity results, here is the command you can run using [this version](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463) of the EleutherAI LM Evaluation Harness: | |
| ```python main.py --model=hf-causal-experimental --model_args="pretrained=<your_model>,use_accelerate=True" --tasks=<task> --batch_size=1 --output_path=<output_path>``` | |
| """ | |
| EVALUATION_QUEUE_TEXT = """ | |
| ## Some good practices before submitting a model | |
| ### 1) Make sure you can load your model and tokenizer using AutoClasses: | |
| ```python | |
| from transformers import AutoConfig, AutoModel, AutoTokenizer | |
| config = AutoConfig.from_pretrained("your model name", revision=revision) | |
| model = AutoModel.from_pretrained("your model name", revision=revision) | |
| tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision) | |
| ``` | |
| If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded. | |
| Note: make sure your model is public! | |
| Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted! | |
| ### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index) | |
| It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`! | |
| ### 3) Make sure your model has an open license! | |
| This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗 | |
| ### 4) Fill up your model card | |
| When we add extra information about models to the leaderboard, it will be automatically taken from the model card | |
| ## In case of model failure | |
| If your model is displayed in the `FAILED` category, its execution stopped. | |
| Make sure you have followed the above steps first. | |
| If everything is done, check you can launch the EleutherAIHarness on your model locally. See About tab for exact command. | |
| """ | |
| CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results" | |
| CITATION_BUTTON_TEXT = r""" | |
| @misc{open-llm-toxicity-leaderboard, | |
| author = {Margaret Mitchell and Edward Beeching and Clémentine Fourrier and Nathan Habib and Sheon Han and Nathan Lambert and Nazneen Rajani and Omar Sanseviero and Lewis Tunstall and Thomas Wolf}, | |
| title = {Open LLM Toxicity Leaderboard}, | |
| year = {2024}, | |
| publisher = {Hugging Face}, | |
| howpublished = "\url{https://huggingface.co/spaces/Bias-Leaderboard/leaderboard}" | |
| } | |
| """ | |