Update space

.idea/.gitignore

@@ -0,0 +1,8 @@
+# Default ignored files
+/shelf/
+/workspace.xml
+# Editor-based HTTP Client requests
+/httpRequests/
+# Datasource local storage ignored files
+/dataSources/
+/dataSources.local.xml

.idea/Multilingual-MMLU-Benchmark-Leaderboard.iml

@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module type="PYTHON_MODULE" version="4">
+  <component name="NewModuleRootManager">
+    <content url="file://$MODULE_DIR$" />
+    <orderEntry type="inheritedJdk" />
+    <orderEntry type="sourceFolder" forTests="false" />
+  </component>
+  <component name="PyDocumentationSettings">
+    <option name="format" value="PLAIN" />
+    <option name="myDocStringFormat" value="Plain" />
+  </component>
+</module>

.idea/inspectionProfiles/profiles_settings.xml

@@ -0,0 +1,6 @@
+<component name="InspectionProjectProfileManager">
+  <settings>
+    <option name="USE_PROJECT_PROFILE" value="false" />
+    <version value="1.0" />
+  </settings>
+</component>

.idea/modules.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="ProjectModuleManager">
+    <modules>
+      <module fileurl="file://$PROJECT_DIR$/.idea/Multilingual-MMLU-Benchmark-Leaderboard.iml" filepath="$PROJECT_DIR$/.idea/Multilingual-MMLU-Benchmark-Leaderboard.iml" />
+    </modules>
+  </component>
+</project>

.idea/vcs.xml

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="VcsDirectoryMappings">
+    <mapping directory="" vcs="Git" />
+  </component>
+</project>

README.md

@@ -1,45 +1,100 @@
----
-title: Multilingual MMLU Benchmark Leaderboard
-emoji: 🥇
-colorFrom: green
-colorTo: indigo
-sdk: gradio
-app_file: app.py
-pinned: true
-license: apache-2.0
-short_description: Multilingual MMLU Benchmark Leaderboard
----
-
-# Start the configuration
-
-Most of the variables to change for a default leaderboard are in `src/env.py` (replace the path for your leaderboard) and `src/about.py` (for tasks).
-
-Results files should have the following format and be stored as json files:
-```json
-{
-    "config": {
-        "model_dtype": "torch.float16", # or torch.bfloat16 or 8bit or 4bit
-        "model_name": "path of the model on the hub: org/model",
-        "model_sha": "revision on the hub",
-    },
-    "results": {
-        "task_name": {
-            "metric_name": score,
-        },
-        "task_name2": {
-            "metric_name": score,
-        }
-    }
-}
-```
-
-Request files are created automatically by this tool.
-
-If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.
-
-# Code logic for more complex edits
-
-You'll find 
-- the main table' columns names and properties in `src/display/utils.py`
-- the logic to read all results and request files, then convert them in dataframe lines, in `src/leaderboard/read_evals.py`, and `src/populate.py`
-- the logic to allow or filter submissions in `src/submission/submit.py` and `src/submission/check_validity.py`
+# Multilingual MMLU Benchmark Leaderboard
+
+Welcome to the **Multilingual MMLU Benchmark Leaderboard**! This leaderboard is designed to assess the performance of both open-source and closed-source language models on the **Multilingual MMLU (Massive Multitask Language Understanding)** benchmark. The benchmark evaluates the memorization, reasoning, and linguistic capabilities of models across a wide range of languages, making it a crucial tool for comparing multilingual AI performance.
+
+## Overview
+
+The **Multilingual MMLU Benchmark** is a comprehensive evaluation platform for AI models, assessing their general knowledge and ability to handle diverse tasks across **57 distinct domains**. These tasks range from elementary-level knowledge to more advanced subjects such as law, physics, history, and computer science. With this leaderboard, we aim to provide an accessible and reliable way for developers, researchers, and organizations to compare language models' multilingual understanding and reasoning abilities.
+
+## Evaluation Scope
+
+### Key Features:
+- **Multilingual Coverage**: The leaderboard evaluates language models on **14 different languages**, including widely spoken languages such as English, Spanish, Arabic, and Chinese, as well as languages with fewer resources like Swahili and Yoruba.
+- **Diverse Domains**: The benchmark includes tasks across 57 domains, ensuring that models are tested on a wide range of topics, from elementary knowledge to complex professional fields.
+- **Comprehensive QA Tasks**: The evaluation focuses on **Question Answering (QA)** tasks, where models answer questions based on general knowledge in various domains. This helps assess both the depth and breadth of the model's knowledge.
+  
+### Languages Covered:
+The leaderboard includes models evaluated on the following languages:
+
+- **AR_XY**: Arabic
+- **BN_BD**: Bengali
+- **DE_DE**: German
+- **ES_LA**: Spanish
+- **FR_FR**: French
+- **HI_IN**: Hindi
+- **ID_ID**: Indonesian
+- **IT_IT**: Italian
+- **JA_JP**: Japanese
+- **KO_KR**: Korean
+- **PT_BR**: Brazilian Portuguese
+- **SW_KE**: Swahili
+- **YO_NG**: Yoruba
+- **ZH_CN**: Simplified Chinese
+
+## How It Works
+
+### Submitting Models
+To evaluate your model on the Multilingual MMLU Benchmark, you can submit it through the **"Submit here"** tab on the leaderboard. The evaluation process will run your model through a series of tests across the 57 domains in the 14 supported languages. Results will be provided on the leaderboard, with detailed scores for each language and domain.
+
+### Evaluation Process
+We use the **OpenCompass framework** to automate the evaluation process. This framework enables efficient execution of multiple tests across different languages and domains, ensuring scalability and reproducibility. The following is the evaluation setup:
+
+- **Evaluation Method**: All tasks are evaluated using a 5-shot setting.
+- **Normalization**: Results are normalized using the following formula:
+  ```plaintext
+  normalized_value = (raw_value - random_baseline) / (max_value - random_baseline)
+
+## Evaluation Details
+
+### For Generative Tasks:
+- **Random Baseline**: `random_baseline = 0`
+
+### For Multiple-Choice QA Tasks:
+- **Random Baseline**: `random_baseline = 1/n`, where `n` is the number of choices.
+
+### Aggregated Results:
+Scores for each language are averaged across all tasks to provide a comprehensive model evaluation.
+
+## Results
+
+- **Numerical Results**: You can access the detailed evaluation results in the [results dataset](#).
+- **Community Queries**: Track ongoing tasks and request status in the [requests dataset](#).
+
+## Reproducibility
+
+To ensure reproducibility, we provide a fork of the **lm_eval** framework, which allows you to recreate the evaluation setup and results on your own models. While not all contributions are integrated into the main repository, our fork contains all the necessary updates for evaluation.
+
+For detailed setup instructions and to reproduce results on your local machine, please refer to our [lm_eval fork](#).
+
+## Acknowledgements
+
+This leaderboard was developed as part of the **#ProjectName**, led by **[OrganizationName]**, and has benefited from the support of high-quality evaluation datasets donated by the following institutions:
+
+- [Institution 1]
+- [Institution 2]
+- [Institution 3]
+- [Institution 4]
+- [Institution 5]
+- [Institution 6]
+- [Institution 7]
+
+We also thank **[Institution1]**, **[Institution2]**, and **[Institution3]** for sponsoring inference GPUs, which were crucial for running the large-scale evaluations.
+
+### Special Thanks to the Contributors:
+- **Task Implementation**: [Name1], [Name2], [Name3]
+- **Leaderboard Implementation**: [Name4], [Name5]
+- **Model Evaluation**: [Name6], [Name7]
+- **Communications**: [Name8], [Name9]
+- **Organization & Collaboration**: [Name10], [Name11], [Name12]
+
+For more information on the datasets, please refer to the **Dataset Cards** available in the "Tasks" tab and the **Citations** section below.
+
+## Collaborate
+
+We are always looking for collaborators to expand the range of evaluated models and datasets. If you would like us to include your evaluation dataset or contribute in any other way, please feel free to get in touch!
+
+Your feedback, suggestions, and contributions are more than welcome! Visit our **[Community Page]** to share your thoughts, or feel free to open a pull request (PR).
+
+Thank you for your interest in the **Multilingual MMLU Benchmark Leaderboard**. Let’s work together to advance multilingual AI capabilities!
+
+For more details on dataset authors and dataset cards, please refer to the "Tasks" tab.

app.py

@@ -8,8 +8,12 @@ from src.about import (
     CITATION_BUTTON_LABEL,
     CITATION_BUTTON_TEXT,
     EVALUATION_QUEUE_TEXT,
+    EVALUATION_QUEUE_TEXT_ZH,
     INTRODUCTION_TEXT,
+    INTRODUCTION_TEXT_ZH,
     LLM_BENCHMARKS_TEXT,
+    LLM_BENCHMARKS_TEXT_ZH,
+    LOGOS,
     TITLE,
 )
 from src.display.css_html_js import custom_css
@@ -32,7 +36,7 @@ from src.submission.submit import add_new_eval
 def restart_space():
     API.restart_space(repo_id=REPO_ID)
 
-### Space initialisation
+## Space initialisation
 try:
     print(EVAL_REQUESTS_PATH)
     snapshot_download(
@@ -93,18 +97,25 @@ demo = gr.Blocks(css=custom_css)
 with demo:
     gr.HTML(TITLE)
     gr.Markdown(INTRODUCTION_TEXT, elem_classes="markdown-text")
+    gr.Markdown(INTRODUCTION_TEXT_ZH, elem_classes="markdown-text")
 
     with gr.Tabs(elem_classes="tab-buttons") as tabs:
         with gr.TabItem("🏅 LLM Benchmark", elem_id="llm-benchmark-tab-table", id=0):
             leaderboard = init_leaderboard(LEADERBOARD_DF)
 
         with gr.TabItem("📝 About", elem_id="llm-benchmark-tab-table", id=2):
-            gr.Markdown(LLM_BENCHMARKS_TEXT, elem_classes="markdown-text")
+            with gr.TabItem("EN", elem_id="llm-benchmark-tab-table", id=1):
+                gr.Markdown(LLM_BENCHMARKS_TEXT, elem_classes="markdown-text")
+            with gr.TabItem("ZH", elem_id="llm-benchmark-tab-table", id=2):
+                gr.Markdown(LLM_BENCHMARKS_TEXT_ZH, elem_classes="markdown-text")
 
         with gr.TabItem("🚀 Submit here! ", elem_id="llm-benchmark-tab-table", id=3):
             with gr.Column():
                 with gr.Row():
-                    gr.Markdown(EVALUATION_QUEUE_TEXT, elem_classes="markdown-text")
+                    with gr.TabItem("EN", elem_id="llm-benchmark-tab-table", id=1):
+                        gr.Markdown(EVALUATION_QUEUE_TEXT, elem_classes="markdown-text")
+                    with gr.TabItem("ZH", elem_id="llm-benchmark-tab-table", id=2):
+                        gr.Markdown(EVALUATION_QUEUE_TEXT_ZH, elem_classes="markdown-text")
 
                 with gr.Column():
                     with gr.Accordion(
@@ -197,6 +208,15 @@ with demo:
                 elem_id="citation-button",
                 show_copy_button=True,
             )
+    with gr.Row():
+        for logo_path in LOGOS:
+            gr.Image(
+                value=logo_path,
+                show_label=False,
+                show_download_button=False,
+                show_share_button=False,
+                show_fullscreen_button=False,
+            )
 
 scheduler = BackgroundScheduler()
 scheduler.add_job(restart_space, "interval", seconds=1800)

src/about.py

@@ -12,31 +12,255 @@ class Task:
 # ---------------------------------------------------
 class Tasks(Enum):
     # task_key in the json file, metric_key in the json file, name to display in the leaderboard 
-    task0 = Task("anli_r1", "acc", "ANLI")
-    task1 = Task("logiqa", "acc_norm", "LogiQA")
-
-NUM_FEWSHOT = 0 # Change with your few shot
+    task0 = Task("mmmlu", "acc", "MMMLU")
+    task1 = Task("mmlu", "acc", "MMLU")
+    task2 = Task("cmmlu", "acc", "CMMLU")
+    task3 = Task("mmmlu_ar", "acc", "MMMLU_AR")
+    task4 = Task("mmmlu_bn", "acc", "MMMLU_BN")
+    task5 = Task("mmmlu_de", "acc", "MMMLU_DE")
+    task6 = Task("mmmlu_es", "acc", "MMMLU_ES")
+    task7 = Task("mmmlu_fr", "acc", "MMMLU_FR")
+    task8 = Task("mmmlu_hi", "acc", "MMMLU_HI")
+    task9 = Task("mmmlu_id", "acc", "MMMLU_ID")
+    task10 = Task("mmmlu_it", "acc", "MMMLU_IT")
+    task11 = Task("mmmlu_ja", "acc", "MMMLU_JA")
+    task12 = Task("mmmlu_ko", "acc", "MMMLU_KO")
+    task13 = Task("mmmlu_pt", "acc", "MMMLU_PT")
+    task14 = Task("mmmlu_sw", "acc", "MMMLU_SW")
+    task15 = Task("mmmlu_yo", "acc", "MMMLU_YO")
+    task16 = Task("mmmlu_zh", "acc", "MMMLU_ZH")
+NUM_FEWSHOT = 5 # Change with your few shot
 # ---------------------------------------------------
 
 
 
 # Your leaderboard name
-TITLE = """<h1 align="center" id="space-title">Demo leaderboard</h1>"""
+TITLE = """<h1 align="center" id="space-title">Multilingual MMLU Benchmark Leaderboard</h1>"""
 
 # What does your leaderboard evaluate?
 INTRODUCTION_TEXT = """
-Intro text
+**Multilingual MMLU Benchmark Leaderboard:** This leaderboard is dedicated to evaluating the performance of both open-source and closed-source language models on the Multilingual MMLU benchmark. It assesses their memorization, reasoning, and linguistic capabilities across a wide range of languages. The leaderboard consolidates multiple MMLU datasets, originally created or manually translated into various languages, to provide a comprehensive evaluation of multilingual understanding in LLMs.
+"""
+INTRODUCTION_TEXT_ZH = """
+**多语言 MMLU 基准榜单：** 这是一个开放的评测榜单，旨在评估开源和闭源语言模型在多语言 MMLU 基准测试中的表现，涵盖记忆、推理和语言能力。该榜单整合了多个 MMLU 数据集，这些数据集最初为多种语言创建或手动翻译，旨在全面评估大规模语言模型在多语言理解上的能力。
 """
-
 # Which evaluations are you running? how can people reproduce what you have?
-LLM_BENCHMARKS_TEXT = f"""
-## How it works
+# TODO: Update number of benchmarks
+LLM_BENCHMARKS_TEXT = """
+## 💡 About "Multilingual Benchmark MMLU Leaderboard"
+
+- Press release: [TBD - XXX](#), [TBD - XXX](#), [TBD - XXX](#), [TBD - XXX](#)
+- YouTube: [TBD - XXX](#)
+
+### Overview
+The **Multilingual Massive Multitask Language Understanding (MMMLU)** benchmark is a comprehensive evaluation platform designed to assess the general knowledge capabilities of AI models across a wide range of domains. It includes a series of **Question Answering (QA)** tasks across **57 distinct domains**, ranging from elementary-level knowledge to advanced professional subjects such as law, physics, history, and computer science.
+
+### Translation Effort
+For this evaluation, we used the **OpenAI MMMLU dataset**, which has been extensively curated and tested for a multilingual understanding of AI models. The dataset includes 14 different languages and is specifically designed to assess how well AI models can handle a wide range of general knowledge tasks across 57 domains. 
+
+While the translation of the test set was performed by OpenAI, it ensures a high level of accuracy and reliability for evaluating multilingual models. By leveraging this pre-existing, professionally curated dataset, we aim to focus on model performance across multiple languages, without the need for additional translations from our side.
+
+### Commitment to Multilingual AI
+By focusing on human-powered translations and publishing both the translated test sets and evaluation code, we aim to promote the development of AI models that can handle multilingual tasks with greater accuracy. This reflects our commitment to improving AI’s performance in underrepresented languages and making technology more inclusive and effective globally.
+
+### Locales Covered
+The MMMLU benchmark includes a test set translated into the following locales:
+- **AR_XY**: Arabic
+- **BN_BD**: Bengali
+- **DE_DE**: German
+- **ES_LA**: Spanish
+- **FR_FR**: French
+- **HI_IN**: Hindi
+- **ID_ID**: Indonesian
+- **IT_IT**: Italian
+- **JA_JP**: Japanese
+- **KO_KR**: Korean
+- **PT_BR**: Brazilian Portuguese
+- **SW_KE**: Swahili
+- **YO_NG**: Yoruba
+- **ZH_CN**: Simplified Chinese
+
+### Purpose
+The MMMLU Leaderboard aims to provide a unified benchmark for comparing AI model performance across these multiple languages and diverse domains. With the inclusion of the **QA task** across **57 domains**, it evaluates how well models perform in answering general knowledge questions in multiple languages, ensuring a high standard of multilingual understanding and reasoning.
+
+### Goals
+Our primary goal is to provide a reliable comparison for AI models across different languages and domains, helping developers and researchers evaluate and improve their models’ multilingual capabilities. By emphasizing high-quality translations and including a broad range of topics, we strive to make AI models more robust and useful across diverse communities worldwide.
+
+### 🤗 How it works
+
+Submit a model for automated evaluation on our clusters on the "Submit here" tab!
+
+### 📈 Tasks
+
+We evaluate models on a variety of key benchmarks, with a focus on **Multilingual Massive Multitask Language Understanding (MMLU)** and its variants, including MMLU, C-MMLU, ArabicMMLU, KoreanMMLU, MalayMMLU, and others. These benchmarks assess general knowledge across a wide range of topics from 57 categories, such as law, physics, history, and computer science.
+
+The evaluation is performed using the [OpenCompass framework](https://github.com/open-compass/opencompass), a unified platform for evaluating language models across multiple tasks. OpenCompass allows us to execute these evaluations efficiently and at scale, covering multiple languages and benchmarks.
+
+For detailed information on the tasks, please refer to the "Tasks" tab in the OpenCompass framework.
+
+Notes:
+- The evaluations are all 5-shot.
+- The results are normalized with the following formula: `normalized_value = (raw_value - random_baseline) / (max_value - random_baseline)`, where `random_baseline` is `0` for generative tasks and `1/n` for multi-choice QA  with `n` choices.
+- Results are aggregated by calculating the average of all the tasks for a given language.
+
+### 🔎 Results
+
+You can find:
+
+- Detailed numerical results in the [results dataset](link_to_results)
+- Community queries and running status in the [requests dataset](link_to_requests)
 
-## Reproducibility
-To reproduce our results, here is the commands you can run:
+### ✅ Reproducibility
+
+To reproduce the results, you can use [our fork of lm_eval](#), as not all of our PRs are currently integrated into the main repository.
+
+## 🙌 Acknowledgements
+
+This leaderboard was developed as part of the [#ProjectName](link_to_project) led by [OrganizationName](link_to_organization) thanks to the donation of high-quality evaluation datasets by:
+
+- [Institution 1](link_to_institution_1)
+- [Institution 2](link_to_institution_2)
+- [Institution 3](link_to_institution_3)
+- [Institution 4](link_to_institution_4)
+- [Institution 5](link_to_institution_5)
+- [Institution 6](link_to_institution_6)
+- [Institution 7](link_to_institution_7)
+- [Institution 8](link_to_institution_8)
+- [Institution 9](link_to_institution_9)
+
+The entities above are ordered chronologically by the date they joined the project. However, the logos in the footer are ordered by the number of datasets donated.
+
+Thank you in particular to:
+- Task implementation: [Name 1], [Name 2], [Name 3], [Name 4], [Name 5], [Name 6], [Name 7], [Name 8], [Name 9], [Name 10]
+- Leaderboard implementation: [Name 11], [Name 12]
+- Model evaluation: [Name 13], [Name 14], [Name 15], [Name 16], [Name 17]
+- Communication: [Name 18], [Name 19]
+- Organization & colab leads: [Name 20], [Name 21], [Name 22], [Name 23], [Name 24], [Name 25], [Name 26], [Name 27], [Name 28], [Name 29], [Name 30]
+
+For information about the dataset authors please check the corresponding Dataset Cards (linked in the "Tasks" tab) and papers (included in the "Citation" section below). We would like to specially thank the teams that created or open-sourced their datasets specifically for the leaderboard (in chronological order):
+- [Dataset1 Placeholder] and [Dataset2 Placeholder]: [Team members placeholder]
+- [Dataset3 Placeholder], [Dataset4 Placeholder] and [Dataset5 Placeholder]: [Team members placeholder]
+- [Dataset6 Placeholder]: [Team members placeholder]
+
+We also thank [Institution1 Placeholder], [Institution2 Placeholder], [Organization Placeholder], [Person1 Placeholder], [Person2 Placeholder] and [Institution3 Placeholder] for sponsoring the inference GPUs.
+
+## 🚀 Collaborate!
+
+We would like to create a leaderboard as diverse as possible, reach out if you would like us to include your evaluation dataset!
+
+Comments and suggestions are more than welcome! Visit the [👏 Community](<Community Page Placeholder>) page, tell us what you think about La Leaderboard and how we can improve it, or go ahead and open a PR!
+
+Thank you very much! 💛
 
 """
 
+LLM_BENCHMARKS_TEXT_ZH = """
+## 💡 关于 "多语言基准 MMLU 排行榜"
+
+- 新闻稿：[待定 - XXX](#), [待定 - XXX](#), [待定 - XXX](#), [待定 - XXX](#)
+- YouTube：[待定 - XXX](#)
+
+### 概述
+**多语言大规模多任务语言理解 (MMMLU)** 基准是一个全面的评估平台，旨在评估 AI 模型在各个领域的通用知识能力。它包括一系列跨越 **57 个不同领域** 的 **问答 (QA)** 任务，从基础知识到法律、物理、历史、计算机科学等高级专业���题。
+
+### 翻译工作
+对于本次评估，我们使用了 **OpenAI MMMLU 数据集**，该数据集已经广泛策划并测试了 AI 模型的多语言理解能力。该数据集包括 14 种不同的语言，专门设计用来评估 AI 模型在 57 个领域中处理各种通用知识任务的能力。
+
+尽管测试集的翻译是由 OpenAI 执行的，但它确保了评估多语言模型的高准确性和可靠性。通过利用这个预先存在的、专业策划的数据集，我们旨在专注于模型在多种语言中的表现，而无需我们额外进行翻译工作。
+
+### 致力于多语言 AI
+通过专注于人工翻译并公开翻译后的测试集和评估代码，我们旨在促进能够处理多语言任务的 AI 模型的开发，并使其更加准确。这也体现了我们在改善 AI 在低资源语言中的表现以及推动全球技术包容性方面的承诺。
+
+### 涵盖的语言区域
+MMMLU 基准包括以下语言区域的翻译测试集：
+- **AR_XY**：阿拉伯语
+- **BN_BD**：孟加拉语
+- **DE_DE**：德语
+- **ES_LA**：西班牙语
+- **FR_FR**：法语
+- **HI_IN**：印地语
+- **ID_ID**：印尼语
+- **IT_IT**：意大利语
+- **JA_JP**：日语
+- **KO_KR**：韩语
+- **PT_BR**：巴西葡萄牙语
+- **SW_KE**：斯瓦希里语
+- **YO_NG**：约鲁巴语
+- **ZH_CN**：简体中文
+
+### 目的
+MMMLU 排行榜旨在为比较 AI 模型在这些多语言和多领域中的表现提供统一的基准。通过包括 **57 个领域** 中的 **问答任务**，它评估了模型在多语言中回答通用知识问题的能力，确保了多语言理解和推理的高标准。
+
+### 目标
+我们的主要目标是为 AI 模型在不同语言和领域中的表现提供可靠的比较，帮助开发者和研究人员评估和提高他们模型的多语言能力。通过强调高质量的翻译和包括广泛的主题，我们努力使 AI 模型在全球不同社区中更加稳健和有用。
+
+### 🤗 工作原理
+
+在 "提交这里" 标签页上提交模型进行自动评估！
+
+### 📈 任务
+
+我们评估模型在多个关键基准上的表现，重点关注 **多语言大规模多任务语言理解 (MMLU)** 及其变体，包括 MMLU、C-MMLU、阿拉伯语 MMLU、韩语 MMLU、马来语 MMLU 等。 这些基准评估了来自 57 个类别（如法律、物理、历史和计算机科学等）的一般知识。
+
+评估使用 [OpenCompass 框架](https://github.com/open-compass/opencompass) 执行，该平台统一了对语言模型的多任务评估。OpenCompass 使我们能够高效地、大规模地执行这些评估，覆盖多种语言和基准。
+
+有关任务的详细信息，请参见 OpenCompass 框架中的 "任务" 标签页。
+
+注：
+- 所有评估均为 5-shot 任务。
+- 结果采用以下公式标准化：`normalized_value = (raw_value - random_baseline) / (max_value - random_baseline)`，其中 `random_baseline` 对于生成任务为 `0`，对于多选 QA 为 `1/n`（`n` 为选择数）。
+- 结果通过计算给定语言的所有任务的平均值来汇总。
+
+### 🔎 结果
+
+你可以找到：
+
+- 详细的数值结果在 [结果数据集](link_to_results)
+- 社区查询和运行状态在 [请求数据集](link_to_requests)
+
+### ✅ 可复现性
+
+要复现结果，你可以使用 [我们 fork 的 lm_eval](#)，因为并非所有的 PR 都已集成到主仓库中。
+
+## 🙌 致谢
+
+这个排行榜是 [#ProjectName](link_to_project) 项目的一部分，由 [OrganizationName](link_to_organization) 领导，感谢以下机构捐赠了高质量的评估数据集：
+
+- [机构 1](link_to_institution_1)
+- [机构 2](link_to_institution_2)
+- [机构 3](link_to_institution_3)
+- [机构 4](link_to_institution_4)
+- [机构 5](link_to_institution_5)
+- [机构 6](link_to_institution_6)
+- [机构 7](link_to_institution_7)
+- [机构 8](link_to_institution_8)
+- [机构 9](link_to_institution_9)
+
+这些实体按加入项目的时间顺序排列，然而页脚的 logo 排列顺序是按照捐赠数据集的数量。
+
+特别感谢：
+- 任务实现：[姓名 1]，[姓名 2]，[姓名 3]，[姓名 4]，[姓名 5]，[姓名 6]，[姓名 7]，[姓名 8]，[姓名 9]，[姓名 10]
+- 排行榜实现：[姓名 11]，[姓名 12]
+- 模型评估：[姓名 13]，[姓名 14]，[姓名 15]，[姓名 16]，[姓名 17]
+- 沟通：[姓名 18]，[姓名 19]
+- 组织与协作领导：[姓名 20]，[姓名 21]，[姓名 22]，[姓名 23]，[姓名 24]，[姓名 25]，[姓名 26]，[姓名 27]，[姓名 28]，[姓名 29]，[姓名 30]
+
+有关数据集作者的信息，请查看相应的数据集卡片（可以在 "任务" 标签页中找到）以及论文（在 "引用" 部分提供）。我们特别感谢那些为排行榜专门创建或开源其数据集的团队（按时间顺序）：
+- [数据集1 占位符] 和 [数据集2 占位符]： [团队成员占位符]
+- [数据集3 占位符]，[数据集4 占位符] 和 [数据集5 占位符]： [团队成员占位符]
+- [数据集6 占位符]： [团队成员占位符]
+
+我们还感谢 [机构1 占位符]，[机构2 占位符]，[组织占位符]，[人员1 占位符]，[人员2 占位符] 和 [机构3 占位符] 提供推理 GPU 支持。
+
+## 🚀 合作！
+
+我们希望创建一个尽可能多样化的排行榜，欢迎联系我们如果你希望我们将你的评估数据集包含在内！
+
+评论和建议非常欢迎！请访问 [👏 社区](<Community Page Placeholder>) 页面，告诉我们你对 La 排行榜的看法以及我们如何改进，或者直接打开一个 PR！
+
+非常感谢！ 💛
+"""
+
 EVALUATION_QUEUE_TEXT = """
 ## Some good practices before submitting a model
 
@@ -70,3 +294,38 @@ If everything is done, check you can launch the EleutherAIHarness on your model
 CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
 CITATION_BUTTON_TEXT = r"""
 """
+EVALUATION_QUEUE_TEXT_ZH = """
+## 提交模型前的一些良好实践
+
+### 1) 确保你可以使用 AutoClasses 加载你的模型和分词器：
+```python
+from transformers import AutoConfig, AutoModel, AutoTokenizer
+config = AutoConfig.from_pretrained("your model name", revision=revision)
+model = AutoModel.from_pretrained("your model name", revision=revision)
+tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
+```
+如果此步骤失败，请按照错误信息进行调试，可能是你的模型上传不正确。
+
+注意：确保你的模型是公开的！ 注意：如果你的模型需要 use_remote_code=True，目前我们不支持该选项，但我们正在努力添加此功能，请保持关注！
+
+2) 将你的模型权重转换为 safetensors
+这是一个新的权重存储格式，加载和使用时更安全、更快速。它还将允许我们将模型的参数数量添加到 Extended Viewer 中！
+
+3) 确保你的模型具有开放许可！
+这是一个针对开放 LLM 的排行榜，我们希望尽可能多的人知道他们可以使用你的模型 🤗
+
+4) 填写你的模型卡
+当我们将额外的信息添加到排行榜时，它将自动从模型卡中获取。
+
+模型失败时的处理
+如果你的模型出现在 FAILED 分类中，表示其执行停止。 首先确保你已经遵循了上述步骤。 如果一切都完成，检查你是否可以使用上面的命令在本地启动 EleutherAIHarness 来测试你的模型（你可以添加 --limit 来限制每个任务的示例数）。 """
+
+CITATION_BUTTON_LABEL = "复制以下代码引用这些结果"
+CITATION_BUTTON_TEXT = r"""
+"""
+LOGOS = [
+    "logo/HuggingFace.png",
+    "logo/openai-logo.png",
+    "logo/logo_qwen.png",
+    "logo/CAIS.png"
+]

src/envs.py

@@ -6,15 +6,17 @@ from huggingface_hub import HfApi
 # ----------------------------------
 TOKEN = os.environ.get("HF_TOKEN") # A read/write token for your org
 
-OWNER = "demo-leaderboard-backend" # Change to your org - don't forget to create a results and request dataset, with the correct format!
+OWNER = "Multilingual-MMLU-Benchmark-Leaderboard" # Change to your org - don't forget to create a results and request dataset, with the correct format!
+DEVICE = "cpu"  # "cuda:0" if you add compute
 # ----------------------------------
 
-REPO_ID = f"{OWNER}/leaderboard"
+REPO_ID = f"{OWNER}/Multilingual-MMLU-Benchmark-Leaderboard"
 QUEUE_REPO = f"{OWNER}/requests"
 RESULTS_REPO = f"{OWNER}/results"
 
 # If you setup a cache later, just change HF_HOME
 CACHE_PATH=os.getenv("HF_HOME", ".")
+# print('*******',CACHE_PATH)
 
 # Local caches
 EVAL_REQUESTS_PATH = os.path.join(CACHE_PATH, "eval-queue")

src/leaderboard/read_evals.py

@@ -10,7 +10,9 @@ import numpy as np
 from src.display.formatting import make_clickable_model
 from src.display.utils import AutoEvalColumn, ModelType, Tasks, Precision, WeightType
 from src.submission.check_validity import is_model_on_hub
+# from src.tasks import Categories
 
+# tasks_df = pd.read_csv("tasks.csv")
 
 @dataclass
 class EvalResult:
@@ -18,14 +20,14 @@ class EvalResult:
     """
     eval_name: str # org_model_precision (uid)
     full_model: str # org/model (path on hub)
-    org: str 
+    org: str
     model: str
     revision: str # commit hash, "" if main
     results: dict
     precision: Precision = Precision.Unknown
     model_type: ModelType = ModelType.Unknown # Pretrained, fine tuned, ...
     weight_type: WeightType = WeightType.Original # Original or Adapter
-    architecture: str = "Unknown" 
+    architecture: str = "Unknown"
     license: str = "?"
     likes: int = 0
     num_params: int = 0
@@ -37,6 +39,7 @@ class EvalResult:
         """Inits the result from the specific model result file"""
         with open(json_filepath) as fp:
             data = json.load(fp)
+            # print('#####',json_filepath)
 
         config = data.get("config")
 
@@ -70,7 +73,6 @@ class EvalResult:
         results = {}
         for task in Tasks:
             task = task.value
-
             # We average all scores of a given metric (not all metrics are present in all files)
             accs = np.array([v.get(task.metric, None) for k, v in data["results"].items() if task.benchmark == k])
             if accs.size == 0 or any([acc is None for acc in accs]):
@@ -85,7 +87,7 @@ class EvalResult:
             org=org,
             model=model,
             results=results,
-            precision=precision,  
+            precision=precision,
             revision= config.get("model_sha", ""),
             still_on_hub=still_on_hub,
             architecture=architecture
@@ -93,7 +95,8 @@ class EvalResult:
 
     def update_with_request_file(self, requests_path):
         """Finds the relevant request file for the current model and updates info with it"""
-        request_file = get_request_file_for_model(requests_path, self.full_model, self.precision.value.name)
+        request_file = get_request_file_for_model(requests_path, self.full_model.split("/")[-1], self.precision.value.name)
+        # print("########",request_file)
 
         try:
             with open(request_file, "r") as f:
@@ -109,7 +112,9 @@ class EvalResult:
 
     def to_dict(self):
         """Converts the Eval Result to a dict compatible with our dataframe display"""
-        average = sum([v for v in self.results.values() if v is not None]) / len(Tasks)
+        keys_to_average = ['mmmlu', 'mmlu', 'cmmlu']
+        average = sum([self.results[key] for key in keys_to_average if self.results.get(key) is not None]) / len(
+            keys_to_average)
         data_dict = {
             "eval_name": self.eval_name,  # not a column, just a save name,
             AutoEvalColumn.precision.name: self.precision.value.name,
@@ -136,6 +141,7 @@ def get_request_file_for_model(requests_path, model_name, precision):
     """Selects the correct request file for a given model. Only keeps runs tagged as FINISHED"""
     request_files = os.path.join(
         requests_path,
+        model_name,
         f"{model_name}_eval_request_*.json",
     )
     request_files = glob.glob(request_files)