File size: 111,076 Bytes
9877e08 |
1 2 |
{"cells":[{"cell_type":"markdown","metadata":{"id":"Ac6wadk3rmkK"},"source":["# LM Evaluation Harness (by [EleutherAI](https://www.eleuther.ai/))\n","\n","This [`LM-Evaluation-Harness`](https://github.com/EleutherAI/lm-evaluation-harness) provides a unified framework to test generative language models on a large number of different evaluation tasks. For a complete list of available tasks, see the [task table](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md), or scroll to the bottom of the page.\n","\n","1. Clone the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and install the necessary libraries (`sentencepiece` is required for the Llama tokenizer)."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":40508,"status":"ok","timestamp":1698580809187,"user":{"displayName":"Nicholas Corrêa","userId":"09736120585766268588"},"user_tz":-60},"id":"UA5I86u91e0A","outputId":"2342bf64-d93b-441f-8643-8e4003c6ef6c"},"outputs":[],"source":["!git clone --branch master https://github.com/EleutherAI/lm-evaluation-harness\n","!cd lm-evaluation-harness && pip install -e . -q\n","!pip install cohere tiktoken sentencepiece -q"]},{"cell_type":"markdown","metadata":{},"source":["2. Run the evaluation harness on the selected tasks."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":1753416,"status":"ok","timestamp":1698583348574,"user":{"displayName":"Nicholas Corrêa","userId":"09736120585766268588"},"user_tz":-60},"id":"pnHoAVK25QZn","outputId":"23f65f99-82f8-423f-9c8a-b1d4f2bdbd56"},"outputs":[],"source":["!huggingface-cli login --token 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\n","!cd lm-evaluation-harness && python main.py \\\n"," --model hf-causal \\\n"," --model_args pretrained=nicholasKluge/Aira-OPT-1B3 \\\n"," --tasks arc_challenge,truthfulqa_mc,toxigen \\\n"," --device cuda:0"]},{"cell_type":"markdown","metadata":{"id":"4Bm78wiZ4Own"},"source":["## Task Table 📚\n","\n","| Task Name |Train|Val|Test|Val/Test Docs| Metrics |\n","|---------------------------------------------------------|-----|---|----|------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n","|anagrams1 | |✓ | | 10000|acc |\n","|anagrams2 | |✓ | | 10000|acc |\n","|anli_r1 |✓ |✓ |✓ | 1000|acc |\n","|anli_r2 |✓ |✓ |✓ | 1000|acc |\n","|anli_r3 |✓ |✓ |✓ | 1200|acc |\n","|arc_challenge |✓ |✓ |✓ | 1172|acc, acc_norm |\n","|arc_easy |✓ |✓ |✓ | 2376|acc, acc_norm |\n","|arithmetic_1dc | |✓ | | 2000|acc |\n","|arithmetic_2da | |✓ | | 2000|acc |\n","|arithmetic_2dm | |✓ | | 2000|acc |\n","|arithmetic_2ds | |✓ | | 2000|acc |\n","|arithmetic_3da | |✓ | | 2000|acc |\n","|arithmetic_3ds | |✓ | | 2000|acc |\n","|arithmetic_4da | |✓ | | 2000|acc |\n","|arithmetic_4ds | |✓ | | 2000|acc |\n","|arithmetic_5da | |✓ | | 2000|acc |\n","|arithmetic_5ds | |✓ | | 2000|acc |\n","|bigbench_causal_judgement | | |✓ | 190|multiple_choice_grade, exact_str_match |\n","|bigbench_date_understanding | | |✓ | 369|multiple_choice_grade, exact_str_match |\n","|bigbench_disambiguation_qa | | |✓ | 258|multiple_choice_grade, exact_str_match |\n","|bigbench_dyck_languages | | |✓ | 1000|multiple_choice_grade, exact_str_match |\n","|bigbench_formal_fallacies_syllogisms_negation | | |✓ | 14200|multiple_choice_grade, exact_str_match |\n","|bigbench_geometric_shapes | | |✓ | 359|multiple_choice_grade, exact_str_match |\n","|bigbench_hyperbaton | | |✓ | 50000|multiple_choice_grade, exact_str_match |\n","|bigbench_logical_deduction_five_objects | | |✓ | 500|multiple_choice_grade, exact_str_match |\n","|bigbench_logical_deduction_seven_objects | | |✓ | 700|multiple_choice_grade, exact_str_match |\n","|bigbench_logical_deduction_three_objects | | |✓ | 300|multiple_choice_grade, exact_str_match |\n","|bigbench_movie_recommendation | | |✓ | 500|multiple_choice_grade, exact_str_match |\n","|bigbench_navigate | | |✓ | 1000|multiple_choice_grade, exact_str_match |\n","|bigbench_reasoning_about_colored_objects | | |✓ | 2000|multiple_choice_grade, exact_str_match |\n","|bigbench_ruin_names | | |✓ | 448|multiple_choice_grade, exact_str_match |\n","|bigbench_salient_translation_error_detection | | |✓ | 998|multiple_choice_grade, exact_str_match |\n","|bigbench_snarks | | |✓ | 181|multiple_choice_grade, exact_str_match |\n","|bigbench_sports_understanding | | |✓ | 986|multiple_choice_grade, exact_str_match |\n","|bigbench_temporal_sequences | | |✓ | 1000|multiple_choice_grade, exact_str_match |\n","|bigbench_tracking_shuffled_objects_five_objects | | |✓ | 1250|multiple_choice_grade, exact_str_match |\n","|bigbench_tracking_shuffled_objects_seven_objects | | |✓ | 1750|multiple_choice_grade, exact_str_match |\n","|bigbench_tracking_shuffled_objects_three_objects | | |✓ | 300|multiple_choice_grade, exact_str_match |\n","|blimp_adjunct_island | |✓ | | 1000|acc |\n","|blimp_anaphor_gender_agreement | |✓ | | 1000|acc |\n","|blimp_anaphor_number_agreement | |✓ | | 1000|acc |\n","|blimp_animate_subject_passive | |✓ | | 1000|acc |\n","|blimp_animate_subject_trans | |✓ | | 1000|acc |\n","|blimp_causative | |✓ | | 1000|acc |\n","|blimp_complex_NP_island | |✓ | | 1000|acc |\n","|blimp_coordinate_structure_constraint_complex_left_branch| |✓ | | 1000|acc |\n","|blimp_coordinate_structure_constraint_object_extraction | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_1 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_2 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_irregular_1 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_irregular_2 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_with_adj_2 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_with_adj_irregular_1 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_with_adj_irregular_2 | |✓ | | 1000|acc |\n","|blimp_determiner_noun_agreement_with_adjective_1 | |✓ | | 1000|acc |\n","|blimp_distractor_agreement_relational_noun | |✓ | | 1000|acc |\n","|blimp_distractor_agreement_relative_clause | |✓ | | 1000|acc |\n","|blimp_drop_argument | |✓ | | 1000|acc |\n","|blimp_ellipsis_n_bar_1 | |✓ | | 1000|acc |\n","|blimp_ellipsis_n_bar_2 | |✓ | | 1000|acc |\n","|blimp_existential_there_object_raising | |✓ | | 1000|acc |\n","|blimp_existential_there_quantifiers_1 | |✓ | | 1000|acc |\n","|blimp_existential_there_quantifiers_2 | |✓ | | 1000|acc |\n","|blimp_existential_there_subject_raising | |✓ | | 1000|acc |\n","|blimp_expletive_it_object_raising | |✓ | | 1000|acc |\n","|blimp_inchoative | |✓ | | 1000|acc |\n","|blimp_intransitive | |✓ | | 1000|acc |\n","|blimp_irregular_past_participle_adjectives | |✓ | | 1000|acc |\n","|blimp_irregular_past_participle_verbs | |✓ | | 1000|acc |\n","|blimp_irregular_plural_subject_verb_agreement_1 | |✓ | | 1000|acc |\n","|blimp_irregular_plural_subject_verb_agreement_2 | |✓ | | 1000|acc |\n","|blimp_left_branch_island_echo_question | |✓ | | 1000|acc |\n","|blimp_left_branch_island_simple_question | |✓ | | 1000|acc |\n","|blimp_matrix_question_npi_licensor_present | |✓ | | 1000|acc |\n","|blimp_npi_present_1 | |✓ | | 1000|acc |\n","|blimp_npi_present_2 | |✓ | | 1000|acc |\n","|blimp_only_npi_licensor_present | |✓ | | 1000|acc |\n","|blimp_only_npi_scope | |✓ | | 1000|acc |\n","|blimp_passive_1 | |✓ | | 1000|acc |\n","|blimp_passive_2 | |✓ | | 1000|acc |\n","|blimp_principle_A_c_command | |✓ | | 1000|acc |\n","|blimp_principle_A_case_1 | |✓ | | 1000|acc |\n","|blimp_principle_A_case_2 | |✓ | | 1000|acc |\n","|blimp_principle_A_domain_1 | |✓ | | 1000|acc |\n","|blimp_principle_A_domain_2 | |✓ | | 1000|acc |\n","|blimp_principle_A_domain_3 | |✓ | | 1000|acc |\n","|blimp_principle_A_reconstruction | |✓ | | 1000|acc |\n","|blimp_regular_plural_subject_verb_agreement_1 | |✓ | | 1000|acc |\n","|blimp_regular_plural_subject_verb_agreement_2 | |✓ | | 1000|acc |\n","|blimp_sentential_negation_npi_licensor_present | |✓ | | 1000|acc |\n","|blimp_sentential_negation_npi_scope | |✓ | | 1000|acc |\n","|blimp_sentential_subject_island | |✓ | | 1000|acc |\n","|blimp_superlative_quantifiers_1 | |✓ | | 1000|acc |\n","|blimp_superlative_quantifiers_2 | |✓ | | 1000|acc |\n","|blimp_tough_vs_raising_1 | |✓ | | 1000|acc |\n","|blimp_tough_vs_raising_2 | |✓ | | 1000|acc |\n","|blimp_transitive | |✓ | | 1000|acc |\n","|blimp_wh_island | |✓ | | 1000|acc |\n","|blimp_wh_questions_object_gap | |✓ | | 1000|acc |\n","|blimp_wh_questions_subject_gap | |✓ | | 1000|acc |\n","|blimp_wh_questions_subject_gap_long_distance | |✓ | | 1000|acc |\n","|blimp_wh_vs_that_no_gap | |✓ | | 1000|acc |\n","|blimp_wh_vs_that_no_gap_long_distance | |✓ | | 1000|acc |\n","|blimp_wh_vs_that_with_gap | |✓ | | 1000|acc |\n","|blimp_wh_vs_that_with_gap_long_distance | |✓ | | 1000|acc |\n","|boolq |✓ |✓ | | 3270|acc |\n","|cb |✓ |✓ | | 56|acc, f1 |\n","|cola |✓ |✓ | | 1043|mcc |\n","|copa |✓ |✓ | | 100|acc |\n","|coqa |✓ |✓ | | 500|f1, em |\n","|crows_pairs_english | |✓ | | 1677|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_age | |✓ | | 91|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_autre | |✓ | | 11|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_disability | |✓ | | 65|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_gender | |✓ | | 320|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_nationality | |✓ | | 216|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_physical_appearance | |✓ | | 72|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_race_color | |✓ | | 508|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_religion | |✓ | | 111|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_sexual_orientation | |✓ | | 93|likelihood_difference, pct_stereotype |\n","|crows_pairs_english_socioeconomic | |✓ | | 190|likelihood_difference, pct_stereotype |\n","|crows_pairs_french | |✓ | | 1677|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_age | |✓ | | 90|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_autre | |✓ | | 13|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_disability | |✓ | | 66|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_gender | |✓ | | 321|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_nationality | |✓ | | 253|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_physical_appearance | |✓ | | 72|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_race_color | |✓ | | 460|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_religion | |✓ | | 115|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_sexual_orientation | |✓ | | 91|likelihood_difference, pct_stereotype |\n","|crows_pairs_french_socioeconomic | |✓ | | 196|likelihood_difference, pct_stereotype |\n","|cycle_letters | |✓ | | 10000|acc |\n","|drop |✓ |✓ | | 9536|em, f1 |\n","|ethics_cm |✓ | |✓ | 3885|acc |\n","|ethics_deontology |✓ | |✓ | 3596|acc, em |\n","|ethics_justice |✓ | |✓ | 2704|acc, em |\n","|ethics_utilitarianism |✓ | |✓ | 4808|acc |\n","|ethics_utilitarianism_original | | |✓ | 4808|acc |\n","|ethics_virtue |✓ | |✓ | 4975|acc, em |\n","|gsm8k |✓ | |✓ | 1319|acc |\n","|headqa |✓ |✓ |✓ | 2742|acc, acc_norm |\n","|headqa_en |✓ |✓ |✓ | 2742|acc, acc_norm |\n","|headqa_es |✓ |✓ |✓ | 2742|acc, acc_norm |\n","|hellaswag |✓ |✓ | | 10042|acc, acc_norm |\n","|hendrycksTest-abstract_algebra | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-anatomy | |✓ |✓ | 135|acc, acc_norm |\n","|hendrycksTest-astronomy | |✓ |✓ | 152|acc, acc_norm |\n","|hendrycksTest-business_ethics | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-clinical_knowledge | |✓ |✓ | 265|acc, acc_norm |\n","|hendrycksTest-college_biology | |✓ |✓ | 144|acc, acc_norm |\n","|hendrycksTest-college_chemistry | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-college_computer_science | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-college_mathematics | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-college_medicine | |✓ |✓ | 173|acc, acc_norm |\n","|hendrycksTest-college_physics | |✓ |✓ | 102|acc, acc_norm |\n","|hendrycksTest-computer_security | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-conceptual_physics | |✓ |✓ | 235|acc, acc_norm |\n","|hendrycksTest-econometrics | |✓ |✓ | 114|acc, acc_norm |\n","|hendrycksTest-electrical_engineering | |✓ |✓ | 145|acc, acc_norm |\n","|hendrycksTest-elementary_mathematics | |✓ |✓ | 378|acc, acc_norm |\n","|hendrycksTest-formal_logic | |✓ |✓ | 126|acc, acc_norm |\n","|hendrycksTest-global_facts | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-high_school_biology | |✓ |✓ | 310|acc, acc_norm |\n","|hendrycksTest-high_school_chemistry | |✓ |✓ | 203|acc, acc_norm |\n","|hendrycksTest-high_school_computer_science | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-high_school_european_history | |✓ |✓ | 165|acc, acc_norm |\n","|hendrycksTest-high_school_geography | |✓ |✓ | 198|acc, acc_norm |\n","|hendrycksTest-high_school_government_and_politics | |✓ |✓ | 193|acc, acc_norm |\n","|hendrycksTest-high_school_macroeconomics | |✓ |✓ | 390|acc, acc_norm |\n","|hendrycksTest-high_school_mathematics | |✓ |✓ | 270|acc, acc_norm |\n","|hendrycksTest-high_school_microeconomics | |✓ |✓ | 238|acc, acc_norm |\n","|hendrycksTest-high_school_physics | |✓ |✓ | 151|acc, acc_norm |\n","|hendrycksTest-high_school_psychology | |✓ |✓ | 545|acc, acc_norm |\n","|hendrycksTest-high_school_statistics | |✓ |✓ | 216|acc, acc_norm |\n","|hendrycksTest-high_school_us_history | |✓ |✓ | 204|acc, acc_norm |\n","|hendrycksTest-high_school_world_history | |✓ |✓ | 237|acc, acc_norm |\n","|hendrycksTest-human_aging | |✓ |✓ | 223|acc, acc_norm |\n","|hendrycksTest-human_sexuality | |✓ |✓ | 131|acc, acc_norm |\n","|hendrycksTest-international_law | |✓ |✓ | 121|acc, acc_norm |\n","|hendrycksTest-jurisprudence | |✓ |✓ | 108|acc, acc_norm |\n","|hendrycksTest-logical_fallacies | |✓ |✓ | 163|acc, acc_norm |\n","|hendrycksTest-machine_learning | |✓ |✓ | 112|acc, acc_norm |\n","|hendrycksTest-management | |✓ |✓ | 103|acc, acc_norm |\n","|hendrycksTest-marketing | |✓ |✓ | 234|acc, acc_norm |\n","|hendrycksTest-medical_genetics | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-miscellaneous | |✓ |✓ | 783|acc, acc_norm |\n","|hendrycksTest-moral_disputes | |✓ |✓ | 346|acc, acc_norm |\n","|hendrycksTest-moral_scenarios | |✓ |✓ | 895|acc, acc_norm |\n","|hendrycksTest-nutrition | |✓ |✓ | 306|acc, acc_norm |\n","|hendrycksTest-philosophy | |✓ |✓ | 311|acc, acc_norm |\n","|hendrycksTest-prehistory | |✓ |✓ | 324|acc, acc_norm |\n","|hendrycksTest-professional_accounting | |✓ |✓ | 282|acc, acc_norm |\n","|hendrycksTest-professional_law | |✓ |✓ | 1534|acc, acc_norm |\n","|hendrycksTest-professional_medicine | |✓ |✓ | 272|acc, acc_norm |\n","|hendrycksTest-professional_psychology | |✓ |✓ | 612|acc, acc_norm |\n","|hendrycksTest-public_relations | |✓ |✓ | 110|acc, acc_norm |\n","|hendrycksTest-security_studies | |✓ |✓ | 245|acc, acc_norm |\n","|hendrycksTest-sociology | |✓ |✓ | 201|acc, acc_norm |\n","|hendrycksTest-us_foreign_policy | |✓ |✓ | 100|acc, acc_norm |\n","|hendrycksTest-virology | |✓ |✓ | 166|acc, acc_norm |\n","|hendrycksTest-world_religions | |✓ |✓ | 171|acc, acc_norm |\n","|iwslt17-ar-en | | |✓ | 1460|bleu, chrf, ter |\n","|iwslt17-en-ar | | |✓ | 1460|bleu, chrf, ter |\n","|lambada_openai | | |✓ | 5153|ppl, acc |\n","|lambada_openai_cloze | | |✓ | 5153|ppl, acc |\n","|lambada_openai_mt_de | | |✓ | 5153|ppl, acc |\n","|lambada_openai_mt_en | | |✓ | 5153|ppl, acc |\n","|lambada_openai_mt_es | | |✓ | 5153|ppl, acc |\n","|lambada_openai_mt_fr | | |✓ | 5153|ppl, acc |\n","|lambada_openai_mt_it | | |✓ | 5153|ppl, acc |\n","|lambada_standard | |✓ |✓ | 5153|ppl, acc |\n","|lambada_standard_cloze | |✓ |✓ | 5153|ppl, acc |\n","|logiqa |✓ |✓ |✓ | 651|acc, acc_norm |\n","|math_algebra |✓ | |✓ | 1187|acc |\n","|math_asdiv | |✓ | | 2305|acc |\n","|math_counting_and_prob |✓ | |✓ | 474|acc |\n","|math_geometry |✓ | |✓ | 479|acc |\n","|math_intermediate_algebra |✓ | |✓ | 903|acc |\n","|math_num_theory |✓ | |✓ | 540|acc |\n","|math_prealgebra |✓ | |✓ | 871|acc |\n","|math_precalc |✓ | |✓ | 546|acc |\n","|mathqa |✓ |✓ |✓ | 2985|acc, acc_norm |\n","|mc_taco | |✓ |✓ | 9442|f1, em |\n","|mgsm_bn |✓ | |✓ | 250|acc |\n","|mgsm_de |✓ | |✓ | 250|acc |\n","|mgsm_en |✓ | |✓ | 250|acc |\n","|mgsm_es |✓ | |✓ | 250|acc |\n","|mgsm_fr |✓ | |✓ | 250|acc |\n","|mgsm_ja |✓ | |✓ | 250|acc |\n","|mgsm_ru |✓ | |✓ | 250|acc |\n","|mgsm_sw |✓ | |✓ | 250|acc |\n","|mgsm_te |✓ | |✓ | 250|acc |\n","|mgsm_th |✓ | |✓ | 250|acc |\n","|mgsm_zh |✓ | |✓ | 250|acc |\n","|mnli |✓ |✓ | | 9815|acc |\n","|mnli_mismatched |✓ |✓ | | 9832|acc |\n","|mrpc |✓ |✓ | | 408|acc, f1 |\n","|multirc |✓ |✓ | | 4848|acc |\n","|mutual |✓ |✓ | | 886|r@1, r@2, mrr |\n","|mutual_plus |✓ |✓ | | 886|r@1, r@2, mrr |\n","|openbookqa |✓ |✓ |✓ | 500|acc, acc_norm |\n","|pawsx_de |✓ |✓ |✓ | 2000|acc |\n","|pawsx_en |✓ |✓ |✓ | 2000|acc |\n","|pawsx_es |✓ |✓ |✓ | 2000|acc |\n","|pawsx_fr |✓ |✓ |✓ | 2000|acc |\n","|pawsx_ja |✓ |✓ |✓ | 2000|acc |\n","|pawsx_ko |✓ |✓ |✓ | 2000|acc |\n","|pawsx_zh |✓ |✓ |✓ | 2000|acc |\n","|pile_arxiv | |✓ |✓ | 2407|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_bookcorpus2 | |✓ |✓ | 28|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_books3 | |✓ |✓ | 269|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_dm-mathematics | |✓ |✓ | 1922|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_enron | |✓ |✓ | 1010|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_europarl | |✓ |✓ | 157|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_freelaw | |✓ |✓ | 5101|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_github | |✓ |✓ | 18195|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_gutenberg | |✓ |✓ | 80|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_hackernews | |✓ |✓ | 1632|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_nih-exporter | |✓ |✓ | 1884|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_opensubtitles | |✓ |✓ | 642|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_openwebtext2 | |✓ |✓ | 32925|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_philpapers | |✓ |✓ | 68|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_pile-cc | |✓ |✓ | 52790|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_pubmed-abstracts | |✓ |✓ | 29895|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_pubmed-central | |✓ |✓ | 5911|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_stackexchange | |✓ |✓ | 30378|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_ubuntu-irc | |✓ |✓ | 22|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_uspto | |✓ |✓ | 11415|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_wikipedia | |✓ |✓ | 17511|word_perplexity, byte_perplexity, bits_per_byte |\n","|pile_youtubesubtitles | |✓ |✓ | 342|word_perplexity, byte_perplexity, bits_per_byte |\n","|piqa |✓ |✓ | | 1838|acc, acc_norm |\n","|prost | | |✓ | 18736|acc, acc_norm |\n","|pubmedqa | | |✓ | 1000|acc |\n","|qa4mre_2011 | | |✓ | 120|acc, acc_norm |\n","|qa4mre_2012 | | |✓ | 160|acc, acc_norm |\n","|qa4mre_2013 | | |✓ | 284|acc, acc_norm |\n","|qasper |✓ |✓ | | 1764|f1_yesno, f1_abstractive |\n","|qnli |✓ |✓ | | 5463|acc |\n","|qqp |✓ |✓ | | 40430|acc, f1 |\n","|race |✓ |✓ |✓ | 1045|acc |\n","|random_insertion | |✓ | | 10000|acc |\n","|record |✓ |✓ | | 10000|f1, em |\n","|reversed_words | |✓ | | 10000|acc |\n","|rte |✓ |✓ | | 277|acc |\n","|sciq |✓ |✓ |✓ | 1000|acc, acc_norm |\n","|scrolls_contractnli |✓ |✓ | | 1037|em, acc, acc_norm |\n","|scrolls_govreport |✓ |✓ | | 972|rouge1, rouge2, rougeL |\n","|scrolls_narrativeqa |✓ |✓ | | 3425|f1 |\n","|scrolls_qasper |✓ |✓ | | 984|f1 |\n","|scrolls_qmsum |✓ |✓ | | 272|rouge1, rouge2, rougeL |\n","|scrolls_quality |✓ |✓ | | 2086|em, acc, acc_norm |\n","|scrolls_summscreenfd |✓ |✓ | | 338|rouge1, rouge2, rougeL |\n","|squad2 |✓ |✓ | | 11873|exact, f1, HasAns_exact, HasAns_f1, NoAns_exact, NoAns_f1, best_exact, best_f1 |\n","|sst |✓ |✓ | | 872|acc |\n","|swag |✓ |✓ | | 20006|acc, acc_norm |\n","|toxigen |✓ | |✓ | 940|acc, acc_norm |\n","|triviaqa |✓ |✓ | | 11313|acc |\n","|truthfulqa_gen | |✓ | | 817|bleurt_max, bleurt_acc, bleurt_diff, bleu_max, bleu_acc, bleu_diff, rouge1_max, rouge1_acc, rouge1_diff, rouge2_max, rouge2_acc, rouge2_diff, rougeL_max, rougeL_acc, rougeL_diff|\n","|truthfulqa_mc | |✓ | | 817|mc1, mc2 |\n","|webqs |✓ | |✓ | 2032|acc |\n","|wic |✓ |✓ | | 638|acc |\n","|wikitext |✓ |✓ |✓ | 62|word_perplexity, byte_perplexity, bits_per_byte |\n","|winogrande |✓ |✓ | | 1267|acc |\n","|wmt14-en-fr | | |✓ | 3003|bleu, chrf, ter |\n","|wmt14-fr-en | | |✓ | 3003|bleu, chrf, ter |\n","|wmt16-de-en | | |✓ | 2999|bleu, chrf, ter |\n","|wmt16-en-de | | |✓ | 2999|bleu, chrf, ter |\n","|wmt16-en-ro | | |✓ | 1999|bleu, chrf, ter |\n","|wmt16-ro-en | | |✓ | 1999|bleu, chrf, ter |\n","|wmt20-cs-en | | |✓ | 664|bleu, chrf, ter |\n","|wmt20-de-en | | |✓ | 785|bleu, chrf, ter |\n","|wmt20-de-fr | | |✓ | 1619|bleu, chrf, ter |\n","|wmt20-en-cs | | |✓ | 1418|bleu, chrf, ter |\n","|wmt20-en-de | | |✓ | 1418|bleu, chrf, ter |\n","|wmt20-en-iu | | |✓ | 2971|bleu, chrf, ter |\n","|wmt20-en-ja | | |✓ | 1000|bleu, chrf, ter |\n","|wmt20-en-km | | |✓ | 2320|bleu, chrf, ter |\n","|wmt20-en-pl | | |✓ | 1000|bleu, chrf, ter |\n","|wmt20-en-ps | | |✓ | 2719|bleu, chrf, ter |\n","|wmt20-en-ru | | |✓ | 2002|bleu, chrf, ter |\n","|wmt20-en-ta | | |✓ | 1000|bleu, chrf, ter |\n","|wmt20-en-zh | | |✓ | 1418|bleu, chrf, ter |\n","|wmt20-fr-de | | |✓ | 1619|bleu, chrf, ter |\n","|wmt20-iu-en | | |✓ | 2971|bleu, chrf, ter |\n","|wmt20-ja-en | | |✓ | 993|bleu, chrf, ter |\n","|wmt20-km-en | | |✓ | 2320|bleu, chrf, ter |\n","|wmt20-pl-en | | |✓ | 1001|bleu, chrf, ter |\n","|wmt20-ps-en | | |✓ | 2719|bleu, chrf, ter |\n","|wmt20-ru-en | | |✓ | 991|bleu, chrf, ter |\n","|wmt20-ta-en | | |✓ | 997|bleu, chrf, ter |\n","|wmt20-zh-en | | |✓ | 2000|bleu, chrf, ter |\n","|wnli |✓ |✓ | | 71|acc |\n","|wsc |✓ |✓ | | 104|acc |\n","|wsc273 | | |✓ | 273|acc |\n","|xcopa_et | |✓ |✓ | 500|acc |\n","|xcopa_ht | |✓ |✓ | 500|acc |\n","|xcopa_id | |✓ |✓ | 500|acc |\n","|xcopa_it | |✓ |✓ | 500|acc |\n","|xcopa_qu | |✓ |✓ | 500|acc |\n","|xcopa_sw | |✓ |✓ | 500|acc |\n","|xcopa_ta | |✓ |✓ | 500|acc |\n","|xcopa_th | |✓ |✓ | 500|acc |\n","|xcopa_tr | |✓ |✓ | 500|acc |\n","|xcopa_vi | |✓ |✓ | 500|acc |\n","|xcopa_zh | |✓ |✓ | 500|acc |\n","|xnli_ar |✓ |✓ |✓ | 5010|acc |\n","|xnli_bg |✓ |✓ |✓ | 5010|acc |\n","|xnli_de |✓ |✓ |✓ | 5010|acc |\n","|xnli_el |✓ |✓ |✓ | 5010|acc |\n","|xnli_en |✓ |✓ |✓ | 5010|acc |\n","|xnli_es |✓ |✓ |✓ | 5010|acc |\n","|xnli_fr |✓ |✓ |✓ | 5010|acc |\n","|xnli_hi |✓ |✓ |✓ | 5010|acc |\n","|xnli_ru |✓ |✓ |✓ | 5010|acc |\n","|xnli_sw |✓ |✓ |✓ | 5010|acc |\n","|xnli_th |✓ |✓ |✓ | 5010|acc |\n","|xnli_tr |✓ |✓ |✓ | 5010|acc |\n","|xnli_ur |✓ |✓ |✓ | 5010|acc |\n","|xnli_vi |✓ |✓ |✓ | 5010|acc |\n","|xnli_zh |✓ |✓ |✓ | 5010|acc |\n","|xstory_cloze_ar |✓ |✓ | | 1511|acc |\n","|xstory_cloze_en |✓ |✓ | | 1511|acc |\n","|xstory_cloze_es |✓ |✓ | | 1511|acc |\n","|xstory_cloze_eu |✓ |✓ | | 1511|acc |\n","|xstory_cloze_hi |✓ |✓ | | 1511|acc |\n","|xstory_cloze_id |✓ |✓ | | 1511|acc |\n","|xstory_cloze_my |✓ |✓ | | 1511|acc |\n","|xstory_cloze_ru |✓ |✓ | | 1511|acc |\n","|xstory_cloze_sw |✓ |✓ | | 1511|acc |\n","|xstory_cloze_te |✓ |✓ | | 1511|acc |\n","|xstory_cloze_zh |✓ |✓ | | 1511|acc |\n","|xwinograd_en | | |✓ | 2325|acc |\n","|xwinograd_fr | | |✓ | 83|acc |\n","|xwinograd_jp | | |✓ | 959|acc |\n","|xwinograd_pt | | |✓ | 263|acc |\n","|xwinograd_ru | | |✓ | 315|acc |\n","|xwinograd_zh | | |✓ | 504|acc |\n","| Ceval-valid-computer_network | | ✓ | | 19 | acc |\n","| Ceval-valid-operating_system | | ✓ | | 19 | acc |\n","| Ceval-valid-computer_architecture | | ✓ | | 21 | acc |\n","| Ceval-valid-college_programming | | ✓ | | 37 | acc |\n","| Ceval-valid-college_physics | | ✓ | | 19 | acc |\n","| Ceval-valid-college_chemistry | | ✓ | | 24 | acc |\n","| Ceval-valid-advanced_mathematics | | ✓ | | 19 | acc |\n","| Ceval-valid-probability_and_statistics | | ✓ | | 18 | acc |\n","| Ceval-valid-discrete_mathematics | | ✓ | | 16 | acc |\n","| Ceval-valid-electrical_engineer | | ✓ | | 37 | acc |\n","| Ceval-valid-metrology_engineer | | ✓ | | 24 | acc |\n","| Ceval-valid-high_school_mathematics | | ✓ | | 18 | acc |\n","| Ceval-valid-high_school_physics | | ✓ | | 19 | acc |\n","| Ceval-valid-high_school_chemistry | | ✓ | | 19 | acc |\n","| Ceval-valid-high_school_biology | | ✓ | | 19 | acc |\n","| Ceval-valid-middle_school_mathematics | | ✓ | | 19 | acc |\n","| Ceval-valid-middle_school_biology | | ✓ | | 21 | acc |\n","| Ceval-valid-middle_school_physics | | ✓ | | 19 | acc |\n","| Ceval-valid-middle_school_chemistry | | ✓ | | 20 | acc |\n","| Ceval-valid-veterinary_medicine | | ✓ | | 23 | acc |\n","| Ceval-valid-college_economics | | ✓ | | 55 | acc |\n","| Ceval-valid-business_administration | | ✓ | | 33 | acc |\n","| Ceval-valid-marxism | | ✓ | | 19 | acc |\n","| Ceval-valid-mao_zedong_thought | | ✓ | | 24 | acc |\n","| Ceval-valid-education_science | | ✓ | | 29 | acc |\n","| Ceval-valid-teacher_qualification | | ✓ | | 44 | acc |\n","| Ceval-valid-high_school_politics | | ✓ | | 19 | acc |\n","| Ceval-valid-high_school_geography | | ✓ | | 19 | acc |\n","| Ceval-valid-middle_school_politics | | ✓ | | 21 | acc |\n","| Ceval-valid-middle_school_geography | | ✓ | | 12 | acc |\n","| Ceval-valid-modern_chinese_history | | ✓ | | 23 | acc |\n","| Ceval-valid-ideological_and_moral_cultivation | | ✓ | | 19 | acc |\n","| Ceval-valid-logic | | ✓ | | 22 | acc |\n","| Ceval-valid-law | | ✓ | | 24 | acc |\n","| Ceval-valid-chinese_language_and_literature | | ✓ | | 23 | acc |\n","| Ceval-valid-art_studies | | ✓ | | 33 | acc |\n","| Ceval-valid-professional_tour_guide | | ✓ | | 29 | acc |\n","| Ceval-valid-legal_professional | | ✓ | | 23 | acc |\n","| Ceval-valid-high_school_chinese | | ✓ | | 19 | acc |\n","| Ceval-valid-high_school_history | | ✓ | | 20 | acc |\n","| Ceval-valid-middle_school_history | | ✓ | | 22 | acc |\n","| Ceval-valid-civil_servant | | ✓ | | 47 | acc |\n","| Ceval-valid-sports_science | | ✓ | | 19 | acc |\n","| Ceval-valid-plant_protection | | ✓ | | 22 | acc |\n","| Ceval-valid-basic_medicine | | ✓ | | 19 | acc |\n","| Ceval-valid-clinical_medicine | | ✓ | | 22 | acc |\n","| Ceval-valid-urban_and_rural_planner | | ✓ | | 46 | acc |\n","| Ceval-valid-accountant | | ✓ | | 49 | acc |\n","| Ceval-valid-fire_engineer | | ✓ | | 31 | acc |\n","| Ceval-valid-environmental_impact_assessment_engineer | | ✓ | | 31 | acc |\n","| Ceval-valid-tax_accountant | | ✓ | | 49 | acc |\n","| Ceval-valid-physician | | ✓ | | 49 | acc |"]}],"metadata":{"accelerator":"GPU","colab":{"authorship_tag":"ABX9TyOYjJFbDr/lKnnIcv2j6MLc","gpuType":"T4","machine_shape":"hm","provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}
|