Parameters

• output_dir (str) — Local folder path where you want results to be saved.
• save_details (bool, defaults to True) — If True, details are saved to the output_dir.
• push_to_hub (bool, defaults to False) — If True, details are pushed to the hub. Results are pushed to {hub_results_org}/details__{sanitized model_name} for the model model_name, a public dataset, if public is True else {hub_results_org}/details__{sanitized model_name}_private, a private dataset.
• push_to_tensorboard (bool, defaults to False) — If True, will create and push the results for a tensorboard folder on the hub.
• hub_results_org (str, optional) — The organisation to push the results to. See more details about the datasets organisation in EvaluationTracker.save.
• tensorboard_metric_prefix (str, defaults to “eval”) — Prefix for the metrics in the tensorboard logs.
• public (bool, defaults to False) — If True, results and details are pushed to public orgs.
• nanotron_run_info (~nanotron.config.GeneralArgs, optional) — Reference to information about Nanotron models runs.

Parameters

• lighteval_sha (str) — Current commit sha of lighteval used for the evaluation (for reproducibility purposes)
• num_fewshot_seeds (int) — Number of seeds for the few-shot sampling. If equal to or below 1, the experiment is done once only, with a single few-shot seed (equal to 0). If above, the experiment is reproduced several times, with a different sampling/shuffling for the few-shot examples, which follows what is done in HELM for example.
• override_batch_size (int) — Manages the batch size. If strictly positive, its value is used as the batch size for all experiments. Else, the batch size is automatically inferred depending on what fits in memory.
• max_samples (int) — If set, cuts the number of samples per task to max_samples. Note: This should only be used for debugging purposes!
• job_id (int) — If the evaluation suite is launched as a slurm job, stores the current job id. Purely informative parameter used to retrieve scheduler logs.
• start_time (float) — Start time of the experiment. Logged at class init.
• end_time (float) — End time of the experiment. Logged when calling GeneralConfigLogger.log_end_time
• total_evaluation_time_secondes (str) — Inferred total evaluation time in seconds (from the start and end times).
• model_name (str) — Name of the currently evaluated model.
• model_sha (str) — Commit hash of the currently evaluated model on the hub if available.
• model_dtype (str) — Dtype of the model weights, as obtained when loading the model config.
• model_size (str) — Model size as obtained when loading the model config.

Parameters

• hashes (dict[str, listHash) — Maps each task name to the list of all its samples’ Hash.
• compiled_hashes (dict[str, CompiledHash) — Maps each task name to its CompiledHas, an aggregation of all the individual sample hashes.
• details (dict[str, listDetail]) — Maps each task name to the list of its samples’ details. Example: winogrande: [sample1_details, sample2_details, …]
• compiled_details (dict[str, CompiledDetail]) — : Maps each task name to the list of its samples’ compiled details.
• compiled_details_over_all_tasks (CompiledDetailOverAllTasks) — Aggregated details over all the tasks.

Parameters

• metrics_value (dict[str, dict[str, list[float]]]) — Maps each task to its dictionary of metrics to scores for all the example of the task. Example: {“winogrande|winogrande_xl”: {“accuracy”: [0.5, 0.5, 0.5, 0.5, 0.5, 0.5]}‌}
• metric_aggregated (dict[str, dict[str, float]]) — Maps each task to its dictionary of metrics to aggregated scores over all the example of the task. Example: {“winogrande|winogrande_xl”: {“accuracy”: 0.5}‌}

Parameters

• version (dict[str, int]) — Maps the task names with the task versions.

Parameters

• tasks_config (dict[str, LightevalTaskConfig]) — Maps each task to its associated LightevalTaskConfig