evaluation/README.md

Benchmarks evaluations for tablegpt

🔥About • 💻Usage •

About

This is a repo opened for evaluation on different table-related benchmarks for tablegpt.

Given the complexity of table QA tasks and the uncertainty of input instructions, we provide evaluation datasets and scripts for 7 capabilities:

• ✨Code correction based on tables
• ✨Refusal of ambiguous questions
• ✨Table & field recall in multi-table scenarios
• ✨Table QA output code executable
• ✨Table-Bench.
• ✨Text2Sql.
• ✨TableInstruct, which includes a series of table-related evaluation benchmarks.

In addition, we have integrated other general abilities benchmarks like HumanEval, MBPP and MMLU/CMMLU. We have built an inference method based on the local model path using vLLM as the backend, and defined a set of example prompts templates for the above benchmarks.

Usage

⏬ To use this framework, please first install the repository from GitHub:

git clone https://github.com/tablegpt/tablegpt-eval
cd tablegpt-eval
pip install -r requirements.txt

[!Tips]

1. You can run all the benchmarks with the default params by running command bash run_benchmarks.sh.
2. If you want more configuration options for running parameters, refer to the typical Python script.
3. Download the .db files before running text2sql evaluation scripts. Download urls refer to /table_related_benchmarks/evalset/bird_data/dev_databases/README.md(Bird dev) & table_related_benchmarks/evalset/spider_data/dev_database/README.md (Spider dev) & table_related_benchmarks/evalset/spider_data/test_database/README.md (Spider test).

Code correction eval

We provide a non-executable eval dataset based on the Python language. Eval dataset path:

table_related_benchmarks/evalset/code_correction_test/correction_set.json

We use the executable_pass_rate and absolute_match_rate of the corrected code in pass-1 to evaluate the model's code correction ability. You can perform code-correction evaluation by running the following Python command:

python table_related_benchmarks/run_code_correction_eval.py \
        --model_path  <EVAL MODEL PATH> \
        --template  <CHAT_TEMPLATE_NAME, support [llama3, baichuan, chatglm, None], default None> \
        --eval_results_save_path <PATH TO SAVE THE EVAL RESULTS> \
        --gpus_num <NUMBER OF GPU TO RUN INFERENCE> \
        --temperature <ONE OF THE INFERENCE PARAMETER>

Ambiguous reject eval

We provide 298 table-based queries, with a ratio of about 1:3 between queries marked as ambiguous (to be rejected) and queries that should be accepted and correctly answered. Dataset path:

# test queries
evalset/reject_test/test_query.json
# queries with ground truth
evalset/reject_test/ground_truth.json

We use accuracy, recall, and F1 score as metrics to evaluate the LLM's ability in this task. You can perform reject evaluation by running the following Python command:

python table_related_benchmarks/run_reject_eval.py \
    --model_path <EVAL MODEL PATH>  \
    --save_path <LLM OUTPUT CONTENT SAVE PATH> \
    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> \
    --temperature <ONE OF THE INFERENCE PARAMETER>

Table&Fields recall eval

The provided eval dataset path:

table_related_benchmarks/evalset/retrieval_test/recall_set.json

We use a series of evaluation metrics such as recall, precision, Jaccard similarity, and Hamming loss to assess the LLM's performance in table and field retrieval tasks. You can perform recall evaluation by running the following Python command:

python table_related_benchmarks/run_recall_eval.py \
    --model_path <EVAL MODEL PATH> \
    --temperature <TEMPERATURE> \
    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

Table QA executable

Provide 2178 table based queries, eval dataset path:

table_related_benchmarks/evalset/table_qa_execuate_test/tableqa_samples_with_paths.jsonl

We employ executable_pass_rate of pass-1 to employ the model's tableQA code generation ability. You can perform tableQA evaluation by running the following Python command:

python table_related_benchmarks/run_tableqa_execution_eval.py \
    --model_path <EVAL MODEL PATH> \
    --temperature <ONE OF THE INFERENCE PARAMETER> \
    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

TableBench evaluation

The provided eval dataset path:

table_related_benchmarks/evalset/TableBench

In the evaluation of TableBench, Rough-L was used to assess general QA questions, while pass@1 was used as the evaluation metric for visualization-type samples. You can perform TableBench evaluation by the following command:

python table_related_benchmarks/run_table_bench_eval.py \
    --model_path <EVAL MODEL PATH> \
    --temperature <ONE OF THE INFERENCE PARAMETER> \
    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

TableInstruct

The provided eval dataset path:

table_related_benchmarks/evalset/TableInstruct

You can perform TableInstruct evaluation by the following command:

python table_related_benchmarks/run_table_instruct_eval.py \
    --model_path <EVAL MODEL PATH> \
    --temperature <ONE OF THE INFERENCE PARAMETER> \
    --gpus_num <NUMBER OF GPU TO RUN INFERENCE> 

Text2Sql

python table_related_benchmarks/run_text2sql_eval.py --model_path <EVAL MODEL PATH> 

HumanEval

Perform HumanEval evaluation by the following command:

python general_benchmarks/HumanEval/eval_instruct_vllm.py --model_path <EVAL MODEL PATH> 

MBPP

Perform MBPP evaluation by the following command:

python general_benchmarks/MBPP/eval_instruct_vllm.py --model_path <EVAL MODEL PATH> 

MMLU & CMMLU

python general_benchmarks/MMLU/evaluator.py --task <mmlu or cmmlu> --lang <en or zh> --model_path <EVAL MODEL PATH>