Spaces:

CONDA-Workshop

/

Data-Contamination-Database

like 16

Running

What are you reporting:

• Evaluation dataset(s) found in a pre-training corpus. (e.g. COPA found in ThePile)
• Evaluation dataset(s) found in a pre-trained model. (e.g. FLAN T5 has been trained on ANLI)

Evaluation dataset(s): Name(s) of the evaluation dataset(s). If available in the HuggingFace Hub please write the path (e.g. uonlp/CulturaX), otherwise provide a link to a paper, GitHub or dataset-card.
xlangai/spider
Contaminated model(s): Name of the model(s) (if any) that have been contaminated with the evaluation dataset. If available in the HuggingFace Hub please list the corresponding paths (e.g. allenai/OLMo-7B).
GPT-3.5

Briefly describe your method to detect data contamination

• Data-based approach
• Model-based approach

Paper introduces contamination detection for Text-to-SQL datasets by measuring the previous knowledge that a model has on the sql database dumps contained in these datasets. A clue to determine whether the data contamination has occurred is that the model is able to reconstruct missing information regarding the database schema. This paper shows that data contamination is responsible for overestimating the performance of GPT3.5 on Text-to-SQL and clearly demonstrate that GPT-3.5 possesses prior knowledge on the contents of the Spider validation set in contrast to its ignorance of the author's constructed Text-to-SQL unseen dataset, Termite.

Citation

Is there a paper that reports the data contamination or describes the method used to detect data contamination?

URL: https://arxiv.org/pdf/2402.08100
Citation: @misc{ranaldi2024investigating, title={Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translation}, author={Federico Ranaldi and Elena Sofia Ruzzetti and Dario Onorati and Leonardo Ranaldi and Cristina Giannone and Andrea Favalli and Raniero Romagnoli and Fabio Massimo Zanzotto}, year={2024}, eprint={2402.08100}, archivePrefix={arXiv}, primaryClass={cs.CL} }

Important! If you wish to be listed as an author in the final report, please complete this information for all the authors of this Pull Request.

• Full name: Bhavish Pahwa
• Institution: Microsoft Research
• Email: t-bpahwa@microsoft.com