Edit model card

Model Description

This HF repository contains LLMs instruction tuned with full-parameter fine-tuning and then used to study whether monolingual or multilingual instruction tuning is more favourable.

Instruction tuning details

  • Base model: EleutherAI/pythia-6.9b-deduped
  • Instruction tuning language: multilingual (Bulgarian, Czech, Chinese, German, Finnish, French, Russian, and Spanish)
  • Training method: full-parameter fine-tuning.
  • Best checkpoint: best cross-entropy on a validation set, trained for 3 epochs.
  • Dataset: machine-translated from yahma/alpaca-cleaned. You can download our data HERE.

Usage

The model checkpoint should be loaded using transformers library.

Please refer to our Github repository HERE for inference and training instructions.

Citation

@inproceedings{chen-etal-2024-monolingual,
  title="Monolingual or multilingual instruction tuning: Which makes a better {Alpaca}",
  author="Pinzhen Chen and Shaoxiong Ji and Nikolay Bogoychev and Andrey Kutuzov and Barry Haddow and Kenneth Heafield",
  year="2024",
  booktitle = "Findings of the Association for Computational Linguistics: EACL 2024",
}
Downloads last month
12
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Collection including HPLT/sft-fpft-multilingual-pythia-6.9b-deduped