This is the model checkpoint release for Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models.
All the fine-tuned model checkpoints are released in this repository. The naming convention of the revisions are olmo1b_hf_{checkpoint}_{train_dataset}_{epoch}_{lr}.
To load a specific model checkpoint, use the following command.
model = AutoModelForCausalLM.from_pretrained(
model_name_or_path="KaiserWhoLearns/PTvsSFT_OLMo1b",
trust_remote_code=trust_remote_code,
revision="your revision"
)
All the checkpoints are fine-tuned based on the checkpoints of OLMo1b-HF.
Citation:
@misc{sun2024amurocharanalyzing,
title={Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models},
author={Kaiser Sun and Mark Dredze},
year={2024},
eprint={2408.06663},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2408.06663},
}