How to merge the 2 bin files into the pytorch_model.bin file for usage?
#2
by
spanielrassler
- opened
I'm sure this is a stupid questions, but how do I merge the bin files into one file that can be used as a model? Final model name should be pytorch_model.bin but with 2 separate .bin files not sure how to do that.
The transformers library automatically shards large models with the save_pretrained() method into partial checkpoints and an index that maps parameter names to the files they are stored in. The model be loaded by passing the model directory (e.g., ./my_model_directory/) to from_pretrained().
These links may be helpful https://huggingface.co/docs/transformers/big_models and https://github.com/huggingface/transformers/blob/v4.27.2/src/transformers/modeling_utils.py#L1833