How to add new files without cloning the entire repo?

by ggerganov - opened Apr 2

ggml.ai org Apr 2

@phymbert Now that we have Grok-1 in the repo, I'm not sure how to commit new files without having to download extra ~150GB of the grok shards. Is there some workaround?

If not, we might want to move the model to a separate repo, or demonstrate the sharding with something smaller.

julien-c

Apr 3

if using git: you can clone the repo using GIT_LFS_SKIP_SMUDGE=1 env var (only downloads pointers to large files, not the actual files) and you'll still be able to git commit as usual
alternatives to git:
- use our HTTP API: huggingface-cli upload ...
- or just drag'n'drop files from your browser to upload them (the hashing implem should be efficient, but ~150GB still might be a lot)

julien-c

Apr 3

That being said, we do recommend (as a convention) splitting different models into different repos

ggerganov

ggml.ai org Apr 3

That being said, we do recommend (as a convention) splitting different models into different repos

Noted, will split the models in different repos accordingly

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment