facebook/mms-1b-all · is mms compatible with PEFT (parameter efficient fine tuning)?

Jul 5, 2023

PEFT is amazing, I have been fine tunning whisper large model which require about ~24 Vram to fine tune but with PEFT I was able to use my 16GB just fine, I was wondering if PEFT is compatible with MMS yet? if not is there is a TBA date?

sanchit-gandhi

Jul 5, 2023

You can check out this blog-post for a low-resource approach to fine-tuning MMS for ASR: https://huggingface.co/blog/mms_adapters

We only fine tune the adapter weights (a few million params, only a fraction of the full model weights), so I expect 16GB VRAM is more than enough here! Let me know if it's not though - happy to provide some more pointers for reducing memory with MMS fine-tuning if it's still too high!

mohblnk

Jul 6, 2023

You can check out this blog-post for a low-resource approach to fine-tuning MMS for ASR: https://huggingface.co/blog/mms_adapters

We only fine tune the adapter weights (a few million params, only a fraction of the full model weights), so I expect 16GB VRAM is more than enough here! Let me know if it's not though - happy to provide some more pointers for reducing memory with MMS fine-tuning if it's still too high!

I have taken a quick look and looks like I'm in for a treat, Thank you @sanchit-gandhi