How to use an LM like n-gram LM with w2v-bert-2.0?
I have fine-tuned w2v-bert-2.0 for a low-resource language. The results are good, but could be better with an LM like n-gram LM. I am interested whether that's possible or not? Or what are the alternatives?
Yes, it's really possible, take a look at this guide: https://huggingface.co/blog/wav2vec2-with-ngram
Note that you should adapt the code for this particular model! From my experience you get a nice boost
Thank you for your response!
I took a look at the guide, but couldn't find an equivalent of Wav2Vec2ProcessorWithLM for w2v-bert-2.0.
Does the n-gram LM or decoder need to be called manually? If yes, could you please provide another guide if there is one?
@ylacombe I'm not sure if w2v-bert-2.0 can even be combined with an ngram since the Wav2Vec2ProcessorWithLM function in the shared blog post expects a Wav2Vec2 Feature Extractor while w2v-bert-2.0 contains a SeamlessM4T Feature Extractor.
Apparently this has already been discussed and fixed by the repo owners but just hasn't been merged yet with the main branch. For anyone else facing this problem, adding Wav2Vec2ProcessorWithLM.feature_extractor_class = "AutoFeatureExtractor" before creating the processor is a hacky solution to get it working for now.
@UmarRamzan slightly confused on how to use the following line of code, could you please show me the code in a bit more collaborative way. (more lines, the entire function/class implementation)
from transformers import Wav2Vec2ProcessorWithLM
# just have to add this line
Wav2Vec2ProcessorWithLM.feature_extractor_class = "AutoFeatureExtractor"
processor_with_lm = Wav2Vec2ProcessorWithLM(
feature_extractor=processor.feature_extractor,
tokenizer=processor.tokenizer,
decoder=decoder
)
and the rest of the code is as provided here https://huggingface.co/blog/wav2vec2-with-ngram