Solving the issue "AttributeError: 'NllbTokenizerFast' object has no attribute 'lang_code_to_id'"

When trying to do a simple translation from french to Mooré, I encountered this error.


This seems to be related to an recent update from transformers library. To be more precise they removed the "lang_code_to_id" attribute from NllbTokenize and NllbTokenizerFast classes.

To solve the issue, I just replaced the "lang_code_to_id" attribute by "convert_tokens_to_ids()" method.

Hope this help to keep your great space updated.

src/translation.py

@@ -94,7 +94,7 @@ def translate_facebook(s:str, src_iso:str, dest_iso:str)-> str:
 
     # Inference
     encoded = tokenizer(s, return_tensors="pt")
-    translated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id[f"{dest_iso}_Latn"], max_length=30)
+    translated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.convert_tokens_to_ids(f"{dest_iso}_Latn"), max_length=30)
     translation = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
 
     return translation