--- license: mit datasets: - allenai/MADLAD-400 language: - en - ko - el - ru - bg base_model: - mistralai/Mistral-7B-v0.1 --- VocADT is a solution for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model’s weights fixed. VocADT offers a flexible and scalable solution without requiring external resources or language constraints. ## New Vocabulary Adapted Models Only the input/output embeddings are replaced, while all other original weights of base model remain fixed. These are the merged version: after training the adapters, we merge the original embeddings with the adapter to generate the new embeddings. | Name | Adapted Model | Base Model | New Vocab Size | Focused Languages | |---|---|---|---|---| | VocADT-Latin | [h-j-han/Mistral-7B-VocADT-50k-Latin](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Latin) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Swahili (sw), Indonesian (id), Estonian (et), Haitian Creole (ht), English (en)| | VocADT-Mixed | [h-j-han/Mistral-7B-VocADT-50k-Mixed](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Mixed) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Korean (ko), Greek (el), Russian (ru), Bulgarian (bg), English (en) | | VocADT-Cyrillic | [h-j-han/Mistral-7B-VocADT-50k-Cyrillic](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Cyrillic) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Russian (ru), Bulgarian (bg), Ukrainian (uk), Kazakh (kk), English (en) | ## Quick Start ```python from transformers import AutoModelForCausalLM, AutoTokenizer # model_name = "mistralai/Mistral-7B-v0.1 # Base Model model_name = "h-j-han/Mistral-7B-VocADT-50k-Mixed" # Vocabulary Adapted Model tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name) prefix = "\nEnglish: Hello \nKorean: 안녕하세요 \nEnglish: Thank you\nKorean: 고맙습니다\nEnglish: " line = "I lived in Korea for seven years" suffix = f"\nKorean:" prompt = prefix + line + suffix inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=8) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) # Base Model Output: "한국에 7년" # This short incomplete phrase in Korean is 8 tokens for the base model. # VocADT Output: "저는 한국에 7년 동안 살았습니다." # Complete and good output within 8 tokens ``` ## Reference We provide code in Github repo : https://github.com/h-j-han/VocADT Also, please find details in this paper : ``` @misc{han2024vocadt, title={Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?}, author={HyoJung Han and Akiko Eriguchi and Haoran Xu and Hieu Hoang and Marine Carpuat and Huda Khayrallah}, year={2024}, eprint={2410.09644}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2410.09644}, } ```