|
--- |
|
license: bsd-3-clause |
|
language: |
|
- vi |
|
--- |
|
# PhoGPT: Generative Pre-training for Vietnamese |
|
|
|
We release a 7.5B-parameter generative model series named PhoGPT for Vietnamese, which includes the base pre-trained monolingual model **PhoGPT-7B5** and its instruction-following variant **PhoGPT-7B5-Instruct**. |
|
|
|
PhoGPT-7B5 is a Transformer decoder-based model that incorporates flash attention (Triton) and ALiBi for context length extrapolation. We pre-trained PhoGPT-7B5 from scratch on a 41GB pre-training corpus of Vietnamese texts. We then fine-tuned the pre-trained PhoGPT-7B5 for instruction following using a dataset of 150K Vietnamese prompt-response pairs, resulting in PhoGPT-7B5-Instruct. |
|
|
|
We recommend users use the latest PhoGPT versions - PhoGPT-4B and PhoGPT-4B-Instruct - for better performance and efficiency. |
|
|
|
For further information or requests, please go to [PhoGPT's homepage](https://github.com/VinAIResearch/PhoGPT)! |
