README.md

---
language:
- vie
pipeline_tag: text-generation

Trained: Fine-tuning
Config file: 2.7B
---
# Model Card for Model ID

This model is pretrained and fine-tuned with Vietnamese language, based on GPT-NeoX which is a large language model developed by EleutherAI. 


## Model Details

### Training Data
- **Pre-train:** 
Culturax Vietnamese Dataset(450GB) + AI-Hub Vietnamese Dataset(1.3GB) + Crawled Vietnamese Wikipedia Dataset(630MB) + viwik18 Dataset(1.27GB)
- **Fine-tuning:** 
12MB Vietnamese Question & Answer dataset  
Vietnamese Alpaca(16412 rows) + Vietnamese QA Dataset based on viwik18(14293 rows)

### Training Hardware
Trained on A100 40GB GPU and 48 core CPU. Took 18 hours to reach 10 epochs.

### Hyperparameters
<figure style="width:30em">

| Hyperparameter         | Value       |
| ---------------------- | ----------- |
| num_train_epochs       | 2670182400  |
| train_batch_size       | 2           |
| learning_rate          | 0.0001      |
| warmup_steps           | 1000        |
| weight_decay           | 0           |
</figure>

### How to use 
 The model can be loaded using the `AutoModelForCausalLM` functionality:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-finetune")
model = AutoModelForCausalLM.from_pretrained("eunyounglee/GPT-NeoX-2.7B-Vietnamese-finetune")
```