---
# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/model-cards
license: apache-2.0
language:
  - zh
widget:
  - text: >-
      A chat between a curious user and an artificial intelligence assistant.
      The assistant gives helpful, detailed, and polite answers to the user's
      questions. USER: 你好，請問你可以幫我寫一封推薦信嗎？ ASSISTANT:
library_name: transformers
pipeline_tag: text-generation
extra_gated_heading: Acknowledge license to accept the repository.
extra_gated_prompt: Please contact the author for access.
extra_gated_button_content: Acknowledge license 同意以上內容
extra_gated_fields:
  Name: text
  Mail: text
  Organization: text
  Country: text
  Any utilization of the Taiwan LLM repository mandates the explicit acknowledgment and attribution to the original author: checkbox
  使用Taiwan LLM必須明確地承認和歸功於優必達株式會社 Ubitus 以及原始作者: checkbox
---
<img src="https://cdn-uploads.huggingface.co/production/uploads/5df9c78eda6d0311fd3d541f/CmusIT5OlSXvFrbTJ7l-C.png" alt="Taiwan LLM Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>

# 🌟 Checkout [Taiwan-LLM Demo Chat-UI](http://www.twllm.com) 🌟

# Model Card for Taiwan LLM 13B v2.0 chat

Taiwan LLM is an advanced language model tailored for Traditional Chinese, focusing on the linguistic and cultural contexts of Taiwan. 
Developed from a large base model, it's enriched with diverse Taiwanese textual sources and refined through Supervised Fine-Tuning. 
This model excels in language understanding and generation, aligning closely with Taiwan's cultural nuances. 
It demonstrates improved performance on various benchmarks like TC-Eval, showcasing its contextual comprehension and cultural relevance. 
For detailed insights into Taiwan LLM's development and features, refer to our [technical report](https://github.com/MiuLab/Taiwan-LLaMa/blob/main/twllm_paper.pdf).


## Model description

- **Model type:** A 13B parameter GPT-like model fine-tuned on a mix of publicly available, synthetic datasets.
- **Language(s) (NLP):** Primarily Traditional Chinese (zh-tw)
- **Finetuned from model:** [yentinglin/Taiwan-LLM-13B-v2.0-base](https://huggingface.co/yentinglin/Taiwan-LLM-13B-v2.0-base)

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/MiuLab/Taiwan-LLaMa
- **Demo:** https://twllm.com/

## Performance


![image/png](https://cdn-uploads.huggingface.co/production/uploads/5df9c78eda6d0311fd3d541f/HTwIzw6RDha2-PhuWqSuI.png)

TMMLUS+ score: 24.76727075757576

## Intended uses

Here's how you can run the model using the `pipeline()` function from 🤗 Transformers:

```python
# pip install transformers>=4.34
# pip install accelerate

import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="yentinglin/Taiwan-LLM-13B-v2.0-chat", torch_dtype=torch.bfloat16, device_map="auto")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {
        "role": "system",
        "content": "你是一個人工智慧助理",
    },
    {"role": "user", "content": "東北季風如何影響台灣氣候？"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```

### Training hyperparameters

![image/png](https://cdn-uploads.huggingface.co/production/uploads/5df9c78eda6d0311fd3d541f/MdvHwdUvH-c926qyRAw7K.png)

![image/png](https://cdn-uploads.huggingface.co/production/uploads/5df9c78eda6d0311fd3d541f/kKpkvxDzOEyiAoTqmzRYO.png)


![image/png](https://cdn-uploads.huggingface.co/production/uploads/5df9c78eda6d0311fd3d541f/FsnlJ_fkRxf7fn5RKZnjE.png)

The following hyperparameters were used during training:
- learning_rate: 5e-05
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 5.0

## Citation

If you find Taiwan LLM is useful in your work, please cite it with:

```
@misc{lin2023taiwan,
      title={Taiwan LLM: Bridging the Linguistic Divide with a Culturally Aligned Language Model}, 
      author={Yen-Ting Lin and Yun-Nung Chen},
      year={2023},
      eprint={2311.17487},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

# Acknowledgement

Taiwan LLM v2 is conducted in collaboration with [Ubitus K.K.](http://ubitus.net). Ubitus provides valuable compute resources for the project.

## Open LLM Leaderboard
|                         Task                         |Version|    Metric    |Value |   |Stderr|
|------------------------------------------------------|------:|--------------|-----:|---|-----:|
|leaderboard:arc:challenge:25                          |      0|acc           |0.5529|±  |0.0145|
|                                                      |       |acc_norm      |0.5862|±  |0.0144|
|leaderboard:gsm8k:5                                   |      0|qem           |0.3177|±  |0.0128|
|leaderboard:hellaswag:10                              |      0|acc           |0.6307|±  |0.0048|
|                                                      |       |acc_norm      |0.8327|±  |0.0037|
|leaderboard:mmlu:_average:5                           |       |acc           |0.5483|±  |0.0356|
|leaderboard:mmlu:abstract_algebra:5                   |      0|acc           |0.3400|±  |0.0476|
|leaderboard:mmlu:anatomy:5                            |      0|acc           |0.5111|±  |0.0432|
|leaderboard:mmlu:astronomy:5                          |      0|acc           |0.5789|±  |0.0402|
|leaderboard:mmlu:business_ethics:5                    |      0|acc           |0.5100|±  |0.0502|
|leaderboard:mmlu:clinical_knowledge:5                 |      0|acc           |0.6000|±  |0.0302|
|leaderboard:mmlu:college_biology:5                    |      0|acc           |0.5764|±  |0.0413|
|leaderboard:mmlu:college_chemistry:5                  |      0|acc           |0.4100|±  |0.0494|
|leaderboard:mmlu:college_computer_science:5           |      0|acc           |0.4500|±  |0.0500|
|leaderboard:mmlu:college_mathematics:5                |      0|acc           |0.3800|±  |0.0488|
|leaderboard:mmlu:college_medicine:5                   |      0|acc           |0.5434|±  |0.0380|
|leaderboard:mmlu:college_physics:5                    |      0|acc           |0.2941|±  |0.0453|
|leaderboard:mmlu:computer_security:5                  |      0|acc           |0.7000|±  |0.0461|
|leaderboard:mmlu:conceptual_physics:5                 |      0|acc           |0.4468|±  |0.0325|
|leaderboard:mmlu:econometrics:5                       |      0|acc           |0.2719|±  |0.0419|
|leaderboard:mmlu:electrical_engineering:5             |      0|acc           |0.4552|±  |0.0415|
|leaderboard:mmlu:elementary_mathematics:5             |      0|acc           |0.3175|±  |0.0240|
|leaderboard:mmlu:formal_logic:5                       |      0|acc           |0.3413|±  |0.0424|
|leaderboard:mmlu:global_facts:5                       |      0|acc           |0.3700|±  |0.0485|
|leaderboard:mmlu:high_school_biology:5                |      0|acc           |0.6323|±  |0.0274|
|leaderboard:mmlu:high_school_chemistry:5              |      0|acc           |0.4581|±  |0.0351|
|leaderboard:mmlu:high_school_computer_science:5       |      0|acc           |0.5400|±  |0.0501|
|leaderboard:mmlu:high_school_european_history:5       |      0|acc           |0.6364|±  |0.0376|
|leaderboard:mmlu:high_school_geography:5              |      0|acc           |0.6970|±  |0.0327|
|leaderboard:mmlu:high_school_government_and_politics:5|      0|acc           |0.7617|±  |0.0307|
|leaderboard:mmlu:high_school_macroeconomics:5         |      0|acc           |0.4974|±  |0.0254|
|leaderboard:mmlu:high_school_mathematics:5            |      0|acc           |0.3296|±  |0.0287|
|leaderboard:mmlu:high_school_microeconomics:5         |      0|acc           |0.5336|±  |0.0324|
|leaderboard:mmlu:high_school_physics:5                |      0|acc           |0.3709|±  |0.0394|
|leaderboard:mmlu:high_school_psychology:5             |      0|acc           |0.7468|±  |0.0186|
|leaderboard:mmlu:high_school_statistics:5             |      0|acc           |0.4074|±  |0.0335|
|leaderboard:mmlu:high_school_us_history:5             |      0|acc           |0.7108|±  |0.0318|
|leaderboard:mmlu:high_school_world_history:5          |      0|acc           |0.7046|±  |0.0297|
|leaderboard:mmlu:human_aging:5                        |      0|acc           |0.6323|±  |0.0324|
|leaderboard:mmlu:human_sexuality:5                    |      0|acc           |0.5878|±  |0.0432|
|leaderboard:mmlu:international_law:5                  |      0|acc           |0.6694|±  |0.0429|
|leaderboard:mmlu:jurisprudence:5                      |      0|acc           |0.7037|±  |0.0441|
|leaderboard:mmlu:logical_fallacies:5                  |      0|acc           |0.6564|±  |0.0373|
|leaderboard:mmlu:machine_learning:5                   |      0|acc           |0.3393|±  |0.0449|
|leaderboard:mmlu:management:5                         |      0|acc           |0.7087|±  |0.0450|
|leaderboard:mmlu:marketing:5                          |      0|acc           |0.8333|±  |0.0244|
|leaderboard:mmlu:medical_genetics:5                   |      0|acc           |0.5400|±  |0.0501|
|leaderboard:mmlu:miscellaneous:5                      |      0|acc           |0.7382|±  |0.0157|
|leaderboard:mmlu:moral_disputes:5                     |      0|acc           |0.6127|±  |0.0262|
|leaderboard:mmlu:moral_scenarios:5                    |      0|acc           |0.3788|±  |0.0162|
|leaderboard:mmlu:nutrition:5                          |      0|acc           |0.6046|±  |0.0280|
|leaderboard:mmlu:philosophy:5                         |      0|acc           |0.6270|±  |0.0275|
|leaderboard:mmlu:prehistory:5                         |      0|acc           |0.6204|±  |0.0270|
|leaderboard:mmlu:professional_accounting:5            |      0|acc           |0.3582|±  |0.0286|
|leaderboard:mmlu:professional_law:5                   |      0|acc           |0.3931|±  |0.0125|
|leaderboard:mmlu:professional_medicine:5              |      0|acc           |0.5184|±  |0.0304|
|leaderboard:mmlu:professional_psychology:5            |      0|acc           |0.5556|±  |0.0201|
|leaderboard:mmlu:public_relations:5                   |      0|acc           |0.6818|±  |0.0446|
|leaderboard:mmlu:security_studies:5                   |      0|acc           |0.6122|±  |0.0312|
|leaderboard:mmlu:sociology:5                          |      0|acc           |0.7164|±  |0.0319|
|leaderboard:mmlu:us_foreign_policy:5                  |      0|acc           |0.8200|±  |0.0386|
|leaderboard:mmlu:virology:5                           |      0|acc           |0.4578|±  |0.0388|
|leaderboard:mmlu:world_religions:5                    |      0|acc           |0.7661|±  |0.0325|
|leaderboard:truthfulqa:mc:0                           |      0|truthfulqa_mc1|0.2840|±  |0.0158|
|                                                      |       |truthfulqa_mc2|0.4423|±  |0.0146|
|leaderboard:winogrande:5                              |      0|acc           |0.7593|±  |0.0120|


## TC-Eval
# DRCD TODO
community|tc-eval-v2:drcd|0|0
# Table TODO
community|tc-eval-v2:penguin_table|0|0
# MMLU
community|tc-eval-v2:tmmluplus-accounting|5|0
community|tc-eval-v2:tmmluplus-administrative_law|5|0
community|tc-eval-v2:tmmluplus-advance_chemistry|5|0
community|tc-eval-v2:tmmluplus-agriculture|5|0
community|tc-eval-v2:tmmluplus-anti_money_laundering|5|0
community|tc-eval-v2:tmmluplus-auditing|5|0
community|tc-eval-v2:tmmluplus-basic_medical_science|5|0
community|tc-eval-v2:tmmluplus-business_management|5|0
community|tc-eval-v2:tmmluplus-chinese_language_and_literature|5|0
community|tc-eval-v2:tmmluplus-clinical_psychology|5|0
community|tc-eval-v2:tmmluplus-computer_science|5|0
community|tc-eval-v2:tmmluplus-culinary_skills|5|0
community|tc-eval-v2:tmmluplus-dentistry|5|0
community|tc-eval-v2:tmmluplus-economics|5|0
community|tc-eval-v2:tmmluplus-education|5|0
community|tc-eval-v2:tmmluplus-education_(profession_level)|5|0
community|tc-eval-v2:tmmluplus-educational_psychology|5|0
community|tc-eval-v2:tmmluplus-engineering_math|5|0
community|tc-eval-v2:tmmluplus-finance_banking|5|0
community|tc-eval-v2:tmmluplus-financial_analysis|5|0
community|tc-eval-v2:tmmluplus-fire_science|5|0
community|tc-eval-v2:tmmluplus-general_principles_of_law|5|0
community|tc-eval-v2:tmmluplus-geography_of_taiwan|5|0
community|tc-eval-v2:tmmluplus-human_behavior|5|0
community|tc-eval-v2:tmmluplus-insurance_studies|5|0
community|tc-eval-v2:tmmluplus-introduction_to_law|5|0
community|tc-eval-v2:tmmluplus-jce_humanities|5|0
community|tc-eval-v2:tmmluplus-junior_chemistry|5|0
community|tc-eval-v2:tmmluplus-junior_chinese_exam|5|0
community|tc-eval-v2:tmmluplus-junior_math_exam|5|0
community|tc-eval-v2:tmmluplus-junior_science_exam|5|0
community|tc-eval-v2:tmmluplus-junior_social_studies|5|0
community|tc-eval-v2:tmmluplus-logic_reasoning|5|0
community|tc-eval-v2:tmmluplus-macroeconomics|5|0
community|tc-eval-v2:tmmluplus-management_accounting|5|0
community|tc-eval-v2:tmmluplus-marketing_management|5|0
community|tc-eval-v2:tmmluplus-mechanical|5|0
community|tc-eval-v2:tmmluplus-music|5|0
community|tc-eval-v2:tmmluplus-national_protection|5|0
community|tc-eval-v2:tmmluplus-nautical_science|5|0
community|tc-eval-v2:tmmluplus-occupational_therapy_for_psychological_disorders|5|0
community|tc-eval-v2:tmmluplus-official_document_management|5|0
community|tc-eval-v2:tmmluplus-optometry|5|0
community|tc-eval-v2:tmmluplus-organic_chemistry|5|0
community|tc-eval-v2:tmmluplus-pharmacology|5|0
community|tc-eval-v2:tmmluplus-pharmacy|5|0
community|tc-eval-v2:tmmluplus-physical_education|5|0
community|tc-eval-v2:tmmluplus-physics|5|0
community|tc-eval-v2:tmmluplus-politic_science|5|0
community|tc-eval-v2:tmmluplus-real_estate|5|0
community|tc-eval-v2:tmmluplus-secondary_physics|5|0
community|tc-eval-v2:tmmluplus-statistics_and_machine_learning|5|0
community|tc-eval-v2:tmmluplus-taiwanese_hokkien|5|0
community|tc-eval-v2:tmmluplus-taxation|5|0
community|tc-eval-v2:tmmluplus-technical|5|0
community|tc-eval-v2:tmmluplus-three_principles_of_people|5|0
community|tc-eval-v2:tmmluplus-trade|5|0
community|tc-eval-v2:tmmluplus-traditional_chinese_medicine_clinical_medicine|5|0
community|tc-eval-v2:tmmluplus-trust_practice|5|0
community|tc-eval-v2:tmmluplus-ttqav2|5|0
community|tc-eval-v2:tmmluplus-tve_chinese_language|5|0
community|tc-eval-v2:tmmluplus-tve_design|5|0
community|tc-eval-v2:tmmluplus-tve_mathematics|5|0
community|tc-eval-v2:tmmluplus-tve_natural_sciences|5|0
community|tc-eval-v2:tmmluplus-veterinary_pathology|5|0
community|tc-eval-v2:tmmluplus-veterinary_pharmacology|5|0