KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models
Introduction
In the current landscape of language models, achieving high performance in Korean NLP tasks requires specialized instruction datasets tailored to the unique aspects of the Korean language. To address the scarcity of such datasets, we introduce KIT-19, a comprehensive Korean Instruction Toolkit that encompasses 19 distinct tasks for fine-tuning Korean Large Language Models (LLMs). Unlike existing datasets that largely rely on translated instructions or outputs from models like ChatGPT, KIT-19 is meticulously crafted to capture the nuances of Korean language and culture, offering a robust foundation for advancing Korean LLMs.
Overview of KIT-19 Datasets
KIT-19 amalgamates 19 existing open-source datasets, each converted into an instruction format to facilitate instruction tuning for Korean LLMs. Here's a brief overview of the datasets included in KIT-19:
| Task Category | Datasets Included |
|---|---|
| Hate Speech Detection | APEACH, UnSmile, HateScore |
| Boolean Question Answering (QA) | KoBEST_BoolQ |
| Natural Language Inference (NLI) | KoBEST_COPA, korNLI |
| Text Generation | KoBEST_HellaSwag, kowiki_text |
| Semantic Textual Similarity (STS) | korSTS, pawsx_paraphr, ParaKQC, KoBEST_WIC, Style_KQC, Question_Pair |
| Sentiment Analysis (SA) | NSMC |
| Intent Argument Extraction | sae4k_sum, petitions_archive |
| Math | math_korean |
| Closed Book QA | kowiki_text (utilized differently for Closed Book QA and Text Generation) |
| Summarization | lbox_summarization |
Each dataset is selected and formated to ensure a wide coverage of tasks and scenarios relevant to the Korean language, making KIT-19 an exhaustive resource for developing and fine-tuning Korean LLMs.
Fine-Tuned Models
To demonstrate the effectiveness of KIT-19, we have fine-tuned representative Korean Pretrained LLMs, including Polyglot-Ko-5.8b and Polyglot-Ko-1.3b. The fine-tuned models showcase significant performance improvements across a variety of benchmark datasets:
- KoBEST_COPA
- KoBEST_BoolQ
- KoBEST_HellaSwag
- KLUE_STS
- KoBEST_SentiNeg
- KLUE_YNAT
The experimental results affirm that models trained with KIT-19 significantly outperform existing Korean LLMs, highlighting the potency and necessity of instruction datasets crafted specifically for the Korean language.
Benchmark Performance
Below is the performance comparison of different models on various benchmark datasets. The models trained with KIT-19 (KIT-5.8b and KIT-1.3b) are compared against Polyglot-Ko-1.3b, Polyglot-Ko-5.8b, KoAlpaca-5.8b, and Kullm-polyglot-5.8b-v2.
| Benchmark Dataset | Metric | Polyglot-ko-1.3b | Polyglot-ko-5.8b | KoAlpaca-5.8B | kullm-polyglot-5.8b-v2 | KIT-5.8b | KIT-1.3b |
|---|---|---|---|---|---|---|---|
| KoBEST_COPA | ACC | 72.00% | 77.60% | 69.80% | 76.60% | 91.60% | 83.80% |
| F1 (macro) | 71.96% | 77.55% | 69.77% | 76.53% | 91.59% | 83.78% | |
| KoBEST_BoolQ | ACC | 49.86% | 53.63% | 56.34% | 50.28% | 66.24% | 50.71% |
| F1 (macro) | 35.52% | 43.56% | 50.64% | 33.71% | 66.14% | 34.78% | |
| KoBEST_HellaSwag | ACC | 40.60% | 48.80% | 38.20% | 44.40% | 97.60% | 81.60% |
| ACC_Norm | 53.00% | 59.80% | 46.20% | 55.20% | 98.20% | 89.80% | |
| F1 (macro) | 40.13% | 48.53% | 38.15% | 44.25% | 97.61% | 81.49% | |
| KLUE_STS | ACC | 42.39% | 45.28% | 51.83% | 42.39% | 65.51% | 42.20% |
| F1 | 59.54% | 60.34% | 33.86% | 59.54% | 69.71% | 56.52% | |
| KoBEST_SentiNeg | ACC | 69.27% | 50.38% | 38.79% | 50.38% | 71.54% | 80.86% |
| F1 | 68.19% | 33.95% | 38.48% | 33.50% | 68.98% | 80.86% | |
| KLUE_YNAT | F1 | 33.24% | 33.62% | 20.91% | 32.20% | 28.15% | 38.34% |
Bold results indicate the best performance in each category.
Conclusion and Future Work
KIT-19 stands as a pivotal development in the Korean NLP landscape, addressing the critical need for comprehensive instruction datasets that encapsulate the linguistic and cultural intricacies of the Korean language. With KIT-19, we aim to push the boundaries of what's possible with Korean LLMs, laying a solid foundation for future advancements in the field.
We are committed to continuously expanding KIT-19 to cover more domains and further enhance the generability of Korean LLMs. Our hope is that KIT-19 not only serves as a valuable resource for NLP practitioners but also inspires further research and development within the Korean NLP community.
The KIT-19 dataset and the fine-tuned models are publicly available for research and development purposes, fueling advancements in Korean language modeling and applications.
For more information, access to the datasets, and models, please visit our GitHub repository.
Contributors: Dongjun Jang, Sungjoo Byun, Hyemi Jo, Hyopil Shin from the Department of Linguistics, Seoul National University
This work is supported by the linguistic insights and technological advances in NLP and aims to contribute to the broader academic and practical applications of language models.