devngho/ko_edu_classifier_v2_LaBSE
์ด ๋ชจ๋ธ์ sentence-transformers/LaBSE์ classifier๋ฅผ ์ถ๊ฐํ ๋ชจ๋ธ์ ๋๋ค. HuggingFaceFW/fineweb-edu-classifier์ ํ๊ตญ์ด ๋ฒ์ ์ ๋ชฉํ๋ก, ํ๊ตญ์ด ์น ํ์ด์ง์ ๊ต์ก์ฑ ์ ์๋ฅผ ํ๊ฐํฉ๋๋ค. ํ์ต์๋ blueapple8259/c4-ko-cleaned-2์์ ์ถ์ถํ 500k ์ํ์ Qwen/Qwen2.5-32B-Instruct๋ก ํ๊ฐํ devngho/ko_llm_annotations ๋ฐ์ดํฐ์ ์ด ์ฌ์ฉ๋์์ต๋๋ค.
์ด ์ฐ๊ตฌ๋ Google์ TPU Research Cloud (TRC)์ Cloud TPU ์ ๊ณต์ผ๋ก ์ํ๋์์ต๋๋ค. โก
์์ธ
- ์ ์: devngho
- ์ธ์ด: ko
- ๋ผ์ด์ ์ค: mit
- ๊ธฐ๋ฐ ๋ชจ๋ธ: sentence-transformers/LaBSE
ํ์ต ์์ธ
- learning_rate: 3e-4 (cosine)
- warmup_ratio: 0.1
- batch_size: 512
- optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
- duration: 3h 21m
ํ์ต ์ฅ๋น
TPU v4-8
์ฑ๋ฅ
Validation Report:
precision recall f1-score support
0 0.51 0.28 0.36 198
1 0.69 0.50 0.58 1553
2 0.39 0.69 0.49 1159
3 0.54 0.42 0.47 967
4 0.54 0.10 0.16 219
accuracy 0.50 4096
macro avg 0.53 0.40 0.42 4096
weighted avg 0.55 0.50 0.50 4096
Confusion Matrix:
[[ 56 108 34 0 0]
[ 51 782 677 43 0]
[ 2 202 797 155 3]
[ 0 37 508 407 15]
[ 0 1 53 144 21]]
ํ๊ตญ์ด ์๋ฒ ๋ฉ์ ํ๊ณ์ qwen2.5 32b ๋ชจ๋ธ์ ํ๊ฐ ํ๊ณ๋ก ์ฑ๋ฅ์ด ๋ฎ์ ๊ฒ์ผ๋ก ๋ณด์ ๋๋ค. 3 ์ด์๊ณผ ๋ฏธ๋ง์ผ๋ก ๊ตฌ๋ถํ ๋ f1 score๋ ์ฝ 0.59์ ๋๋ค.
devngho/ko_edu_classifier_v2_LaBSE
This model is sentence-transformers/LaBSE with classfier head. It is designed to evaluate the educational value of Korean web pages, similar to the HuggingFaceFW/fineweb-edu-classifier, but focused on Korean content. The training data comes from devngho/ko_llm_annotations dataset, contains 500k samples extracted from blueapple8259/c4-ko-cleaned-2 and evaluated using Qwen/Qwen2.5-32B-Instruct.
This research was supported with Cloud TPUs from Google's TPU Research Cloud (TRC).โก
- Developed by: devngho
- Language(s): ko
- License: mit
- Base model: sentence-transformers/LaBSE
Training detail
- learning_rate: 3e-4 (cosine)
- warmup_ratio: 0.1
- batch_size: 512
- optimizer: adamw(b1=0.9, b2=0.98, eps=1e-8, weight_decay=0.01)
- duration: 3h 21m
Training hardware
TPU v4-8
Performance
Validation Report:
precision recall f1-score support
0 0.51 0.28 0.36 198
1 0.69 0.50 0.58 1553
2 0.39 0.69 0.49 1159
3 0.54 0.42 0.47 967
4 0.54 0.10 0.16 219
accuracy 0.50 4096
macro avg 0.53 0.40 0.42 4096
weighted avg 0.55 0.50 0.50 4096
Confusion Matrix:
[[ 56 108 34 0 0]
[ 51 782 677 43 0]
[ 2 202 797 155 3]
[ 0 37 508 407 15]
[ 0 1 53 144 21]]
The low performance is likely due to the limitations of Korean embeddings and the evaluation limitations of the Qwen2.5 32B model. The F1 score is about 0.59 when separating above and below 3.
- Downloads last month
- 0
Model tree for devngho/ko_edu_classifier_v2_LaBSE
Base model
sentence-transformers/LaBSE