File size: 2,246 Bytes
7d9d4ae 34510d6 7d9d4ae 34510d6 7d9d4ae fc8d748 73a59a2 7d9d4ae 2d36827 7d9d4ae 7d0486a 7d9d4ae 7d0486a 7d9d4ae ccfff81 a0bc11e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
---
model-index:
- name: EEVE-Math-10.8B
results:
- task:
type: text-generation
dataset:
name: gsm8k-ko
type: gsm8k
metrics:
- name: pass@1
type: pass@1
value: 0.539
verified: false
base_model: yanolja/EEVE-Korean-10.8B-v1.0
license: cc-by-sa-4.0
language:
- ko
tags:
- math
---
# EEVE-Math-10.8B-SFT
`EEVE-Math` νλ‘μ νΈλ
- Orca-Math-200k λ²μ ([Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://arxiv.org/pdf/2402.14830.pdf))
- gsm8k λ²μ, lm_eval νμ©
- Mergekitμ μ΄μ©ν dare-ties μ¬μ© ([DARE](https://arxiv.org/abs/2311.03099))
μ λν λ΄μ©μ ν¬κ΄νκ³ μμ΅λλ€.
> μ΄ λͺ¨λΈμ orca-math-word-problems-193k-korean λ°μ΄ν°μ
μ μ΄μ©νμ¬ νμ΅λμμ΅λλ€. μλ΅ μ€ μΌλΆλ LaTeX νμμ μ΄μ©νμ¬ κ²°κ³Όλ₯Ό λ°ννμ§λ§, μμ±λ νμμ΄ μλ μ μμ΅λλ€. νμ¬ M1 stageκΉμ§ μ§νλμμ΅λλ€.
| Model | gsm8k-ko(pass@1) |
|---|---|
| Base | 0.4049 |
| SFT(M1) | 0.508 |
| SFT(M1) -> SFT | **0.539** |
| SFT(M1) -> KTO(M2) | - |
| SFT -> KTO(M2) -> KTO(final) | - |
## Specifications
- SFT(M1) -> SFT λ¨κ³
## Base Model
[yanolja/EEVE-Korean-10.8B-v1.0](https://huggingface.co/yanolja/EEVE-Korean-10.8B-v1.0)
## Dataset
[orca-math-word-problems-193k-korean](https://huggingface.co/datasets/kuotient/orca-math-word-problems-193k-korean)
## Evaluation
[gsm8k-ko](https://huggingface.co/datasets/kuotient/gsm8k-ko), kobest
```
git clone https://github.com/kuotient/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .
```
```
lm_eval --model hf \
--model_args pretrained=yanolja/EEVE-Korean-Instruct-2.8B-v1.0 \
--tasks gsm8k-ko \
--device cuda:0 \
--batch_size auto:4
```
| Model | gsm8k(pass@1) | boolq(acc) | copa(acc) | hellaswag(acc) | Overall |
|---|---|---|---|---|---|
| yanolja/EEVE-Korean-10.8B-v1.0 | 0.4049 | - | - | - | - | - |
| yanolja/EEVE-Korean-Instruct-10.8B-v1.0 | 0.4511 | **0.8668** | **0.7450** | **0.4940** | 0.6392 |
| **EEVE-Math-10.8B** | **0.5390** | 0.8027 | 0.7260 | 0.4760 | 0.6359 |
| [**EEVE-Instruct-Math-10.8B**](https://huggingface.co/kuotient/EEVE-Instruct-Math-10.8B) | 0.4951 | 0.8283 | 0.7500 | 0.4880 | **0.6403** | |