Commit
•
7e34145
1
Parent(s):
57f36e6
Update README.md
Browse files
README.md
CHANGED
|
@@ -4,6 +4,22 @@
|
|
| 4 |
|
| 5 |
See https://huggingface.co/shisa-ai/shisa-v1-llama3-70b for the working model
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
Quick and dirty GGUF quants. Maybe some iMatrix soon. BF16 conversion included in this repo.
|
| 8 |
|
| 9 |
split:
|
|
|
|
| 4 |
|
| 5 |
See https://huggingface.co/shisa-ai/shisa-v1-llama3-70b for the working model
|
| 6 |
|
| 7 |
+
First turn seems to work well (eg, benchmarks fine) but after about turn three, the model starts to output random tokens...
|
| 8 |
+
|
| 9 |
+
## Performance
|
| 10 |
+
Measured using a [fork](https://github.com/shisa-ai/shaberi) of [Lightblue's Shaberi benchmark framework](https://github.com/lightblue-tech/japanese_llm_eval):
|
| 11 |
+
|
| 12 |
+
## Performance
|
| 13 |
+
Measured using a [fork](https://github.com/shisa-ai/shaberi) of [Lightblue's Shaberi benchmark framework](https://github.com/lightblue-tech/japanese_llm_eval):
|
| 14 |
+
|
| 15 |
+
| Model | Average | ELYZA-tasks-100 | MT-Bench | Rakuda | Tengu-Bench |
|
| 16 |
+
|----------------------------------------|---------|-----------------|----------|--------|-------------|
|
| 17 |
+
| **shisa-ai/shisa-v1-llama3-70b** | **7.30**| **7.34** | **7.67** | **8.15** | **6.04** |
|
| 18 |
+
| **shisa-ai/shisa-v1-llama3-70b.Q4_K_M**| **7.22**| **7.22** | **7.27** | **8.20** | **6.19** |
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
---
|
| 22 |
+
|
| 23 |
Quick and dirty GGUF quants. Maybe some iMatrix soon. BF16 conversion included in this repo.
|
| 24 |
|
| 25 |
split:
|