|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- LeoLM/OpenSchnabeltier |
|
language: |
|
- de |
|
pipeline_tag: conversational |
|
tags: |
|
- hermeo |
|
- laser |
|
--- |
|
|
|
(Evaluation WIP) |
|
|
|
## Hermes + Leo + German Laser = Germeo |
|
|
|
Germeo-7B-AWQ |
|
A German-English understanding, but German-only speaking model merged from Hermeo-7B. |
|
|
|
### Model details |
|
|
|
**Merged from**: leo-mistral-hessianai-7b-chat and DPOpenHermes-7B-v2 |
|
|
|
**Model type**: Causal decoder-only transformer language model |
|
|
|
**Languages**: German replies with English Understanding Capabilities |
|
|
|
**Laser-Data**: LeoLM/OpenSchnabeltier |
|
|
|
|
|
This is an early experiment on laser and its influence on language understanding. It generally improves the language understanding capabilities. |
|
The hypothesis is that it degrades the probability of English replies and increasing those of German replies. The models internal German capabilities are boosted. |
|
|
|
Will keep you updated.. |
|
|
|
### German benchmarks |
|
|
|
| **German tasks:** | **MMLU-DE** | **Hellaswag-DE** | **ARC-DE** |**Average** | |
|
|-------------------------------|-------------|---------------|--------------|--------------| |
|
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | | |
|
| _7B parameters_ | | | | | |
|
| llama-2-7b | 0.400 | 0.513 | 0.381 | 0.431 | |
|
| leo-hessianai-7b | 0.400 | 0.609 | 0.429 | 0.479 | |
|
| bloom-6b4-clp-german | 0.274 | 0.550 | 0.351 | 0.392 | |
|
| mistral-7b | **0.524** | 0.588 | 0.473 | 0.528 | |
|
| leo-mistral-hessianai-7b | 0.481 | 0.663 | 0.485 | 0.543 | |
|
| leo-mistral-hessianai-7b-chat | 0.458 | 0.617 | 0.465 | 0.513 | |
|
| DPOpenHermes-7B-v2 | 0.517 | 0.603 | 0.515 | 0.545 | |
|
| hermeo-7b | 0.511 | **0.668** | **0.528** | **0.569** | |
|
| **germeo-7b-laser (this model)**| ? | ? | ? | ? | |
|
| _13B parameters_ | | | | | |
|
| llama-2-13b | 0.469 | 0.581 | 0.468 | 0.506 | |
|
| leo-hessianai-13b | **0.486** | **0.658** | **0.509** | **0.551** | |
|
| _70B parameters_ | | | | | |
|
| llama-2-70b | 0.597 | 0.674 | 0.561 | 0.611 | |
|
| leo-hessianai-70b | **0.653** | **0.721** | **0.600** | **0.658** | |
|
|
|
|
|
Even though the model does not generate English text without being explicitly asked, performance on English Benchmarks is still up: |
|
|
|
### English benchmarks |
|
|
|
| **English tasks:** | **MMLU** | **Hellaswag** | **ARC** | **Average** | |
|
|------------------------------------|-------------|---------------|--------------|-------------| |
|
| **Models / Few-shots:** | _(5 shots)_ | _(10 shots)_ | _(24 shots)_ | | |
|
| llama-2-7b | 0.466 | 0.786 | 0.530 | 0.594 | |
|
| leolm-hessianai-7b | 0.423 | 0.759 | 0.522 | 0.568 | |
|
| bloom-6b4-clp-german | 0.264 | 0.525 | 0.328 | 0.372 | |
|
| mistral-7b | **0.635** | **0.832** | 0.607 | **0.691** | |
|
| leolm-mistral-hessianai-7b | 0.550 | 0.777 | 0.518 | 0.615 | |
|
| hermeo-7b | 0.601 | 0.821 | **0.620** | 0.681 | |
|
| germeo-7b-laser (this model) | 0.601 | 0.828 | 0.608 | 0.679 | |
