JinaJudge: Proxy Judgement for Russian LLM Arena
Description
This model is trained to replicate the judgement patterns of GPT-4-1106-Preview in the Russian LLM Arena, designed for faster and more cost-effective evaluation of language models. While the model's focus is on Russian LLM evaluation, it can also be used for English-centric models.
Model Details
This is a small upgrade to the kaleinaNyan/jina-v3-rullmarena-judge model:
- Number of decoder blocks increased from 4 to 5.
- Hidden activations dimensionality reduced from 1024 to 512 (via a projection layer after JINA encoder).
- The resulting model size went from 614M params to 589M params.
- I also tweaked some training hyperparameters, but training data composition is the same.
Surprisingly, these changes gave a tangible performance improvement, so I decided to upload the model. As it turned out (after evaluation on the train set), previous model was not expressive enough.
Evaluation
The validation process was based on existing judgements from the Russian LLM Arena, which were already available. These judgements were filtered and simplified to match the three-class structure used in training.
NOTE: values in parenthesis show relative improvement compared to previous model.
Models evaluated:
- gemma-2-9b-it-sppo-iter3
- glm-4-9b-chat
- gpt-3.5-turbo-1106
- mistral-7b-instruct-v0.3
- storm-7b
Validation Performance:
- Accuracy: 80.76% (+2.67)
- Precision: 78.56% (+2.74)
- Recall: 79.48% (+2.71)
- F1-score: 79.00% (+2.73)
For the test phase, new judgements were generated using GPT-4 for the kolibri-mistral-0427-upd model.
Test Performance:
- Accuracy: 82.72% (+2.64)
- Precision: 80.11% (+3.43)
- Recall: 82.42% (+4.69)
- F1-score: 81.18% (+4.10)
Usage Example
from transformers import AutoModel
jina = AutoModel.from_pretrained("kaleinaNyan/jina-v3-rullmarena-judge-300924", trust_remote_code=True)
prompt_template = """
<user prompt>
{user_prompt}
<end>
<assistant A answer>
{assistant_a}
<end>
<assistant B answer>
{assistant_b}
<end>
""".strip()
prompt = "your prompt"
assistant_a = "assistant a response"
assistant_b = "assistant b response"
example = prompt_template.format(
user_prompt=user_prompt,
assistant_a=assistant_a,
assistant_b=assistant_b,
)
judgement = jina([example])[0].argmax()
judgement_map = {
0: "A is better than B",
1: "A == B",
2: "B is better than A"
}
print(judgement_map[judgement])
Generated ranking
The ranking was obtained using a modified Russian LLM Arena code. All judgements were regenerated using the jina-judge model.
| Model | Score | 95% CI | Average #Tokens |
|---|---|---|---|
| gpt-4-1106-preview | 81.6 | (-2.3, 3.0) | 541 |
| gpt-4.0-mini | 76.0 | (-2.7, 2.4) | 448 |
| qwen-2.5-72b-it | 72.5 | (-3.6, 3.6) | 557 |
| gemma-2-9b-it-sppo-iter3 | 72.1 | (-3.7, 3.6) | 569 |
| gemma-2-27b-it | 71.1 | (-3.3, 3.2) | 482 |
| gemma-2-9b-it | 70.8 | (-3.4, 3.5) | 569 |
| t-lite-instruct-0.1 | 68.3 | (-3.8, 4.5) | 810 |
| suzume-llama-3-8b-multilingual-orpo | 62.9 | (-3.9, 4.0) | 682 |
| glm-4-9b-chat | 60.5 | (-3.9, 4.0) | 516 |
| sfr-iterative-dpo-llama-3-8b-r | 59.9 | (-4.0, 4.3) | 682 |
| c4ai-command-r-v01 | 56.9 | (-4.2, 3.8) | 516 |
| phi-3-medium-4k-instruct | 56.4 | (-2.8, 3.3) | 566 |
| mistral-nemo-instruct-2407 | 56.1 | (-2.9, 3.4) | 682 |
| yandex_gpt_pro | 51.7 | (-3.4, 3.4) | 345 |
| suzume-llama-3-8b-multilingual | 51.3 | (-3.4, 4.0) | 489 |
| hermes-2-theta-llama-3-8b | 50.9 | (-3.2, 3.4) | 485 |
| starling-1m-7b-beta | 50.2 | (-3.3, 3.4) | 495 |
| gpt-3.5-turbo-0125 | 50.0 | (0.0, 0.0) | 220 |
| llama-3-instruct-8b-sppo-iter3 | 49.8 | (-3.4, 4.0) | 763 |
| llama-3-8b-saiga-suzume-ties | 48.2 | (-4.1, 3.9) | 569 |
| llama-3-smaug-8b | 46.6 | (-3.9, 3.8) | 763 |
| vikhr-it-5.4-fp16-orpo-v2 | 46.6 | (-3.7, 4.0) | 379 |
| aya-23-8b | 46.3 | (-3.8, 3.9) | 571 |
| saiga-llama3-8b_v6 | 45.5 | (-3.8, 3.9) | 471 |
| vikhr-it-5.2-fp16-cp | 43.8 | (-3.9, 4.0) | 543 |
| qwen2-7b-instruct | 43.7 | (-2.5, 2.7) | 492 |
| opencchat-3.5-0106 | 43.4 | (-3.3, 3.7) | 485 |
| gpt-3.5-turbo-1106 | 41.7 | (-2.9, 3.5) | 220 |
| kolibri-mistral-0427-upd | 41.5 | (-3.2, 3.5) | 551 |
| paralex-llama-3-8b-sft | 40.6 | (-3.8, 3.3) | 688 |
| mistral-7b-instruct-v0.3 | 40.3 | (-3.3, 3.4) | 469 |
| llama-3-instruct-8b-simpo | 40.2 | (-2.9, 3.7) | 551 |
| gigachat_pro | 40.2 | (-3.2, 3.5) | 294 |
| hermes-2-pro-llama-3-8b | 39.5 | (-2.9, 3.4) | 689 |
| vikhr-it-5.3-fp16-32k | 39.5 | (-2.8, 3.2) | 519 |
| opencchat-3.6-8b-2204522 | 37.7 | (-3.3, 3.7) | 409 |
| meta-llama-3-8b-instruct | 37.5 | (-3.1, 3.5) | 450 |
| kolibri-vikhr-mistral-0427 | 37.1 | (-3.1, 3.8) | 488 |
| neural-chat-v3.3 | 36.5 | (-2.7, 3.6) | 523 |
| vikhr-it-5.1-fp16 | 36.4 | (-3.5, 3.5) | 448 |
| gigachat-lite | 36.0 | (-2.8, 3.0) | 523 |
| saiga-7b | 25.9 | (-3.1, 3.7) | 927 |
| storm-7b | 25.1 | (-3.6, 4.1) | 419 |
| snorkel-mistral-pairrm-dpo | 16.5 | (-3.8, 3.2) | 773 |
- Downloads last month
- 11
Model tree for kaleinaNyan/jina-v3-rullmarena-judge-300924
Base model
jinaai/jina-embeddings-v3