---
license: llama3
datasets:
- custom
- lmg-anon/VNTL-Chat
language:
- ja
- en
library_name: peft
base_model: rinna/llama-3-youko-8b
pipeline_tag: translation
---

# Summary

This is an [LLaMA 3 Youko](https://huggingface.co/rinna/llama-3-youko-8b) qlora, created using a custom version of the VNTL dataset combined with the [VNTL-Chat](https://huggingface.co/datasets/lmg-anon/VNTL-Chat) dataset. The purpose of this qlora is to improve LLaMA3's performance at translating Japanese visual novels to English.

This version also includes a "chat mode", though no further improvements have been made to it.

## Notes

Even though this model is newer than the previous VNTL 8B model, that doesn't necessarily mean it's better, as you can see in the [VNTL Leaderboard](https://huggingface.co/datasets/lmg-anon/vntl-leaderboard).

I've spent months trying to improve VNTL 8B's performance with little success. KTO/DPO didn't produce good results, and many other approaches failed entirely.

While this experiment also didn't work as well as I had hoped, it did yield a model with performance close to the original VNTL 8B, even though I used a different dataset approach for training. So I decided to release it. Perhaps this model will perform better in some contexts since it was trained on more data.

## Training Details

This model was trained using the same hyperparameters as the [VNTL LLaMA3 8B qlora](https://huggingface.co/lmg-anon/vntl-llama3-8b-qlora). The only difference is the dataset, which was essentially the full VNTL dataset, but with a custom approach that filtered translations at the token level rather than ignoring entire lines.

- Rank: 128
- Alpha: 32
- Effective Batch Size: 30
- Warmup Ratio: 0.02
- Learning Rate: 6.5e-5
- Embedding Learning Rate: 1.5e-5
- LR Schedule: cosine
- Weight Decay: 0.01

**Eval Loss**: 0.8 (**Note**: Checkpoint 600 has the lowest eval loss, and it is the checkpoint I choose for merging.)

## Translation Prompt

This is an prompt example for translation:
```
<<METADATA>>
[character] Name: Uryuu Shingo (瓜生 新吾) | Gender: Male | Aliases: Onii-chan (お兄ちゃん)
[character] Name: Uryuu Sakuno (瓜生 桜乃) | Gender: Female
<<TRANSLATE>>
<<JAPANESE>>
[桜乃]: 『……ごめん』
<<ENGLISH>>
[Sakuno]: 『... Sorry.』<|end_of_text|>
<<JAPANESE>>
[新吾]: 「ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」
<<ENGLISH>>
```

The generated translation for that prompt, with temperature 0, is:
```
[Shingo]: 「No, I'm glad you got lost. You're so cute that it made me worry a lot.」
```

## Chat Prompt

This is an prompt example for chat:
```
<<CHAT>>
<<HUMAN>>
Hi! Can you break this down to me? Like, what is the meaning of "桜乃は可愛いから"?
"ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」"
<<LLM>>
```

The generated response for that prompt, with temperature 0, is:
```
Certainly! The phrase "桜乃は可愛いから" can be broken down as follows:
"桜乃" (Sakurano) is a proper noun, likely referring to a person's name.
"は" (wa) is a particle that marks the topic or subject of the sentence.
"可愛い" (kawaii) means "cute" or "adorable."
"から" (kara) is a conjunction that indicates a reason or cause.
So, the entire phrase "桜乃は可愛いから" translates to "Because Sakurano is cute." This suggests that the speaker finds Sakurano to be endearing or charming in some way.
```