lmg-anon/vntl-llama3-8b-202409-qlora

Summary

This is an LLaMA 3 Youko qlora, created using a custom version of the VNTL dataset combined with the VNTL-Chat dataset. The purpose of this qlora is to improve LLaMA3's performance at translating Japanese visual novels to English.

This version also includes a "chat mode", though no further improvements have been made to it.

Notes

Even though this model is newer than the previous VNTL 8B model, that doesn't necessarily mean it's better, as you can see in the VNTL Leaderboard.

I've spent months trying to improve VNTL 8B's performance with little success. KTO/DPO didn't produce good results, and many other approaches failed entirely.

While this experiment also didn't work as well as I had hoped, it did yield a model with performance close to the original VNTL 8B, even though I used a different dataset approach for training. So I decided to release it. Perhaps this model will perform better in some contexts since it was trained on more data.

Training Details

This model was trained using the same hyperparameters as the VNTL LLaMA3 8B qlora. The only difference is the dataset, which was essentially the full VNTL dataset, but with a custom approach that filtered translations at the token level rather than ignoring entire lines.

• Rank: 128
• Alpha: 32
• Effective Batch Size: 30
• Warmup Ratio: 0.02
• Learning Rate: 6.5e-5
• Embedding Learning Rate: 1.5e-5
• LR Schedule: cosine
• Weight Decay: 0.01

Eval Loss: 0.8 (Note: Checkpoint 600 has the lowest eval loss, and it is the checkpoint I choose for merging.)

Translation Prompt

This is an prompt example for translation:

<<METADATA>>
[character] Name: Uryuu Shingo (瓜生 新吾) | Gender: Male | Aliases: Onii-chan (お兄ちゃん)
[character] Name: Uryuu Sakuno (瓜生 桜乃) | Gender: Female
<<TRANSLATE>>
<<JAPANESE>>
[桜乃]: 『……ごめん』
<<ENGLISH>>
[Sakuno]: 『... Sorry.』<|end_of_text|>
<<JAPANESE>>
[新吾]: 「ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」
<<ENGLISH>>

The generated translation for that prompt, with temperature 0, is:

[Shingo]: 「No, I'm glad you got lost. You're so cute that it made me worry a lot.」

Chat Prompt

This is an prompt example for chat:

<<CHAT>>
<<HUMAN>>
Hi! Can you break this down to me? Like, what is the meaning of "桜乃は可愛いから"?
"ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」"
<<LLM>>

The generated response for that prompt, with temperature 0, is:

Certainly! The phrase "桜乃は可愛いから" can be broken down as follows:
"桜乃" (Sakurano) is a proper noun, likely referring to a person's name.
"は" (wa) is a particle that marks the topic or subject of the sentence.
"可愛い" (kawaii) means "cute" or "adorable."
"から" (kara) is a conjunction that indicates a reason or cause.
So, the entire phrase "桜乃は可愛いから" translates to "Because Sakurano is cute." This suggests that the speaker finds Sakurano to be endearing or charming in some way.