lmg-anon's picture
Update README.md
9bf47a6 verified
metadata
license: llama3
datasets:
  - lmg-anon/VNTL-v3.1-1k
  - lmg-anon/VNTL-Chat
language:
  - ja
  - en
library_name: peft
base_model: rinna/llama-3-youko-8b
pipeline_tag: translation

Summary

This is an LLaMA 3 Youko qlora created using a slightly modified version of the VNTL-v3.1-1k dataset, concatenated with the VNTL-Chat dataset.

This was trained mostly with the same hyperparameters as the VNTL 7B v0.3.1 lora, the differences are:

  • Added ["<<METADATA>>", "<<TRANSLATE>>", "<<JAPANESE>>", "<<ENGLISH>>", "<<CHAT>>", "<<HUMAN>>", "<<LLM>>"] as special tokens.
  • Trained the ["embed_tokens", "lm_head"] layers.
  • 10x smaller learning rate, 0.00065 -> 0.000065.

This version also includes a new "chat mode", which was lazily trained just to find out how that would impact the end result. I think this ended up quite good for breaking down or explaining Japanese sentences, but it is terrible for most other things, which is expected given the nature of the VNTL-Chat dataset. To be honest, I wasn't aiming for it to be good, so the fact that it works at all is very nice.

Eval Loss: 0.8

Translation Prompt

This is an prompt example for translation:

<<METADATA>>
[character] Name: Uryuu Shingo (η“œη”Ÿ 新吾) | Gender: Male | Aliases: Onii-chan (γŠε…„γ‘γ‚ƒγ‚“)
[character] Name: Uryuu Sakuno (η“œη”Ÿ ζ‘œδΉƒ) | Gender: Female
<<TRANSLATE>>
<<JAPANESE>>
[ζ‘œδΉƒ]: γ€Žβ€¦β€¦γ”γ‚γ‚“γ€
<<ENGLISH>>
[Sakuno]: γ€Ž... Sorry.』<|end_of_text|>
<<JAPANESE>>
[新吾]: γ€Œγ†γ†γ‚“γ€γ“γ†θ¨€γ£γ‘γ‚ƒγͺγ‚“γ γ‘γ©γ€θΏ·ε­γ§γ‚ˆγ‹γ£γŸγ‚ˆγ€‚ζ‘œδΉƒγ―ε―ζ„›γ„γ‹γ‚‰γ€γ„γ‚γ„γ‚εΏƒι…γ—γ‘γ‚ƒγ£γ¦γŸγ‚“γ γžδΏΊγ€
<<ENGLISH>>

The generated translation for that prompt, with temperature 0, is:

[Shingo]: γ€ŒNo, I know this isn't the best way to say it, but I'm glad you got lost. You're so cute that I was worried about you.」

Chat Prompt

This is an prompt example for chat:

<<CHAT>>
<<HUMAN>>
Hi! Can you break this down to me? Like, what is the meaning of "ζ‘œδΉƒγ―ε―ζ„›γ„γ‹γ‚‰"?
"ううん、こう言っけゃγͺγ‚“γ γ‘γ©γ€θΏ·ε­γ§γ‚ˆγ‹γ£γŸγ‚ˆγ€‚ζ‘œδΉƒγ―ε―ζ„›γ„γ‹γ‚‰γ€γ„γ‚γ„γ‚εΏƒι…γ—γ‘γ‚ƒγ£γ¦γŸγ‚“γ γžδΏΊγ€"
<<LLM>>

The generated response for that prompt, with temperature 0, is:

Certainly! The phrase "ζ‘œδΉƒγ―ε―ζ„›γ„γ‹γ‚‰" literally translates to "Because Sakura-no is cute." In Japanese culture, there is a strong emphasis on physical attractiveness and cuteness, especially when it comes to young girls or women. This phrase suggests that the speaker finds Sakura-no (presumably a girl) to be very attractive and endearing in appearance.