language:
- en
- pl
pipeline_tag: text-generation
inference: false
tags:
- voicelab
- pytorch
- llama-2
- trurl
- trurl-2
Trurl 2 -- Polish Llama 2
The new OPEN TRURL is a finetuned Llama 2, trained on over 1.7b tokens (970k conversational Polish and English samples) with a large context of 4096 tokens. TRURL was trained on a large number of Polish data. TRURL 2 is a collection of fine-tuned generative text models with 7 billion and 13 billion parameters. This is the repository for the 7b fine-tuned model, optimized for dialogue use cases.
Overview
TRURL developers Voicelab.AI
Variations Trurl 2 comes in 7B and 13B versions.
Input Models input text only.
Output Models generate text only.
Model Architecture Trurl is an auto-regressive language model that uses an optimized transformer architecture.
| Training Data | Params | Content Length | Num. Samples | Num. Tokens | start LR | |
|---|---|---|---|---|---|---|
| Trurl 2 | A new mix of private and publicly available online data | 7B | 4k | 970k | 1.7b | 2.0 x 10-5 |
| Trurl 2 | A new mix of private and publicly available online data | 13B | 4k | 970k | 1.7b | 2.0 x 10-5 |
Training data
The training data includes Q&A pairs from various sources including Alpaca comparison data with GPT, Falcon comparison data, Dolly 15k, Oasst1, Phu saferlfhf, ShareGPT version 2023.05.08v0 filtered and cleaned, Voicelab private datasets for JSON data extraction, modification, and analysis, CURLICAT dataset containing journal entries, dataset from Polish wiki with Q&A pairs grouped into conversations, MMLU data in textual format, Voicelab private dataset with sales conversations, arguments and objections, paraphrases, contact reason detection, and corrected dialogues.
Intended Use
Trurl 2 is intended for commercial and research use in Polish and English. Tuned models are intended for assistant-like chat, but also adapted for a variety of natural language generation tasks.
To get the expected features and performance for the chat versions, a specific Llama 2 formatting needs to be followed, including the INST and <<SYS>> tags, BOS and EOS tokens, and the whitespaces and breaklines in between (we recommend calling strip() on inputs to avoid double-spaces). See our reference code in github for details: chat_completion.
Evaluation Results
| Model | Size | hellaswag | arc_challenge | MMLU |
|---|---|---|---|---|
| Llama-2-chat | 7B | 78.55% | 52.9% | 48.32% |
| Llama-2-chat | 13B | 81.94% | 59.04% | 54.64% |
| Trurl 2.0 (with MMLU) | 13B | 80.09% | 59.30% | 78.35% |
| Trurl 2.0 (no MMLU) | 13B | TO-DO | TO-DO | TO-DO |
| Trurl 2.0 (no MMLU) | 7b | TO-DO | TO-DO | TO-DO |
Ethical Considerations and Limitations
Trurl 2, same as a Llama 2, is a new technology that carries risks with use. Testing conducted to date has been in Polish and English, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, Trurl 2’s potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of Trurl 2, developers should perform safety testing and tuning tailored to their specific applications of the model.
Please see the Meta's Responsible Use Guide available at https://ai.meta.com/llama/responsible-use-guide/
Authors
The model was trained by NLP Research Team at Voicelab.ai.
You can contact us here.