File size: 4,346 Bytes
70a39e8 4183238 70a39e8 4183238 70a39e8 63b8988 70a39e8 133ccba 70a39e8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 |
---
language:
- pt
license: llama3
library_name: transformers
tags:
- code
- sql
- finetuned
- portugues-BR
base_model: meta-llama/Meta-Llama-3-8B-Instruct
---
**Lloro SQL**
<img src="https://cdn-uploads.huggingface.co/production/uploads/653176dc69fffcfe1543860a/h0kNd9OTEu1QdGNjHKXoq.png" width="300" alt="Lloro-7b Logo"/>
Lloro SQL, developed by Semantix Research Labs, is a language Model that was trained to effectively transform Portuguese queries into SQL Code. It is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct, that was trained on GretelAI public datasets. The fine-tuning process was performed using the QLORA metodology on a GPU A100 with 40 GB of RAM.
**Model description**
Model type: A 7B parameter fine-tuned on GretelAI public datasets.
Language(s) (NLP): Primarily Portuguese, but the model is capable to understand English as well
Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct
**What is Lloro's intended use(s)?**
Lloro is built for Text2SQL in Portuguese contexts .
Input : Text
Output : Text (Code)
**Usage**
Using an OpenAI compatible inference server (like [vLLM](https://docs.vllm.ai/en/latest/index.html))
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY",
base_url="http://localhost:8000/v1",
)
def generate_responses(instruction, client=client):
chat_response = client.chat.completions.create(
model=<model>,
messages=[
{"role": "system", "content": "Você escreve a instrução SQL que responde às perguntas feitas. Você NÃO FORNECE NENHUM COMENTÁRIO OU EXPLICAÇÃO sobre o que o código faz, apenas a instrução SQL terminando em ponto e vírgula. Você utiliza todos os comandos disponíveis na especificação SQL, como: [SELECT, WHERE, ORDER, LIMIT, CAST, AS, JOIN]."},
{"role": "user", "content": instruction},
]
)
return chat_response.choices[0].message.content
output = generate_responses(user_prompt)
```
**Params**
Training Parameters
| Params | Training Data | Examples | Tokens | LR |
|----------------------------------|---------------------------------|---------------------------------|------------|--------|
| 8B | GretelAI public datasets | 65000 | 18.000.000 | 9e-5 |
**Model Sources**
GretelAI: https://huggingface.co/datasets/gretelai/synthetic_text_to_sql
**Performance**
| Modelo | LLM as Judge | Code Bleu Score | Rouge-L | CodeBert- Precision | CodeBert-Recall | CodeBert-F1 | CodeBert-F3 |
|----------------|--------------|-----------------|---------|----------------------|-----------------|-------------|-------------|
| Llama 3 - Base | 65.48% | 0.4583 | 0.6361 | 0.8815 | 0.8871 | 0.8835 | 0.8862 |
| Llama 3 - FT | 62.57% | 0.6512 | 0.7965 | 0.9458 | 0.9469 | 0.9459 | 0.9466 |
**Training Infos:**
The following hyperparameters were used during training:
| Parameter | Value |
|---------------------------|----------------------|
| learning_rate | 1e-4 |
| weight_decay | 0.001 |
| train_batch_size | 16 |
| eval_batch_size | 8 |
| seed | 42 |
| optimizer | Adam - adamw_8bit |
| lr_scheduler_type | cosine |
| num_epochs | 3.0 |
**QLoRA hyperparameters**
The following parameters related with the Quantized Low-Rank Adaptation and Quantization were used during training:
| Parameter | Value |
|-----------------|---------|
| lora_r | 16 |
| lora_alpha | 64 |
| lora_dropout | 0 |
**Framework versions**
| Library | Version |
|---------------|-----------|
| accelerate | 0.21.0 |
| bitsandbytes | 0.42.0 |
| Datasets | 2.14.3 |
| peft | 0.4.0 |
| Pytorch | 2.0.1 |
| safetensors | 0.4.1 |
| scikit-image | 0.22.0 |
| scikit-learn | 1.3.2 |
| Tokenizers | 0.14.1 |
| Transformers | 4.37.2 |
| trl | 0.4.7 |
|