metadata

language:
  - en
  - it
license: apache-2.0
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - llama
  - trl
  - sft

Meta LLaMA 3.1 8B 4-bit Finetuned Model

This model is a fine-tuned version of Meta-Llama-3.1-8B, developed by ruslanmv for text generation tasks. It leverages 4-bit quantization, making it more efficient for inference while maintaining strong performance in natural language generation.

Model Details

Base Model: unsloth/meta-llama-3.1-8b-bnb-4bit
Finetuned by: ruslanmv
Language: English
License: Apache 2.0
Tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft

Model Usage

Installation

To use this model, you will need to install the necessary libraries:

pip install transformers accelerate bitsandbytes

Loading the Model in Python

Here’s an example of how to load this fine-tuned model using Hugging Face's transformers library:

#!pip install bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# Define the quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

# Ensure you have the right device setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load the model and tokenizer from the Hugging Face Hub with BitsAndBytesConfig
model_name = "ruslanmv/Meta-Llama-3.1-8B-Text-to-SQL-4bit"
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    device_map="auto", 
    quantization_config=bnb_config)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Define EOS token for terminating the sequences
EOS_TOKEN = tokenizer.eos_token

# Define Alpaca-style prompt template
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
"""

# Format the prompt without the response part
prompt = alpaca_prompt.format(
    "Provide the SQL query",
    "Seleziona tutte le colonne della tabella table1 dove la colonna anni è uguale a 2020"
)

# Tokenize the prompt and generate text
inputs = tokenizer([prompt], return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=64, use_cache=True)

# Decode the generated text
generated_text = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]

# Extract the generated response only (remove the prompt part)
response_start = generated_text.find("### Response:") + len("### Response:\n")
response = generated_text[response_start:].strip()

# Print the response (excluding the prompt)
print(response)

and the ansewer is

SELECT * FROM table1 WHERE anni = 2020

Model Features

Text Generation: This model is fine-tuned to generate coherent and contextually accurate text based on the provided input.
Efficiency: Using 4-bit quantization with the bitsandbytes library, it optimizes memory and inference performance.

License

This model is licensed under the Apache 2.0 License. You are free to use, modify, and distribute this model, provided that you comply with the license terms.

Acknowledgments

This model was fine-tuned by ruslanmv based on the original work of unsloth and the meta-llama-3.1-8b-bnb-4bit model.