Gemma_ft_Quote
This model is a fine-tuned version of google/gemma-7b on the english quote dataset using LoRA. It is based on the example provided by google here. The notebook used to fine-tune the model can be found here
Model description
The model can complete popular quotes given to it and add the author of the quote. For example, Given the qoute below:
Quote: With great power comes
The model would complete the quote and add the author of the quote:
Quote: With great power comes great responsibility. Author: Ben Parker.
Given a complete Quoute the model would add the author:
Quote: I'll be back. Author: Arnold Schwarzenegger.
Usage
The model can be used with transformers library. Here's an example of loading the model in 4 bit quantization mode:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
model_id = "Eteims/gemma_ft_quote"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="cuda:0")
This code would easily run in a free colab tier.
After loading the model you can use it for inference:
text = "Quote: Elementary, my dear watson."
device = "cuda:0"
inputs = tokenizer(text, return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training hyperparameters
The following hyperparameters were used during fine-tuning:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2
- training_steps: 10
- mixed_precision_training: Native AMP
Framework versions
- PEFT 0.8.2
- Transformers 4.38.1
- Pytorch 2.3.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2