---
license: other
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
base_model: google/gemma-2b
datasets:
- generator
model-index:
- name: gemma2bit-sql
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# gemma2bit-sql

This model is a fine-tuned version of [google/gemma-2b](https://huggingface.co/google/gemma-2b) on the generator dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7133

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 4
- eval_batch_size: 8
- seed: 1399
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- lr_scheduler_warmup_steps: 10
- training_steps: 100

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.7834        | 0.02  | 5    | 1.5323          |
| 1.383         | 0.04  | 10   | 1.1950          |
| 1.1294        | 0.06  | 15   | 1.0211          |
| 0.964         | 0.08  | 20   | 0.9335          |
| 0.9112        | 0.09  | 25   | 0.8822          |
| 0.8775        | 0.11  | 30   | 0.8455          |
| 0.8375        | 0.13  | 35   | 0.8225          |
| 0.8109        | 0.15  | 40   | 0.8011          |
| 0.783         | 0.17  | 45   | 0.7862          |
| 0.7784        | 0.19  | 50   | 0.7738          |
| 0.7698        | 0.21  | 55   | 0.7625          |
| 0.7627        | 0.23  | 60   | 0.7524          |
| 0.7599        | 0.24  | 65   | 0.7461          |
| 0.7482        | 0.26  | 70   | 0.7406          |
| 0.7368        | 0.28  | 75   | 0.7338          |
| 0.7407        | 0.3   | 80   | 0.7293          |
| 0.7211        | 0.32  | 85   | 0.7243          |
| 0.718         | 0.34  | 90   | 0.7214          |
| 0.7187        | 0.36  | 95   | 0.7167          |
| 0.7251        | 0.38  | 100  | 0.7133          |


### Framework versions

- PEFT 0.9.0
- Transformers 4.38.2
- Pytorch 2.1.0+cu118
- Datasets 2.18.0
- Tokenizers 0.15.2