|
--- |
|
language: |
|
- en |
|
- de |
|
- fr |
|
- it |
|
- pt |
|
- hi |
|
- es |
|
- th |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
license: llama3.2 |
|
base_model: NousResearch/Llama-3.2-1B |
|
tags: |
|
- generated_from_trainer |
|
- facebook |
|
- meta |
|
- pytorch |
|
- llama |
|
- llama-3 |
|
model-index: |
|
- name: llama3.2-1b-synthia-i |
|
results: [] |
|
--- |
|
|
|
# Llama 3.2 1B - Synthia-v1.5-I - Redmond - Fine-tuned Model |
|
|
|
This model is a fine-tuned version of [NousResearch/Llama-3.2-1B](https://huggingface.co/NousResearch/Llama-3.2-1B) on the [Synthia-v1.5-I](https://huggingface.co/datasets/migtissera/Synthia-v1.5-I) dataset. |
|
|
|
Thanks [RedmondAI](https://redmond.ai) for all the GPU Support! |
|
|
|
## Model Description |
|
|
|
The base model is Llama 3.2 1B, a multilingual large language model developed by Meta. This version has been fine-tuned on the Synthia-v1.5-I instruction dataset to improve its instruction-following capabilities. |
|
|
|
### Training Data |
|
|
|
The model was fine-tuned on Synthia-v1.5-I, which contains: |
|
- 20.7k training examples |
|
|
|
|
|
### Training Procedure |
|
|
|
The model was trained with the following hyperparameters: |
|
- Learning rate: 2e-05 |
|
- Train batch size: 1 |
|
- Eval batch size: 1 |
|
- Seed: 42 |
|
- Gradient accumulation steps: 8 |
|
- Total train batch size: 8 |
|
- Optimizer: Paged AdamW 8bit (betas=(0.9,0.999), epsilon=1e-08) |
|
- LR scheduler: Cosine with 100 warmup steps |
|
- Number of epochs: 3 |
|
|
|
### Framework Versions |
|
- Transformers 4.46.1 |
|
- Pytorch 2.3.1+cu121 |
|
- Datasets 3.0.1 |
|
- Tokenizers 0.20.3 |
|
|
|
## Intended Use |
|
|
|
This model is intended for: |
|
- Instruction following tasks |
|
- Conversational AI applications |
|
- Research and development in natural language processing |
|
|
|
## Training Infrastructure |
|
|
|
The model was trained using the Axolotl framework version 0.5.0. |
|
|
|
|
|
## License |
|
|
|
This model is subject to the Llama 3.2 Community License Agreement. Users must comply with all terms and conditions specified in the license. |
|
|
|
[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl) |
|
|
