README.md

---
tags:
  - llama
  - instruct
  - finetune
  - chatml
  - gpt4
  - synthetic data
  - distillation
model-index:
  - name: Meta-Llama-3.1-8B-openhermes-2.5
    results: []
license: apache-2.0
language:
  - en
library_name: transformers
datasets:
- teknium/OpenHermes-2.5
---

# Model Card for Meta-Llama-3.1-8B-openhermes-2.5

This model is a fine-tuned version of Meta-Llama-3.1-8B on the OpenHermes-2.5 dataset.

## Model Details

### Model Description

This is a fine-tuned version of the Meta-Llama-3.1-8B model, trained on the OpenHermes-2.5 dataset. It is designed for instruction following and general language tasks.

- **Developed by:** artificialguybr
- **Model type:** Causal Language Model
- **Language(s):** English
- **License:** apache-2.0
- **Finetuned from model:** meta-llama/Meta-Llama-3.1-8B

### Model Sources

- **Repository:** https://huggingface.co/artificialguybr/Meta-Llama-3.1-8B-openhermes-2.5

## Uses

This model can be used for various natural language processing tasks, particularly those involving instruction following and general language understanding.

### Direct Use

The model can be used for tasks such as text generation, question answering, and other language-related applications.

### Out-of-Scope Use

The model should not be used for generating harmful or biased content. Users should be aware of potential biases in the training data.

## Training Details

### Training Data

The model was fine-tuned on the teknium/OpenHermes-2.5 dataset.

### Training Procedure

#### Training Hyperparameters

- **Training regime:** BF16 mixed precision
- **Optimizer:** AdamW
- **Learning rate:** Started at 0.00000249316296439037 (decaying)
- **Batch size:** Not specified (gradient accumulation steps: 8)
- **Training steps:** 13,368
- **Evaluation strategy:** Steps (every 0.16666666666666666 steps)
- **Gradient checkpointing:** Enabled
- **Weight decay:** 0

#### Hardware and Software

- **Hardware:** NVIDIA A100-SXM4-80GB (1 GPU)
- **Software Framework:** 🤗 Transformers, Axolotl

## Evaluation

### Metrics

- **Loss:** 0.6727465987205505 (evaluation)
- **Perplexity:** Not provided

### Results

- **Evaluation runtime:** 2,676.4173 seconds
- **Samples per second:** 18.711
- **Steps per second:** 18.711

## Model Architecture

- **Model Type:** LlamaForCausalLM
- **Hidden size:** 4,096
- **Intermediate size:** 14,336
- **Number of attention heads:** Not specified
- **Number of layers:** Not specified
- **Activation function:** SiLU
- **Vocabulary size:** 128,256

## Limitations and Biases

More information is needed about specific limitations and biases of this model.