Hugging Face's logo

raincandy-u
/
Llama-3-5B-Sheard

Text Generation
Transformers
Safetensors
PyTorch
English
llama
facebook
meta
llama-3
conversational
Inference Endpoints
text-generation-inference
Model card Files Community
Edit model card

image/png

Llama-3-5B-Sheard

Pruned version of Llama-3-8b.

Tool used: PrunMe, Mergekit.

Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Training

After sliced by mergekit, the model is continue-pretrained on minipile for 1 epoch and ~100k samples. Then we trained it using ORPO on Llama-3-70b generated DPO pairs.

Disclaimer

This model is for testing purposes only, and when the system prompt is not empty, the output may repeat and not stop!

Join our discord

Downloads last month
703
Safetensors
Model size
5.85B params
Tensor type
BF16
·
Inference API
Text Generation
Examples
Input a message to start chatting with raincandy-u/Llama-3-5B-Sheard.
Model is too large to load in Inference API (serverless). To try the model, launch it on Inference Endpoints (dedicated) instead.

Datasets used to train raincandy-u/Llama-3-5B-Sheard