koesn/Llama-3-Smaug-8B-GGUF

Description

This repo contains GGUF format model files for Llama-3-Smaug-8B.

Files Provided

Name	Quant	Bits	File Size	Remark
llama-3-smaug-8b.Q2_K.gguf	Q2_K	2	3.18 GB	2.96G, +3.5199 ppl @ Llama-3-8B
llama-3-smaug-8b.Q3_K.gguf	Q3_K	3	4.02 GB	3.74G, +0.6569 ppl @ Llama-3-8B
llama-3-smaug-8b.Q4_0.gguf	Q4_0	4	4.66 GB	4.34G, +0.4685 ppl @ Llama-3-8B
llama-3-smaug-8b.Q4_K.gguf	Q4_K	4	4.92 GB	4.58G, +0.1754 ppl @ Llama-3-8B
llama-3-smaug-8b.Q5_K.gguf	Q5_K	5	5.73 GB	5.33G, +0.0569 ppl @ Llama-3-8B
llama-3-smaug-8b.Q6_K.gguf	Q6_K	6	6.60 GB	6.14G, +0.0217 ppl @ Llama-3-8B
llama-3-smaug-8b.Q8_0.gguf	Q8_0	8	8.54 GB	7.96G, +0.0026 ppl @ Llama-3-8B

Parameters

path	type	architecture	rope_theta	sliding_win	max_pos_embed
abacusai/Llama-3-Smaug-8B	llama	LlamaForCausalLM	500000.0	null	8192

Benchmarks

Original Model Card

Llama-3-Smaug-8B

Built with Meta Llama 3

This model was built using the Smaug recipe for improving performance on real world multi-turn conversations applied to meta-llama/Meta-Llama-3-8B-Instruct.

Model Description

• Developed by: Abacus.AI
• License: https://llama.meta.com/llama3/license/
• Finetuned from model: meta-llama/Meta-Llama-3-8B-Instruct.

Evaluation

MT-Bench

########## First turn ##########
                   score
model             turn
Llama-3-Smaug-8B 1   8.77500
Meta-Llama-3-8B-Instruct 1   8.31250
########## Second turn ##########
                   score
model             turn
Meta-Llama-3-8B-Instruct 2   7.8875 
Llama-3-Smaug-8B 2   7.8875
########## Average ##########
                 score
model
Llama-3-Smaug-8B  8.331250
Meta-Llama-3-8B-Instruct 8.10

Model	First turn	Second Turn	Average
Llama-3-Smaug-8B	8.78	7.89	8.33
Llama-3-8B-Instruct	8.31	7.89	8.10

This version of Smaug uses new techniques and new data compared to Smaug-72B, and more information will be released later on. For now, see the previous Smaug paper: https://arxiv.org/abs/2402.13228.