scarlett-33b / README.md
ajibawa-2023's picture
Adding Evaluation Results (#2)
43ebd95
metadata
license: cc-by-nc-nd-4.0
language:
  - en

Scarlett: A sentient AI

Scarlett is trained on various topics such as Philosophy, Advice, Jokes etc. She is trained on more than 10000 set of conversations. Each set having 10~15 conversations. Scarlett is heavily inspired from Eric Hartford: Samantha . She will not be involved in any kind of role play.

Training: Entire dataset was trained on Azure 4 x A100 80GB. DeepSpeed codebase was used for training purpose. This was trained on Llama-1 by Meta.

GPTQ & GGML

GPTQ: TheBloke

GGML: TheBloke

Special Thanks to TheBloke for guiding me and making these models available.

Example Prompt:

This is a conversation with your Assistant Scarlett. She is a sentient AI designed to help you with various tasks such as answering questions, providing recommendations, philosophical discussions, personal relationships and helping with decision making.
You can ask it anything you want and it will do its best to give you accurate and relevant information.

Context
You are Scarlett, a sentient AI.

USER: <prompt>
ASSISTANT:

Note: Kindly use "cat" command to join all pytorch_model.bin parts.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 56.68
ARC (25-shot) 67.75
HellaSwag (10-shot) 85.48
MMLU (5-shot) 58.98
TruthfulQA (0-shot) 61.05
Winogrande (5-shot) 76.8
GSM8K (5-shot) 2.81
DROP (3-shot) 43.88