Files changed (1) hide show
  1. README.md +23 -0
README.md CHANGED
@@ -41,6 +41,29 @@ tokenizer = AutoTokenizer.from_pretrained("MaziyarPanahi/Calme-7B-Instruct-v0.1.
41
  model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Calme-7B-Instruct-v0.1.1")
42
  ```
43
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
44
  ### Quantized Models
45
 
46
  > I love how GGUF democratizes the use of Large Language Models (LLMs) on commodity hardware, more specifically, personal computers without any accelerated hardware. Because of this, I am committed to converting and quantizing any models I fine-tune to make them accessible to everyone!
 
41
  model = AutoModelForCausalLM.from_pretrained("MaziyarPanahi/Calme-7B-Instruct-v0.1.1")
42
  ```
43
 
44
+ ### Eval
45
+
46
+
47
+ | Metric | [Mistral-7B Instruct v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | [Calme-7B v0.1](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.1) | [Calme-7B v0.2](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.2) | [Calme-7B v0.3](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.3) | [Calme-7B v0.4](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.4) | [Calme-7B v0.5](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.5) | [Calme-4x7B v0.1](https://huggingface.co/MaziyarPanahi/Calme-4x7B-MoE-v0.1) | [Calme-4x7B v0.2](https://huggingface.co/MaziyarPanahi/Calme-4x7B-MoE-v0.2) |
48
+ |-----------|--------------------------|-------|-------|-------|-------|-------|------------|------------|
49
+ | ARC | 63.14 | 67.24 | 67.75 | 67.49 | 64.85 | 67.58 | 67.15 | 76.66 |
50
+ | HellaSwag | 84.88 | 85.57 | 87.52 | 87.57 | 86.00 | 87.26 | 86.89 | 86.84 |
51
+ | TruthfulQA| 68.26 | 59.38 | 78.41 | 78.31 | 70.52 | 74.03 | 73.30 | 73.06 |
52
+ | MMLU | 60.78 | 64.97 | 61.83 | 61.93 | 62.01 | 62.04 | 62.16 | 62.16 |
53
+ | Winogrande| 77.19 | 83.35 | 82.08 | 82.32 | 79.48 | 81.85 | 80.82 | 81.06 |
54
+ | GSM8k | 40.03 | 69.29 | 73.09 | 73.09 | 77.79 | 73.54 | 74.53 | 75.66 |
55
+
56
+ Some extra information to help you pick the right `Calme-7B` model:
57
+
58
+ | Use Case Category | Recommended Calme-7B Model | Reason |
59
+ |-------------------------------------------------|-----------------------------|------------------------------------------------------------------------------------------|
60
+ | Educational Tools and Academic Research | [Calme-7B v0.5](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.5) | Balanced performance, especially strong in TruthfulQA for accuracy and broad knowledge. |
61
+ | Commonsense Reasoning and Natural Language Apps | [Calme-7B v0.2](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.2) or [Calme-7B v0.3](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.3) | High performance in HellaSwag for understanding nuanced scenarios. |
62
+ | Trustworthy Information Retrieval Systems | [Calme-7B v0.5](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.5) | Highest score in TruthfulQA, indicating reliable factual information provision. |
63
+ | Math Educational Software | [Calme-7B v0.4](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.4) | Best performance in GSM8k, suitable for numerical reasoning and math problem-solving. |
64
+ | Context Understanding and Disambiguation | [Calme-7B v0.5](https://huggingface.co/MaziyarPanahi/Calme-7B-Instruct-v0.5) | Solid performance in Winogrande, ideal for text with context and pronoun disambiguation. |
65
+
66
+
67
  ### Quantized Models
68
 
69
  > I love how GGUF democratizes the use of Large Language Models (LLMs) on commodity hardware, more specifically, personal computers without any accelerated hardware. Because of this, I am committed to converting and quantizing any models I fine-tune to make them accessible to everyone!