Hugging Face's logo

uukuguy
/
speechless-hermes-coig-lite-13b

Text Generation
Transformers
PyTorch
Safetensors
English
llama
llama-2
self-instruct
distillation
synthetic instruction
text-generation-inference
Inference Endpoints
Model card Files Community
1
uukuguy commited on
Commit
108a659
•
1 Parent(s): 841d633

Modified README.md

Browse files
Files changed (1) hide show
  1. README.md +165 -1
README.md CHANGED
@@ -1,3 +1,167 @@
 
1
  ---
2
- license: mit
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
  ---
3
+ language:
4
+ - en
5
+ tags:
6
+ - llama-2
7
+ - self-instruct
8
+ - distillation
9
+ - synthetic instruction
10
+ license:
11
+ - mit
12
  ---
13
+
14
+ # Model Card: speechless-hermes-coig-lite-13b
15
+
16
+ Fine-tune the Nous-Hermes-Llama2-13b with COIG-PC-LITE for Chinese capability.
17
+
18
+
19
+ # Model Card: Nous-Hermes-Llama2-13b
20
+
21
+ Compute provided by our project sponsor Redmond AI, thank you! Follow RedmondAI on Twitter @RedmondAI.
22
+
23
+ ## Model Description
24
+
25
+ Nous-Hermes-Llama2-13b is a state-of-the-art language model fine-tuned on over 300,000 instructions. This model was fine-tuned by Nous Research, with Teknium and Emozilla leading the fine tuning process and dataset curation, Redmond AI sponsoring the compute, and several other contributors.
26
+
27
+ This Hermes model uses the exact same dataset as Hermes on Llama-1. This is to ensure consistency between the old Hermes and new, for anyone who wanted to keep Hermes as similar to the old one, just more capable.
28
+
29
+ This model stands out for its long responses, lower hallucination rate, and absence of OpenAI censorship mechanisms. The fine-tuning process was performed with a 4096 sequence length on an 8x a100 80GB DGX machine.
30
+
31
+ ## Example Outputs:
32
+ ![Example4](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b/resolve/main/example5.png "Example 4")
33
+ ![Example1](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b/resolve/main/Example1.png "Example 1")
34
+ ![Example2](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b/resolve/main/example2.png "Example 2")
35
+ ![Example3](https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b/resolve/main/example3.png "Example 3")
36
+
37
+ ## Model Training
38
+
39
+ The model was trained almost entirely on synthetic GPT-4 outputs. Curating high quality GPT-4 datasets enables incredibly high quality in knowledge, task completion, and style.
40
+
41
+ This includes data from diverse sources such as GPTeacher, the general, roleplay v1&2, code instruct datasets, Nous Instruct & PDACTL (unpublished), and several others, detailed further below
42
+
43
+ ## Collaborators
44
+ The model fine-tuning and the datasets were a collaboration of efforts and resources between Teknium, Karan4D, Emozilla, Huemin Art, and Redmond AI.
45
+
46
+ Special mention goes to @winglian for assisting in some of the training issues.
47
+
48
+ Huge shoutout and acknowledgement is deserved for all the dataset creators who generously share their datasets openly.
49
+
50
+ Among the contributors of datasets:
51
+ - GPTeacher was made available by Teknium
52
+ - Wizard LM by nlpxucan
53
+ - Nous Research Instruct Dataset was provided by Karan4D and HueminArt.
54
+ - GPT4-LLM and Unnatural Instructions were provided by Microsoft
55
+ - Airoboros dataset by jondurbin
56
+ - Camel-AI's domain expert datasets are from Camel-AI
57
+ - CodeAlpaca dataset by Sahil 2801.
58
+
59
+ If anyone was left out, please open a thread in the community tab.
60
+
61
+ ## Prompt Format
62
+
63
+ The model follows the Alpaca prompt format:
64
+ ```
65
+ ### Instruction:
66
+ <prompt>
67
+
68
+ ### Response:
69
+ <leave a newline blank for model to respond>
70
+
71
+ ```
72
+
73
+ or
74
+
75
+ ```
76
+ ### Instruction:
77
+ <prompt>
78
+
79
+ ### Input:
80
+ <additional context>
81
+
82
+ ### Response:
83
+ <leave a newline blank for model to respond>
84
+
85
+ ```
86
+
87
+ ## Benchmark Results
88
+ AGI-Eval
89
+ ```
90
+ | Task |Version| Metric |Value | |Stderr|
91
+ |agieval_aqua_rat | 0|acc |0.2362|± |0.0267|
92
+ | | |acc_norm|0.2480|± |0.0272|
93
+ |agieval_logiqa_en | 0|acc |0.3425|± |0.0186|
94
+ | | |acc_norm|0.3472|± |0.0187|
95
+ |agieval_lsat_ar | 0|acc |0.2522|± |0.0287|
96
+ | | |acc_norm|0.2087|± |0.0269|
97
+ |agieval_lsat_lr | 0|acc |0.3510|± |0.0212|
98
+ | | |acc_norm|0.3627|± |0.0213|
99
+ |agieval_lsat_rc | 0|acc |0.4647|± |0.0305|
100
+ | | |acc_norm|0.4424|± |0.0303|
101
+ |agieval_sat_en | 0|acc |0.6602|± |0.0331|
102
+ | | |acc_norm|0.6165|± |0.0340|
103
+ |agieval_sat_en_without_passage| 0|acc |0.4320|± |0.0346|
104
+ | | |acc_norm|0.4272|± |0.0345|
105
+ |agieval_sat_math | 0|acc |0.2909|± |0.0307|
106
+ | | |acc_norm|0.2727|± |0.0301|
107
+ ```
108
+ GPT-4All Benchmark Set
109
+ ```
110
+ | Task |Version| Metric |Value | |Stderr|
111
+ |arc_challenge| 0|acc |0.5102|± |0.0146|
112
+ | | |acc_norm|0.5213|± |0.0146|
113
+ |arc_easy | 0|acc |0.7959|± |0.0083|
114
+ | | |acc_norm|0.7567|± |0.0088|
115
+ |boolq | 1|acc |0.8394|± |0.0064|
116
+ |hellaswag | 0|acc |0.6164|± |0.0049|
117
+ | | |acc_norm|0.8009|± |0.0040|
118
+ |openbookqa | 0|acc |0.3580|± |0.0215|
119
+ | | |acc_norm|0.4620|± |0.0223|
120
+ |piqa | 0|acc |0.7992|± |0.0093|
121
+ | | |acc_norm|0.8069|± |0.0092|
122
+ |winogrande | 0|acc |0.7127|± |0.0127|
123
+ ```
124
+ BigBench Reasoning Test
125
+ ```
126
+ | Task |Version| Metric |Value | |Stderr|
127
+
128
+ |bigbench_causal_judgement | 0|multiple_choice_grade|0.5526|± |0.0362|
129
+ |bigbench_date_understanding | 0|multiple_choice_grade|0.7344|± |0.0230|
130
+ |bigbench_disambiguation_qa | 0|multiple_choice_grade|0.2636|± |0.0275|
131
+ |bigbench_geometric_shapes | 0|multiple_choice_grade|0.0195|± |0.0073|
132
+ | | |exact_str_match |0.0000|± |0.0000|
133
+ |bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|0.2760|± |0.0200|
134
+ |bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|0.2100|± |0.0154|
135
+ |bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|0.4400|± |0.0287|
136
+ |bigbench_movie_recommendation | 0|multiple_choice_grade|0.2440|± |0.0192|
137
+ |bigbench_navigate | 0|multiple_choice_grade|0.4950|± |0.0158|
138
+ |bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|0.5570|± |0.0111|
139
+ |bigbench_ruin_names | 0|multiple_choice_grade|0.3728|± |0.0229|
140
+ |bigbench_salient_translation_error_detection | 0|multiple_choice_grade|0.1854|± |0.0123|
141
+ |bigbench_snarks | 0|multiple_choice_grade|0.6298|± |0.0360|
142
+ |bigbench_sports_understanding | 0|multiple_choice_grade|0.6156|± |0.0155|
143
+ |bigbench_temporal_sequences | 0|multiple_choice_grade|0.3140|± |0.0147|
144
+ |bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|0.2032|± |0.0114|
145
+ |bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|0.1406|± |0.0083|
146
+ |bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|0.4400|± |0.0287|
147
+ ```
148
+
149
+ These are the highest benchmarks Hermes has seen on every metric, achieving the following average scores:
150
+ - GPT4All benchmark average is now 70.0 - from 68.8 in Hermes-Llama1
151
+ - 0.3657 on BigBench, up from 0.328 on hermes-llama1
152
+ - 0.372 on AGIEval, up from 0.354 on Hermes-llama1
153
+
154
+ These benchmarks currently have us at #1 on ARC-c, ARC-e, Hellaswag, and OpenBookQA, and 2nd place on Winogrande, comparing to GPT4all's benchmarking list, supplanting Hermes 1 for the new top position.
155
+
156
+ ## Resources for Applied Use Cases:
157
+ Check out LM Studio for a nice chatgpt style interface here: https://lmstudio.ai/
158
+ For an example of a back and forth chatbot using huggingface transformers and discord, check out: https://github.com/teknium1/alpaca-discord
159
+ For an example of a roleplaying discord chatbot, check out this: https://github.com/teknium1/alpaca-roleplay-discordbot
160
+
161
+ ## Future Plans
162
+ We plan to continue to iterate on both more high quality data, and new data filtering techniques to eliminate lower quality data going forward.
163
+
164
+ ## Model Usage
165
+ The model is available for download on Hugging Face. It is suitable for a wide range of language tasks, from generating creative text to understanding and following complex instructions.
166
+
167
+ [<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)