File size: 7,697 Bytes
57bfa7b
8d65fef
 
22db51c
8d65fef
57bfa7b
 
 
22db51c
57bfa7b
acb9227
 
 
 
8d65fef
 
 
22db51c
8d65fef
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57bfa7b
 
 
 
b6f25aa
 
 
57bfa7b
 
 
 
23ea87b
37c251f
27219f5
 
e2919a6
27219f5
 
 
 
57bfa7b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37c251f
 
 
 
 
 
 
27219f5
23ea87b
 
37c251f
 
cec8363
 
 
 
 
 
 
 
 
 
 
57bfa7b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22db51c
57bfa7b
 
22db51c
 
 
 
 
 
 
 
 
 
57bfa7b
37c251f
57bfa7b
22db51c
 
 
 
 
 
 
 
8d65fef
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
---
language:
- en
license: cc-by-nc-nd-4.0
library_name: transformers
tags:
- moe
- merge
- medical
- mergekit
datasets:
- medmcqa
- cognitivecomputations/samantha-data
- jondurbin/bagel-v0.3
base_model:
- sethuiyer/Dr_Samantha_7b_mistral
- fblgit/UNA-TheBeagle-7b-v1
pipeline_tag: text-generation
model-index:
- name: MedleyMD
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 66.47
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 86.06
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 65.1
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 52.46
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 80.27
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 68.99
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/MedleyMD
      name: Open LLM Leaderboard
---

# MedleyMD

![logo](https://huggingface.co/sethuiyer/MedleyMD/resolve/main/logo.webp)


MedleyMD is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [sethuiyer/Dr_Samantha_7b_mistral](https://huggingface.co/sethuiyer/Dr_Samantha_7b_mistral)
* [fblgit/UNA-TheBeagle-7b-v1](https://huggingface.co/fblgit/UNA-TheBeagle-7b-v1)

These models were chosen because `fblgit/UNA-TheBeagle-7b-v1` has excellent performance for a 7B parameter model and Dr.Samantha has capabilities of a medical knowledge-focused model (trained on USMLE databases and doctor-patient interactions) with the philosophical, psychological, and relational understanding, scoring 68.82% in topics related to clinical domain and psychology. 

## Benchmark

On a synthetic benchmark of 35 medical diagnosis questions generated by GPT-4, GPT-4 also being an evaluator, MedleyMD scored **96.25/100**.

Nous Benchmark numbers shall be available soon.


## 🧩 Configuration

```yaml
base_model: OpenPipe/mistral-ft-optimized-1227
gate_mode: hidden
dtype: bfloat16

experts:
  - source_model: sethuiyer/Dr_Samantha_7b_mistral
    positive_prompts: ["differential diagnosis", "Clinical Knowledge", "Medical Genetics", "Human Aging", "Human Sexuality", "College Medicine", "Anatomy", "College Biology", "High School Biology", "Professional Medicine", "Nutrition", "High School Psychology", "Professional Psychology", "Virology"]

  - source_model: fblgit/UNA-TheBeagle-7b-v1
    positive_prompts: ["How do you", "Explain the concept of", "Give an overview of", "Compare and contrast between", "Provide information about", "Help me understand", "Summarize", "Make a recommendation on", "chat", "math", "reason", "mathematics", "solve", "count", "python", "javascript", "programming", "algorithm", "tell me", "assistant"]

```

## GGUF
1. [medleymd.Q4_K_M](https://huggingface.co/sethuiyer/MedleyMD-GGUF/resolve/main/medleymd.Q4_K_M.gguf) [7.2GB]
2. [medleymd.Q5_K_M](https://huggingface.co/sethuiyer/MedleyMD-GGUF/resolve/main/medleymd.Q5_K_M.gguf) [9.13GB]


## Ollama

MedleyMD can be used in ollama by running```ollama run stuehieyr/medleymd``` in your terminal. 

If you have limited computing resources, check out this [video](https://www.youtube.com/watch?v=Qa1h7ygwQq8) to learn how to run it on 
a Google Colab backend.

## Prompt format:
This model uses ChatML prompt format.
```
<|im_start|>system
You are Medley, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

```

## 💻 Usage

```python
!pip install -qU transformers bitsandbytes accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "sethuiyer/MedleyMD"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    model_kwargs={"torch_dtype": torch.bfloat16, "load_in_4bit": True},
)

generation_kwargs = {
    "max_new_tokens": 512,
    "do_sample": True,
    "temperature": 0.7,
    "top_k": 50,
    "top_p": 95,
}

messages = [{"role":"system", "content":"You are an helpful AI assistant. Please use </s> when you want to end the answer."},
{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, **generation_kwargs)
print(outputs[0]["generated_text"])
```

```text
A Mixture of Experts (Mixout) is a neural network architecture that combines the strengths of multiple expert networks to make a more accurate and robust prediction.
It is composed of a topmost gating network that assigns weights to each expert network based on their performance on past input samples.
The expert networks are trained independently, and the gating network learns to choose the best combination of these experts to make the final prediction.
Mixout demonstrates a stronger ability to handle complex data distributions and is more efficient in terms of training time and memory usage compared to a
traditional ensemble approach.
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__MedleyMD)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |69.89|
|AI2 Reasoning Challenge (25-Shot)|66.47|
|HellaSwag (10-Shot)              |86.06|
|MMLU (5-Shot)                    |65.10|
|TruthfulQA (0-shot)              |52.46|
|Winogrande (5-shot)              |80.27|
|GSM8k (5-shot)                   |68.99|