|
--- |
|
license: cc-by-nc-nd-4.0 |
|
tags: |
|
- moe |
|
- merge |
|
- medical |
|
- mergekit |
|
- sethuiyer/Dr_Samantha_7b_mistral |
|
- fblgit/UNA-TheBeagle-7b-v1 |
|
language: |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# MedleyMD |
|
|
|
MedleyMD is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): |
|
* [sethuiyer/Dr_Samantha_7b_mistral](https://huggingface.co/sethuiyer/Dr_Samantha_7b_mistral) |
|
* [fblgit/UNA-TheBeagle-7b-v1](https://huggingface.co/fblgit/UNA-TheBeagle-7b-v1) |
|
|
|
## 🧩 Configuration |
|
|
|
```yaml |
|
base_model: OpenPipe/mistral-ft-optimized-1227 |
|
gate_mode: hidden |
|
dtype: bfloat16 |
|
|
|
experts: |
|
- source_model: sethuiyer/Dr_Samantha_7b_mistral |
|
positive_prompts: ["differential diagnosis", "Clinical Knowledge", "Medical Genetics", "Human Aging", "Human Sexuality", "College Medicine", "Anatomy", "College Biology", "High School Biology", "Professional Medicine", "Nutrition", "High School Psychology", "Professional Psychology", "Virology"] |
|
|
|
- source_model: fblgit/UNA-TheBeagle-7b-v1 |
|
positive_prompts: ["How do you", "Explain the concept of", "Give an overview of", "Compare and contrast between", "Provide information about", "Help me understand", "Summarize", "Make a recommendation on", "chat", "math", "reason", "mathematics", "solve", "count", "python", "javascript", "programming", "algorithm", "tell me", "assistant"] |
|
|
|
``` |
|
|
|
## 💻 Usage |
|
|
|
```python |
|
!pip install -qU transformers bitsandbytes accelerate |
|
|
|
from transformers import AutoTokenizer |
|
import transformers |
|
import torch |
|
|
|
model = "sethuiyer/MedleyMD" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model) |
|
pipeline = transformers.pipeline( |
|
"text-generation", |
|
model=model, |
|
model_kwargs={"torch_dtype": torch.bfloat16, "load_in_4bit": True}, |
|
) |
|
|
|
generation_kwargs = { |
|
"max_new_tokens": 512, |
|
"do_sample": True, |
|
"temperature": 0.7, |
|
"top_k": 50, |
|
"top_p": 95, |
|
} |
|
|
|
messages = [{"role":"system", "content":"You are an helpful AI assistant. Please use </s> when you want to end the answer."}, |
|
{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}] |
|
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
|
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) |
|
print(outputs[0]["generated_text"]) |
|
``` |
|
|
|
```text |
|
A Mixture of Experts (Mixout) is a neural network architecture that combines the strengths of multiple expert networks to make a more accurate and robust prediction. |
|
It is composed of a topmost gating network that assigns weights to each expert network based on their performance on past input samples. |
|
The expert networks are trained independently, and the gating network learns to choose the best combination of these experts to make the final prediction. |
|
Mixout demonstrates a stronger ability to handle complex data distributions and is more efficient in terms of training time and memory usage compared to a |
|
traditional ensemble approach. |
|
``` |
