|
--- |
|
base_model: unsloth/deepseek-r1-distill-llama-8b-unsloth-bnb-4bit |
|
library_name: peft |
|
license: mit |
|
datasets: |
|
- FreedomIntelligence/medical-o1-reasoning-SFT |
|
language: |
|
- en |
|
tags: |
|
- medical |
|
--- |
|
|
|
# Model Card for DeepSeek-R1-Medical-COT |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
DeepSeek-R1-Medical-COT is a fine-tuned version of the DeepSeek-R1 model, optimized for medical chain-of-thought (COT) reasoning. It is designed to assist in medical-related tasks such as question-answering, reasoning, and decision support. This model is particularly useful for applications requiring structured reasoning in the medical domain. |
|
|
|
- **Developed by:** Mohamed Mahmoud |
|
- **Funded by [optional]:** Independent project |
|
- **Shared by:** Mohamed Mahmoud |
|
- **Model type:** Transformer-based Large Language Model (LLM) |
|
- **Language(s) (NLP):** English (en) |
|
- **License:** MIT |
|
- **Finetuned from model:** unsloth/deepseek-r1-distill-llama-8b-unsloth-bnb-4bit |
|
|
|
### Model Sources |
|
|
|
- **Repository:** [Hugging Face Model Repo](https://huggingface.co/thesnak/DeepSeek-R1-Medical-COT) |
|
|
|
- **LinkedIn:** [Mohamed Mahmoud](https://www.linkedin.com/in/mohamed-thesnak) |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
The model can be used directly for medical reasoning tasks, including: |
|
|
|
- Answering medical questions |
|
- Assisting in medical decision-making |
|
- Supporting clinical research and literature review |
|
|
|
### Downstream Use |
|
|
|
- Fine-tuning for specialized medical applications |
|
- Integration into chatbots and virtual assistants for medical advice |
|
- Educational tools for medical students |
|
|
|
### Out-of-Scope Use |
|
|
|
- This model is not a replacement for professional medical advice. |
|
- Should not be used for clinical decision-making without expert validation. |
|
- May not perform well in languages other than English. |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
While fine-tuned for medical reasoning, the model may still have biases due to the limitations of its training data. Users should exercise caution and validate critical outputs with medical professionals. |
|
|
|
### Recommendations |
|
|
|
Users should verify outputs, particularly in sensitive medical contexts. The model is best used as an assistive tool rather than a primary decision-making system. |
|
|
|
## How to Get Started with the Model |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
import torch |
|
|
|
model_name = "thesnak/DeepSeek-R1-Medical-COT" |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto") |
|
|
|
input_text = "What are the symptoms of pneumonia?" |
|
inputs = tokenizer(input_text, return_tensors="pt").to("cuda") |
|
outputs = model.generate(**inputs, max_new_tokens=100) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
``` |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
The model was fine-tuned using the **FreedomIntelligence/medical-o1-reasoning-SFT** dataset, which contains medical question-answer pairs designed to improve reasoning capabilities. |
|
|
|
### Training Procedure |
|
|
|
#### Preprocessing |
|
|
|
- Tokenization using LLaMA tokenizer |
|
- Text cleaning and normalization |
|
|
|
#### Training Hyperparameters |
|
|
|
- **Precision:** bf16 mixed precision |
|
- **Optimizer:** AdamW |
|
- **Batch size:** 16 |
|
- **Learning rate:** 2e-5 |
|
- **Epochs:** 3 |
|
|
|
#### Speeds, Sizes, Times |
|
|
|
- **Training time:** Approximately 12 hours on a P100 GPU (Kaggle) |
|
- **Model size:** 8B parameters (bnb 4-bit quantized) |
|
|
|
#### Training Loss |
|
|
|
| Step | Training Loss | |
|
| ---- | ------------- | |
|
| 10 | 1.919000 | |
|
| 20 | 1.461800 | |
|
| 30 | 1.402500 | |
|
| 40 | 1.309000 | |
|
| 50 | 1.344400 | |
|
| 60 | 1.314100 | |
|
|
|
## Evaluation |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
- The model was evaluated on held-out samples from **FreedomIntelligence/medical-o1-reasoning-SFT**. |
|
|
|
#### Factors |
|
|
|
- Performance was assessed on medical reasoning tasks. |
|
|
|
#### Metrics |
|
|
|
- **Perplexity:** Measured for general coherence. |
|
- **Accuracy:** Evaluated based on expert-verified responses. |
|
- **BLEU Score:** Used to assess response relevance. |
|
|
|
### Results |
|
|
|
- **Perplexity:** |
|
- **Accuracy:** |
|
- **BLEU Score:** |
|
|
|
## Model Examination |
|
|
|
Further interpretability analyses can be conducted using tools like Captum and SHAP to analyze how the model derives its medical reasoning responses. |
|
|
|
## Environmental Impact |
|
|
|
- **Hardware Type:** P100 GPU (Kaggle) |
|
- **Hours used:** 2 hours |
|
- **Cloud Provider:** Kaggle |
|
- **Compute Region:** N/A |
|
- **Carbon Emitted:** Estimated at 9.5 kg CO2eq |
|
- **[Kaggle Notebook](https://www.kaggle.com/code/thesnak/fine-tune-deepseek)** |
|
## Technical Specifications |
|
|
|
### Compute Infrastructure |
|
|
|
#### Hardware |
|
|
|
- P100 GPU (16GB VRAM) on Kaggle |
|
|
|
|
|
## Citation |
|
|
|
**BibTeX:** |
|
|
|
```bibtex |
|
@misc{mahmoud2025deepseekmedcot, |
|
title={DeepSeek-R1-Medical-COT}, |
|
author={Mohamed Mahmoud}, |
|
year={2025}, |
|
url={https://huggingface.co/thesnak/DeepSeek-R1-Medical-COT} |
|
} |
|
``` |
|
|
|
## Model Card Authors |
|
|
|
- Mohamed Mahmoud |
|
|
|
## Model Card Contact |
|
|
|
- [LinkedIn](https://www.linkedin.com/in/mohamed-thesnak) |
