|
--- |
|
license: cc-by-nc-4.0 |
|
extra_gated_prompt: >- |
|
You agree not to use the model for healthcare decision-making or commercial |
|
use |
|
extra_gated_fields: |
|
I agree to use this model for non-commercial use ONLY: checkbox |
|
I agree not to use this model for healthcare decision-making: checkbox |
|
language: |
|
- en |
|
--- |
|
|
|
# Clinical Camel |
|
|
|
## Model Description |
|
|
|
Clinical Camel is an open large language model (LLM), fine-tuned on the LLaMA-2 70B architecture using [QLoRA](https://github.com/artidoro/qlora). It is tailored for the medical and clinical research, capable of processing and generating relevant content. |
|
|
|
Review our pre-print for more details: [Clinical Camel - Pre-print](https://arxiv.org/abs/2305.12031) |
|
|
|
## Performance |
|
|
|
Clinical Camel demonstrates competitive performance on medical benchmarks. |
|
|
|
**Table: Five-Shot Performance of Clinical Camel-70B (C70), GPT3.5, GPT4, and Med-PaLM 2 on Various Medical Datasets** |
|
|
|
| Dataset | ClinicalCamel-70B | GPT3.5 | GPT4 | Med-PaLM 2 | |
|
|-----------------------------|-------------|--------|-------|--------------| |
|
| MMLU Anatomy | 65.2 | 60.7 | 80.0 | 77.8 | |
|
| MMLU Clinical Knowledge | 72.8 | 68.7 | 86.4 | 88.3 | |
|
| MMLU College Biology | 81.2 | 72.9 | 93.8 | 94.4 | |
|
| MMLU College Medicine | 68.2 | 63.6 | 76.3 | 80.9 | |
|
| MMLU Medical Genetics | 69.0 | 68.0 | 92.0 | 90.0 | |
|
| MMLU Professional Medicine | 75.0 | 69.8 | 93.8 | 95.2 | |
|
| MedMCQA | 54.2 | 51.0 | 72.4 | 71.3 | |
|
| MedQA (USMLE) | 60.7 | 53.6 | 81.4 | 79.7 | |
|
| PubMedQA | 77.9 | 60.2 | 74.4 | 79.2 | |
|
| USMLE Sample Exam | 64.3 | 58.5 | 86.6 | - | |
|
|
|
## Evaluation Datasets: |
|
|
|
The performance of Clinical Camel was benchmarked across several datasets, including: |
|
- [USMLE Step 1](https://huggingface.co/datasets/augtoma/usmle_step_1) |
|
- [USMLE Step 2](https://huggingface.co/datasets/augtoma/usmle_step_2) |
|
- [USMLE Step 3](https://huggingface.co/datasets/augtoma/usmle_step_3) |
|
- [MedMCQA](https://huggingface.co/datasets/augtoma/medmcqa) |
|
- [MedQA USMLE](https://huggingface.co/datasets/augtoma/medqa_usmle) |
|
|
|
## Evaluation Reproduction: |
|
|
|
To reproduce the evaluations with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) see the 'TaskFiles' folder |
