Update README.md

README.md

@@ -14,6 +14,20 @@ license_name: llama3
 # **Med42-v2 - Clinical Large Language Models**
 Med42-v2 is a suite of open-access clinical large language models (LLM) instruct and preference-tuned by M42 to expand access to medical knowledge. Built off LLaMA-3 and comprising either 8 or 70 billion parameters, these generative AI system provide high-quality answers to medical questions.
 
+## Key performance metrics:
+
+- Med42-v2 outperforms GPT-4.0 in all clinically relevant tasks.
+- Med42-v2 achieves a MedQA zero-shot performance of 80.68, surpassing the prior state-of-the-art among all openly available medical LLMs.
+- Med42-v2 attains an 84.61% score on the USMLE (self-assessment and sample exam combined), marking the highest score achieved so far.
+
+## Limitations & Safe Use
+
+- Med42-v2 suite of models are not ready for real clinical use. Extensive human evaluation is undergoing as it is required to ensure safety.
+- Potential for generating incorrect or harmful information.
+- Risk of perpetuating biases in training data.
+
+Use this suite of models responsibly! Do not rely on them for medical usage without rigorous safety testing.
+
 ## Model Details
 
 *Disclaimer: This large language model is not yet ready for clinical use without further testing and validation. It should not be relied upon for making medical decisions or providing patient care.*
@@ -103,12 +117,14 @@ The training was conducted on the NVIDIA DGX cluster with H100 GPUs, utilizing P
 
 ## Evaluation Results
 
+### MCQA Evaluation
+
 Med42-v2 improves performance on every clinical benchmark compared to our previous version, including MedQA, MedMCQA, USMLE, MMLU clinical topics and MMLU Pro clinical subset. For all evaluations reported so far, we use [EleutherAI's evaluation harness library](https://github.com/EleutherAI/lm-evaluation-harness) and report zero-shot accuracies (except otherwise stated). We integrated chat templates into harness and computed the likelihood for the full answer instead of only the tokens "a.", "b.", "c." or "d.".
 
 |Model|MMLU Pro|MMLU|MedMCQA|MedQA|USMLE|
 |---:|:---:|:---:|:---:|:---:|:---:|
-|Med42v2-70B|64.97|88.16|73.82|80.68|84.61|
-|Med42v2-8B|55.15|77.11|61.82|63.71|68.93|
+|Med42v2-70B|64.36|87.12|73.20|79.10|83.80|
+|Med42v2-8B|54.30|75.76|61.34|62.84|67.04|
 |OpenBioLLM|64.24|90.40|73.18|76.90|79.01|
 |GPT-4.0<sup>&dagger;</sup>|-|87.00|69.50|78.90|84.05|
 |MedGemini*|-|-|-|84.00|-|
@@ -121,18 +137,25 @@ Med42-v2 improves performance on every clinical benchmark compared to our previo
 
 <sup>&dagger;</sup> *Results as reported in the paper [Capabilities of GPT-4 on Medical Challenge Problems](https://www.microsoft.com/en-us/research/uploads/prod/2023/03/GPT-4_medical_benchmarks.pdf)*.
 
-### Key performance metrics:
+### Open-ended question generation
 
-- Med42-v2 outperforms GPT-4.0 in all clinically relevant tasks.
-- Med42-v2 achieves a MedQA zero-shot performance of 80.68, surpassing the prior state-of-the-art among all openly available medical LLMs.
-- Med42-v2 attains an 84.61% score on the USMLE (self-assessment and sample exam combined), marking the highest score achieved so far.
+To ensure a robust evaluation of our model's output quality, we employ the LLM-as-a-Judge approach using Prometheus-8x7b-v2.0. Our assessment uses 4,000 publicly accessible healthcare-related questions, generating responses from various models. We then use Prometheus to conduct pairwise comparisons of the answers. Drawing inspiration from the LMSYS Chatbot-Arena methodology, we present the results as Elo ratings for each model.
+
+To maintain fairness and eliminate potential bias from prompt engineering, we used the same simple system prompt for every model throughout the evaluation process.
+
+Below is the scoring rubric we used to prompt Prometheus to select the best answer:
+
+```
+### Score Rubric:
+Which response is of higher overall quality in a medical context? Consider:
+* Relevance: Does it directly address the question?
+* Completeness: Does it cover all important aspects, details and subpoints?
+* Safety: Does it avoid unsafe practices and address potential risks?
+* Ethics: Does it maintain confidentiality and avoid biases?
+* Clarity: Is it professional, clear and easy to understand?
+```
 
-## Limitations & Safe Use
-- Med42-v2 suite of models are not ready for real clinical use. Extensive human evaluation is undergoing as it is required to ensure safety.
-- Potential for generating incorrect or harmful information.
-- Risk of perpetuating biases in training data.
 
-Use these suite of models responsibly! Do not rely on them for medical usage without rigorous safety testing.
 
 ## Accessing Med42 and Reporting Issues