---
library_name: peft
base_model: mistralai/Mistral-7B-Instruct-v0.1
---

# Model Card for Mistral-7B-Instruct-v0.1-QLoRa-medical-QA

![image/gif](https://cdn-uploads.huggingface.co/production/uploads/6489e1e3eb763749c663f40c/PUBFPpFxsrWRlkYzh7lwX.gif)

<font color="FF0000" size="5"> <b>
This is a QA model for answering medical questions<br /> </b></font>
<br><b>Foundation Model : https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 <br />
Dataset :  https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max <br /></b>
The model has been fine tuned with 2 x GPU T4 (RAM : 2 x 14.8GB) + CPU (RAM : 29GB). <br />


## <b>Model Details</b>

The model is based upon the foundation model : Mistral-7B-Instruct-v0.1.<br />
It has been tuned with Supervised Fine-tuning Trainer and PEFT LoRa.<br />

### Librairies
<ul>
<li>bitsandbytes</li>
<li>einops</li>
<li>peft</li>
<li>trl</li>
<li>datasets</li>
<li>transformers</li>
<li>torch</li>
</ul>


## <b>Bias, Risks, and Limitations</b>

In order to reduce training duration, the model has been trained only with the first 5100 rows of the dataset.<br />

<font color="FF0000">
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.<br />
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.<br />
</font>

## <b>Training Details</b>

<ul>
<li>per_device_train_batch_size = 1</li>
<li>gradient_accumulation_steps = 16</li>
<li>epoch = 5</li>
<li>2 x GPU T4 (RAM : 14.8GB) + CPU (RAM : 29GB)</li>
</ul>

### Notebook used for the training

You can find it in the files and versions tab

### Training Data

https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max

#### Training Hyperparameters


![image/png](https://cdn-uploads.huggingface.co/production/uploads/6489e1e3eb763749c663f40c/C6XTGVrn4D1Sj2kc9Dq2O.png)

#### Times

Training duration : 6287.4s