---
base_model: microsoft/phi-2
library_name: peft
license: apache-2.0
datasets:
- neil-code/dialogsum-test
language:
- en
metrics:
- bleu
pipeline_tag: question-answering
tags:
- QuestionAnswering
- legal
- finan
- chem
- biology
---
license: apache-2.0
language:
- en
metrics:
- rouge
base_model:
- microsoft/phi-2
pipeline_tag: question-answering
---



This repo containes the last checkpoint of my fine tuned model. Click this link to go the final model
https://huggingface.co/JamieAi33/Phi-2_PEFT
# Model Card for PEFT-Fine-Tuned Model

This model card documents a PEFT-fine-tuned version of `microsoft/phi-2` for question-answering tasks. The PEFT fine-tuning improved the model's performance, as detailed in the evaluation section.

## Model Details

### Model Description

- **Developed by:** JamieAi33
- **Finetuned from model:** `microsoft/phi-2`
- **Model type:** PEFT fine-tuned transformer
- **Language(s) (NLP):** English
- **License:** Apache 2.0

The base model `microsoft/phi-2` was adapted using Parameter-Efficient Fine-Tuning (PEFT) for question-answering tasks. The training process focused on improving performance metrics while keeping computational costs low.

---

### Model Sources

- **Repository:** https://huggingface.co/JamieAi33/Phi-2-QLora
- **Paper:** [Optional: Add a reference to PEFT or any relevant paper]
- **Demo:** [Optional: Link to your Hugging Face Space or demo]

---

## Uses

### Direct Use
This model can be used out-of-the-box for question-answering tasks.

### Downstream Use
The model can be fine-tuned further on domain-specific datasets for improved performance.

### Out-of-Scope Use
Avoid using this model for tasks outside question-answering or where fairness, bias, and ethical considerations are critical without further validation.

---

## Bias, Risks, and Limitations

Users should be aware that:
- The model is trained on publicly available data and may inherit biases present in the training data.
- It is optimized for English and may perform poorly in other languages.

---

## How to Get Started with the Model

Here’s an example of loading the model:

```python
from transformers import AutoModel
from peft import PeftModel

base_model = AutoModel.from_pretrained("microsoft/phi-2")
adapter_model = PeftModel.from_pretrained(base_model, "JamieAi33/Phi-2-QLora")



# Model Name: PEFT Fine-Tuned `microsoft/phi-2`

This repository contains a PEFT fine-tuned version of the `microsoft/phi-2` model for question-answering tasks. The fine-tuning process leveraged Parameter-Efficient Fine-Tuning (PEFT) techniques to achieve improved performance.

---

## Metrics

The model's performance was evaluated using the ROUGE metric. Below are the results:

| **Metric**     | **Original Model** | **PEFT Model** | **Absolute Improvement** |
|-----------------|--------------------|----------------|---------------------------|
| **ROUGE-1**    | 29.76%             | 44.51%         | +14.75%                  |
| **ROUGE-2**    | 10.76%             | 15.68%         | +4.92%                   |
| **ROUGE-L**    | 21.69%             | 30.95%         | +9.25%                   |
| **ROUGE-Lsum** | 22.75%             | 31.49%         | +8.74%                   |

---
## Training Configuration

| Hyperparameter        | Value                   |
|-----------------------|-------------------------|
| **Batch Size**        | 1                       |
| **Learning Rate**     | 2e-4                   |
| **Max Steps**         | 1000                   |
| **Optimizer**         | Paged AdamW (8-bit)    |
| **Logging Steps**     | 25                     |
| **Evaluation Steps**  | 25                     |
| **Gradient Checkpointing** | Enabled          |