Update README.md
Browse files
README.md
CHANGED
|
@@ -3,11 +3,40 @@ library_name: transformers
|
|
| 3 |
tags:
|
| 4 |
- trl
|
| 5 |
- sft
|
|
|
|
|
|
|
| 6 |
---
|
| 7 |
|
| 8 |
# Model Card for Model ID
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
|
| 13 |
|
|
|
|
| 3 |
tags:
|
| 4 |
- trl
|
| 5 |
- sft
|
| 6 |
+
base_model:
|
| 7 |
+
- microsoft/Orca-2-7b
|
| 8 |
---
|
| 9 |
|
| 10 |
# Model Card for Model ID
|
| 11 |
|
| 12 |
+
### Description of GlueOrca Model
|
| 13 |
+
|
| 14 |
+
The GlueOrca model is fine-tuned on four distinct tasks from the GLUE benchmark: SST-2 (Sentiment Analysis), MRPC (Paraphrase Detection), CoLA (Linguistic Acceptability), and MNLI (Natural Language Inference). The base model used is "naimul011/GlueOrca," which is designed for various natural language understanding tasks, leveraging a strong pre-training phase followed by fine-tuning to improve task-specific performance.
|
| 15 |
+
|
| 16 |
+
The fine-tuning of GlueOrca focuses on enhancing the model's capabilities in retaining previously learned knowledge while adapting to new tasks, measuring aspects like catastrophic forgetting, learning capability, and training performance across different language tasks.
|
| 17 |
+
|
| 18 |
+
### Benchmark Table for GlueOrca Fine-Tuning Performance
|
| 19 |
+
|
| 20 |
+
| **Model** | **Parameter Size (B)** | **Pretrained Performance** | **Forgetting** | **Learning** | **Training Performance** |
|
| 21 |
+
|-----------------|------------------------|----------------------------|----------------|--------------|--------------------------|
|
| 22 |
+
| Llama-3.2-1B | 1 | 0.50 | 0.24 | 0.33 | 0.54 |
|
| 23 |
+
| Llama-3.2-3B | 3 | 0.56 | 0.225 | 0.36 | 0.61 |
|
| 24 |
+
| Llama-3.1-8B | 8 | 0.56 | 0.59 | 0.84 | 0.67 |
|
| 25 |
+
| Llama-3-8B | 8 | 0.53 | 0.39 | 0.98 | 0.70 |
|
| 26 |
+
| Llama-2-7B | 7 | 0.67 | 0.23 | 0.12 | 0.63 |
|
| 27 |
+
| GPT-J-6B | 6 | 0.50 | 0.39 | 0.45 | 0.54 |
|
| 28 |
+
| Phi-2 | 2.7 | 0.59 | 0.10 | 0.15 | 0.61 |
|
| 29 |
+
| Phi-3.5-mini | 3.82 | 0.69 | **0.02** | 0.30 | 0.76 |
|
| 30 |
+
| Orca-2-7b | 7 | **0.76** | 0.185 | 0.33 | **0.81** |
|
| 31 |
+
| Qwen2.5-0.5B | 0.5 | 0.52 | 0.23 | 0.56 | 0.61 |
|
| 32 |
+
| Qwen2.5-7B | 7 | 0.56 | 0.51 | **1.12** | 0.77 |
|
| 33 |
+
| Qwen2.5-14B | 14 | 0.71 | 0.935 | 0.66 | 0.80 |
|
| 34 |
+
| **GlueOrca** | **7** | **0.61** | **0.35** | **1.05** | **0.75** |
|
| 35 |
+
|
| 36 |
+
### Analysis
|
| 37 |
+
|
| 38 |
+
GlueOrca, fine-tuned on multiple tasks from the GLUE dataset, shows a pre-trained performance of 0.61. The model demonstrates moderate forgetting at 0.35, indicating some loss of prior knowledge. However, it excels in learning capability with a score of 1.05, reflecting its strong ability to adapt to new tasks. The overall training performance is 0.75, making GlueOrca a well-rounded model with balanced retention and learning improvements across various tasks.
|
| 39 |
+
|
| 40 |
|
| 41 |
|
| 42 |
|