AvniMittal13/shwartz2_normal

README.md

@@ -18,7 +18,7 @@ should probably proofread and complete it, then remove this comment. -->
 
 This model is a fine-tuned version of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) on an unknown dataset.
 It achieves the following results on the evaluation set:
-- Loss: 1.8664
+- Loss: 1.4782
 
 ## Model description
 
@@ -38,11 +38,9 @@ More information needed
 
 The following hyperparameters were used during training:
 - learning_rate: 0.0001
-- train_batch_size: 4
+- train_batch_size: 1
 - eval_batch_size: 8
 - seed: 42
-- gradient_accumulation_steps: 4
-- total_train_batch_size: 16
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
 - lr_scheduler_type: cosine
 - lr_scheduler_warmup_ratio: 0.05
@@ -51,18 +49,19 @@ The following hyperparameters were used during training:
 
 ### Training results
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 2.8473        | 0.416  | 13   | 2.9282          |
-| 1.9812        | 0.832  | 26   | 2.1081          |
-| 1.9719        | 1.248  | 39   | 1.9229          |
-| 1.3557        | 1.6640 | 52   | 1.8664          |
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.71          | 0.4   | 200  | 1.4977          |
+| 1.7529        | 0.8   | 400  | 1.4883          |
+| 1.1946        | 1.2   | 600  | 1.4800          |
+| 1.6962        | 1.6   | 800  | 1.4786          |
+| 1.1067        | 2.0   | 1000 | 1.4782          |
 
 
 ### Framework versions
 
 - PEFT 0.13.1.dev0
-- Transformers 4.45.0
-- Pytorch 2.4.1+cu121
+- Transformers 4.44.2
+- Pytorch 2.4.0
 - Datasets 3.0.0
-- Tokenizers 0.20.0
+- Tokenizers 0.19.1

adapter_config.json

@@ -16,13 +16,12 @@
   "megatron_core": "megatron.core",
   "modules_to_save": null,
   "peft_type": "LORA",
-  "r": 16,
+  "r": 4,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "q_proj",
     "k_proj",
-    "o_proj",
+    "q_proj",
     "v_proj"
   ],
   "task_type": "CAUSAL_LM",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6a59f5252ac67a07ba210bb6838e31443b2f8f16745f2caf604ce5edd20e7fed
-size 31483040
+oid sha256:cb5d11f068d9010e806527241c8d42aca2d4ee94afc6478598e9851df4cfc374
+size 7889592

runs/Sep26_03-39-35_d871e25019a7/events.out.tfevents.1727321981.d871e25019a7.30.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e87f9866f4801f995b1346a788858e088b9f090a98bf27fab400dde0c6ece06b
+size 4184

runs/Sep26_03-40-21_d871e25019a7/events.out.tfevents.1727322024.d871e25019a7.30.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60102720cc597e6df9da152c7ad6d327108c65464914d57f5242d75e5227584b
+size 4184

runs/Sep26_03-54-54_307bffa94c34/events.out.tfevents.1727322896.307bffa94c34.30.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3bbffc8e1aacb28942516a7c3372f841c5df4ebff8c1789269cb6722ec4168ac
+size 4184

runs/Sep26_04-13-03_aad8fdad577d/events.out.tfevents.1727323985.aad8fdad577d.30.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3777c5239006c004fc83a1a6c64b033f3ac76028c9b3b4ba43320297ec988c7a
+size 5318

runs/Sep26_04-15-39_aad8fdad577d/events.out.tfevents.1727324143.aad8fdad577d.30.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa3501dfd336fe3851fc2af92d468c3b6b02cde8d49684b1c00b048fb929e62b
+size 4184

runs/Sep26_04-20-40_9b7b186f542d/events.out.tfevents.1727324590.9b7b186f542d.30.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:986b86b3361a3fe2ca3c88f5a8a9438ed0a5657186a9cfceb36bdd8f8066c7c1
+size 217519

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b82ddfea009351f6d27b94c2b01cee4006d8853eaf508b9319fdd2b1635a915
-size 5496
+oid sha256:3ce4ef602c582e23b22f85b2b957ec0b22b2ba87ce4f7cce11f0737541d6572d
+size 5432