End of training

README.md

@@ -44,42 +44,42 @@ More information needed
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 2473901162496.0 | 170424302305280.0 | 22.0554 | 26.3026 | 95.048 | 11.9 | 4060086272.0 | 71468255805440.0 |
-| 2500 | 0.0404 | 744.0 | 5792.0 | 2.6067 | 26.3957 | 94.712 | 11.858 | 450.0 | 4160.0 |
-| 5000 | 0.0808 | 316.0 | 1408.0 | 1.9089 | 26.4305 | 94.588 | 11.842 | 236.0 | 300.0 |
-| 7500 | 0.1212 | 221.0 | 776.0 | 1.6174 | 26.4065 | 94.674 | 11.853 | 180.0 | 176.0 |
-| 10000 | 0.1616 | 165.0 | 620.0 | 1.4274 | 26.4063 | 94.674 | 11.853 | 144.0 | 149.0 |
-| 12500 | 0.2020 | 124.5 | 484.0 | 1.1896 | 26.4269 | 94.6 | 11.844 | 107.5 | 145.0 |
-| 15000 | 0.2424 | 108.0 | 452.0 | 1.0631 | 26.4405 | 94.552 | 11.838 | 92.5 | 110.0 |
-| 17500 | 0.2828 | 91.5 | 354.0 | 0.9650 | 26.417 | 94.636 | 11.848 | 75.5 | 119.0 |
-| 20000 | 0.3232 | 80.5 | 306.0 | 0.8889 | 26.4341 | 94.575 | 11.841 | 70.0 | 225.0 |
-| 22500 | 0.3636 | 73.0 | 233.0 | 0.7611 | 26.3815 | 94.763 | 11.864 | 58.0 | 83.5 |
-| 25000 | 0.4040 | 67.0 | 201.0 | 0.7173 | 26.4268 | 94.601 | 11.844 | 52.25 | 95.5 |
-| 27500 | 0.4444 | 64.0 | 212.0 | 0.6823 | 26.3976 | 94.705 | 11.857 | 49.0 | 141.0 |
-| 30000 | 0.4848 | 64.5 | 201.0 | 0.6783 | 26.337 | 94.924 | 11.884 | 51.5 | 86.0 |
-| 32500 | 0.5253 | 64.5 | 190.0 | 0.6657 | 26.3562 | 94.854 | 11.876 | 47.0 | 68.5 |
-| 35000 | 0.5657 | 60.75 | 179.0 | 0.6094 | 26.3895 | 94.735 | 11.861 | 42.5 | 59.25 |
-| 37500 | 0.6061 | 61.0 | 165.0 | 0.5963 | 26.4159 | 94.64 | 11.849 | 44.5 | 74.5 |
-| 40000 | 0.6465 | 58.75 | 171.0 | 0.5823 | 26.4219 | 94.619 | 11.846 | 39.0 | 93.0 |
-| 42500 | 0.6869 | 56.75 | 167.0 | 0.5589 | 26.4356 | 94.57 | 11.84 | 40.5 | 60.25 |
-| 45000 | 0.7273 | 52.75 | 140.0 | 0.4717 | 26.4084 | 94.667 | 11.852 | 34.25 | 47.75 |
-| 47500 | 0.7677 | 53.0 | 132.0 | 0.4510 | 26.4148 | 94.644 | 11.849 | 33.25 | 38.75 |
-| 50000 | 0.8081 | 50.75 | 133.0 | 0.4386 | 26.348 | 94.884 | 11.879 | 32.5 | 40.25 |
-| 52500 | 0.8485 | 50.0 | 129.0 | 0.4270 | 26.4189 | 94.629 | 11.848 | 32.25 | 38.25 |
-| 55000 | 0.8889 | 49.75 | 125.0 | 0.4124 | 26.4696 | 94.448 | 11.825 | 31.125 | 35.5 |
-| 57500 | 0.9293 | 48.75 | 125.5 | 0.4083 | 26.4221 | 94.618 | 11.846 | 30.625 | 33.5 |
-| 60000 | 0.9697 | 48.75 | 124.5 | 0.4048 | 26.3938 | 94.719 | 11.859 | 30.5 | 33.75 |
-| 61875 | 1.0 | 48.75 | 124.5 | 0.4044 | 26.3794 | 94.771 | 11.865 | 30.5 | 33.75 |
+| 0 | 0 | 1159641169920.0 | 74217034874880.0 | 20.1532 | 29.7822 | 83.943 | 10.51 | 3053453312.0 | 65146063945728.0 |
+| 2500 | 0.0404 | 696.0 | 5344.0 | 2.5821 | 29.9253 | 83.541 | 10.459 | 412.0 | 4160.0 |
+| 5000 | 0.0808 | 318.0 | 1336.0 | 1.9054 | 29.8753 | 83.681 | 10.477 | 256.0 | 217.0 |
+| 7500 | 0.1212 | 211.0 | 856.0 | 1.6063 | 29.8573 | 83.732 | 10.483 | 171.0 | 185.0 |
+| 10000 | 0.1616 | 171.0 | 660.0 | 1.4275 | 29.8856 | 83.652 | 10.473 | 137.0 | 151.0 |
+| 12500 | 0.2020 | 120.0 | 454.0 | 1.1681 | 30.0103 | 83.305 | 10.43 | 97.0 | 133.0 |
+| 15000 | 0.2424 | 101.5 | 396.0 | 1.0398 | 29.9702 | 83.416 | 10.444 | 84.0 | 151.0 |
+| 17500 | 0.2828 | 89.0 | 344.0 | 0.9386 | 29.8083 | 83.869 | 10.5 | 71.0 | 130.0 |
+| 20000 | 0.3232 | 80.0 | 296.0 | 0.8577 | 29.9718 | 83.412 | 10.443 | 68.0 | 111.0 |
+| 22500 | 0.3636 | 67.5 | 220.0 | 0.7534 | 29.9265 | 83.538 | 10.459 | 55.75 | 62.5 |
+| 25000 | 0.4040 | 65.0 | 203.0 | 0.7007 | 29.9277 | 83.535 | 10.459 | 51.25 | 121.0 |
+| 27500 | 0.4444 | 64.0 | 212.0 | 0.6707 | 29.8714 | 83.692 | 10.478 | 46.5 | 117.0 |
+| 30000 | 0.4848 | 65.5 | 212.0 | 0.6720 | 29.8654 | 83.709 | 10.48 | 50.25 | 83.5 |
+| 32500 | 0.5253 | 63.5 | 190.0 | 0.6577 | 29.8752 | 83.681 | 10.477 | 47.25 | 57.75 |
+| 35000 | 0.5657 | 60.25 | 176.0 | 0.6085 | 29.8266 | 83.818 | 10.494 | 40.5 | 75.0 |
+| 37500 | 0.6061 | 61.25 | 185.0 | 0.5872 | 29.9842 | 83.377 | 10.439 | 41.25 | 88.5 |
+| 40000 | 0.6465 | 57.75 | 166.0 | 0.5662 | 30.7366 | 81.336 | 10.183 | 41.0 | 64.5 |
+| 42500 | 0.6869 | 58.25 | 173.0 | 0.5596 | 30.0925 | 83.077 | 10.401 | 40.0 | 53.5 |
+| 45000 | 0.7273 | 52.0 | 144.0 | 0.4697 | 29.9028 | 83.604 | 10.467 | 34.75 | 54.0 |
+| 47500 | 0.7677 | 51.75 | 136.0 | 0.4482 | 29.8743 | 83.684 | 10.477 | 33.75 | 40.0 |
+| 50000 | 0.8081 | 50.25 | 142.0 | 0.4373 | 29.8796 | 83.669 | 10.475 | 32.75 | 37.0 |
+| 52500 | 0.8485 | 50.0 | 133.0 | 0.4278 | 29.9384 | 83.505 | 10.455 | 32.5 | 36.0 |
+| 55000 | 0.8889 | 49.0 | 127.5 | 0.4117 | 29.9783 | 83.394 | 10.441 | 31.0 | 35.5 |
+| 57500 | 0.9293 | 49.25 | 129.0 | 0.4067 | 29.9301 | 83.528 | 10.458 | 31.0 | 34.5 |
+| 60000 | 0.9697 | 48.75 | 127.5 | 0.4037 | 29.9613 | 83.441 | 10.447 | 30.875 | 34.25 |
+| 61875 | 1.0 | 49.0 | 127.5 | 0.4034 | 30.7338 | 81.344 | 10.184 | 30.875 | 34.0 |
 
 # Resource Usage Comparison
 
-- VRAM Use: 7.7851 GB
+- VRAM Use: 7.7830 GB
 
-# Distillation (Teacher -> Student) Architecture Difference:
+`# Distillation (Teacher -> Student) Architecture Difference:
 
 - **Architecture**: `GPT2LMHeadModel` -> `GPT2LMHeadModel`
 - **Total Parameters**: 124,439,808 -> 124,439,808
-- **Data Type (dtype)**: torch.bfloat16 -> torch.bfloat16
+- **Data Type (dtype)**: 124439808 -> torch.bfloat16
 - **Model Size**: 0.24 GB -> 0.24 GB
 
 <details>
@@ -93,7 +93,7 @@ More information needed
 <br/>
 
 # Train Dataset
-Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 145,743,159 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `247,500`
 - Subset: `20231101.en`
@@ -122,7 +122,7 @@ The following hyperparameters were used during training:
 - num_epochs: `1.0`
 - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))`
 - train_embeddings: `True`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc55d09f820>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f01904db0d0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `None`
@@ -154,6 +154,6 @@ The following hyperparameters were used during training:
 
 # Framework Versions
 - Distily 0.2.0
-- Transformers 4.44.1
+- Transformers 4.44.0
 - Pytorch 2.3.0
 - Datasets 2.21.0

config.json

@@ -33,7 +33,7 @@
     }
   },
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.44.1",
+  "transformers_version": "4.44.0",
   "use_cache": true,
   "vocab_size": 50257
 }

generation_config.json

@@ -2,5 +2,5 @@
   "_from_model_config": true,
   "bos_token_id": 50256,
   "eos_token_id": 50256,
-  "transformers_version": "4.44.1"
+  "transformers_version": "4.44.0"
 }

logs/attn_loss_fn=raw_mse, attn_weight=10.0, projector=ensemble/events.out.tfevents.1724300609.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23f92f0f737625b85790bdb547c1212411139d81621289e94527da3f0704d134
+size 29632525

logs/attn_loss_fn=raw_mse, attn_weight=10.0, projector=ensemble/events.out.tfevents.1724310413.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74bbc28fbc9e148c9a2f280b1621a69d6612fb899af526a84ada89fee800d2bd
+size 588

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d4a7dae2baee0e75e88d4e74c05c5897efe9933054784db8db0f5b40ebb7562c
+oid sha256:b08e67403aa270c1ac421f83d13ee0fb0dcd4bf6e493a2d8c1716128dd61a53d
 size 248894656

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:79fb1f9ffd0394d85fe74d01325144f9ff655520b97dd5e040fe13cb6aa2e36a
+oid sha256:e50bda9172f577e1325bdae09c6332012381ef55d6262298c42be31ef40baa0c
 size 1017899144