End of training

README.md

@@ -44,36 +44,36 @@ More information needed
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 2473901162496.0 | 170424302305280.0 | 30.7740 | 26.7323 | 93.52 | 11.709 | 4060086272.0 | 71468255805440.0 |
-| 2500 | 0.0404 | 1208.0 | 12224.0 | 9.8371 | 26.6698 | 93.739 | 11.736 | 776.0 | 13952.0 |
-| 5000 | 0.0808 | 412.0 | 2240.0 | 8.3982 | 26.694 | 93.654 | 11.725 | 290.0 | 436.0 |
-| 7500 | 0.1212 | 245.0 | 908.0 | 7.6617 | 26.6969 | 93.644 | 11.724 | 218.0 | 198.0 |
-| 10000 | 0.1616 | 183.0 | 672.0 | 7.2408 | 26.7324 | 93.519 | 11.709 | 165.0 | 201.0 |
-| 12500 | 0.2020 | 132.0 | 504.0 | 6.6895 | 26.7254 | 93.544 | 11.712 | 114.5 | 154.0 |
-| 15000 | 0.2424 | 113.0 | 434.0 | 6.4134 | 26.6959 | 93.647 | 11.725 | 90.0 | 137.0 |
-| 17500 | 0.2828 | 93.0 | 340.0 | 6.1940 | 26.7239 | 93.549 | 11.712 | 71.0 | 133.0 |
-| 20000 | 0.3232 | 74.5 | 276.0 | 5.9286 | 26.7267 | 93.54 | 11.711 | 64.5 | 123.0 |
-| 22500 | 0.3636 | 67.0 | 214.0 | 5.6598 | 26.7384 | 93.498 | 11.706 | 50.25 | 75.5 |
-| 25000 | 0.4040 | 64.0 | 194.0 | 5.5584 | 26.6783 | 93.709 | 11.732 | 44.25 | 78.0 |
-| 27500 | 0.4444 | 57.75 | 190.0 | 5.4847 | 26.7024 | 93.625 | 11.722 | 41.25 | 64.5 |
-| 30000 | 0.4848 | 59.75 | 196.0 | 5.4728 | 26.6702 | 93.738 | 11.736 | 42.25 | 69.5 |
-| 32500 | 0.5253 | 58.5 | 174.0 | 5.4507 | 26.712 | 93.591 | 11.718 | 40.5 | 66.0 |
-| 35000 | 0.5657 | 57.0 | 172.0 | 5.3752 | 26.682 | 93.696 | 11.731 | 36.5 | 48.0 |
-| 37500 | 0.6061 | 57.75 | 155.0 | 5.3424 | 26.7128 | 93.588 | 11.717 | 38.25 | 53.25 |
-| 40000 | 0.6465 | 55.0 | 156.0 | 5.3208 | 26.7055 | 93.614 | 11.72 | 35.0 | 65.5 |
-| 42500 | 0.6869 | 54.25 | 149.0 | 5.2910 | 26.6805 | 93.701 | 11.731 | 36.0 | 57.0 |
-| 45000 | 0.7273 | 50.25 | 132.0 | 5.1566 | 26.7217 | 93.557 | 11.713 | 30.375 | 50.75 |
-| 47500 | 0.7677 | 50.5 | 124.0 | 5.1251 | 26.7003 | 93.632 | 11.723 | 29.75 | 38.25 |
-| 50000 | 0.8081 | 49.75 | 123.5 | 5.1102 | 26.6997 | 93.634 | 11.723 | 29.0 | 40.0 |
-| 52500 | 0.8485 | 48.75 | 119.5 | 5.0954 | 26.6849 | 93.686 | 11.729 | 29.0 | 35.75 |
-| 55000 | 0.8889 | 48.5 | 117.0 | 5.0744 | 26.7257 | 93.543 | 11.712 | 28.375 | 35.5 |
-| 57500 | 0.9293 | 48.0 | 116.5 | 5.0686 | 26.7133 | 93.586 | 11.717 | 28.125 | 33.25 |
-| 60000 | 0.9697 | 48.0 | 116.5 | 5.0646 | 26.6953 | 93.649 | 11.725 | 28.0 | 33.25 |
-| 61875 | 1.0 | 48.0 | 117.0 | 5.0644 | 26.695 | 93.65 | 11.725 | 28.0 | 33.5 |
+| 0 | 0 | 850403524608.0 | 85212151152640.0 | 21.0960 | 24.9736 | 100.106 | 12.533 | 2952790016.0 | 25013889531904.0 |
+| 2500 | 0.0404 | 748.0 | 6560.0 | 2.6187 | 25.0229 | 99.909 | 12.509 | 464.0 | 2656.0 |
+| 5000 | 0.0808 | 324.0 | 1416.0 | 1.9058 | 24.9884 | 100.046 | 12.526 | 249.0 | 300.0 |
+| 7500 | 0.1212 | 217.0 | 752.0 | 1.6210 | 24.9597 | 100.161 | 12.54 | 181.0 | 190.0 |
+| 10000 | 0.1616 | 172.0 | 716.0 | 1.4406 | 25.0292 | 99.883 | 12.505 | 151.0 | 170.0 |
+| 12500 | 0.2020 | 124.0 | 458.0 | 1.2049 | 25.0192 | 99.923 | 12.51 | 104.0 | 152.0 |
+| 15000 | 0.2424 | 104.5 | 412.0 | 1.0694 | 24.9961 | 100.016 | 12.522 | 88.5 | 145.0 |
+| 17500 | 0.2828 | 92.0 | 346.0 | 0.9815 | 25.0156 | 99.937 | 12.512 | 82.0 | 100.0 |
+| 20000 | 0.3232 | 83.0 | 314.0 | 0.8990 | 25.0162 | 99.935 | 12.512 | 68.0 | 105.0 |
+| 22500 | 0.3636 | 70.5 | 230.0 | 0.7774 | 24.9804 | 100.079 | 12.53 | 57.5 | 72.5 |
+| 25000 | 0.4040 | 65.0 | 222.0 | 0.7207 | 25.0102 | 99.959 | 12.515 | 51.75 | 96.5 |
+| 27500 | 0.4444 | 64.5 | 202.0 | 0.6891 | 25.0139 | 99.945 | 12.513 | 49.25 | 80.0 |
+| 30000 | 0.4848 | 62.0 | 200.0 | 0.6958 | 24.9984 | 100.006 | 12.521 | 48.25 | 66.0 |
+| 32500 | 0.5253 | 65.0 | 221.0 | 0.6815 | 25.0096 | 99.962 | 12.515 | 47.5 | 402.0 |
+| 35000 | 0.5657 | 60.25 | 189.0 | 0.6250 | 25.0328 | 99.869 | 12.504 | 43.25 | 95.5 |
+| 37500 | 0.6061 | 59.0 | 165.0 | 0.6019 | 25.0027 | 99.989 | 12.519 | 43.5 | 72.5 |
+| 40000 | 0.6465 | 57.5 | 155.0 | 0.5777 | 24.9823 | 100.071 | 12.529 | 39.0 | 90.5 |
+| 42500 | 0.6869 | 59.25 | 161.0 | 0.5669 | 25.0242 | 99.903 | 12.508 | 39.5 | 57.0 |
+| 45000 | 0.7273 | 53.0 | 149.0 | 0.4764 | 25.0157 | 99.937 | 12.512 | 34.0 | 42.25 |
+| 47500 | 0.7677 | 50.5 | 130.0 | 0.4510 | 24.979 | 100.084 | 12.531 | 33.0 | 40.0 |
+| 50000 | 0.8081 | 50.5 | 129.0 | 0.4420 | 24.9761 | 100.096 | 12.532 | 31.75 | 37.0 |
+| 52500 | 0.8485 | 50.25 | 134.0 | 0.4333 | 24.9616 | 100.154 | 12.539 | 31.625 | 37.0 |
+| 55000 | 0.8889 | 49.25 | 124.0 | 0.4170 | 24.9558 | 100.177 | 12.542 | 30.375 | 35.0 |
+| 57500 | 0.9293 | 49.0 | 124.5 | 0.4120 | 24.9149 | 100.342 | 12.563 | 30.25 | 34.75 |
+| 60000 | 0.9697 | 49.0 | 123.5 | 0.4089 | 24.9811 | 100.076 | 12.529 | 30.125 | 34.25 |
+| 61875 | 1.0 | 49.0 | 123.0 | 0.4086 | 24.9186 | 100.327 | 12.561 | 30.125 | 34.25 |
 
 # Resource Usage Comparison
 
-- VRAM Use: 7.7851 GB
+- VRAM Use: 7.7831 GB
 
 # Distillation (Teacher -> Student) Architecture Difference:
 
@@ -93,7 +93,7 @@ More information needed
 <br/>
 
 # Train Dataset
-Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 145,731,638 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `247,500`
 - Subset: `20231101.en`
@@ -103,7 +103,7 @@ Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface
 # Training Objective
 
 ```
-DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))
+DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))
 ```
 
 # Hyperparameters
@@ -120,9 +120,9 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: `linear`
 - lr_scheduler_warmup_ratio: `0.5`
 - num_epochs: `1.0`
-- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))`
+- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))`
 - train_embeddings: `True`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc57c38a620>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f6f213d8430>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `None`
@@ -155,5 +155,5 @@ The following hyperparameters were used during training:
 # Framework Versions
 - Distily 0.2.0
 - Transformers 4.44.1
-- Pytorch 2.3.0
+- Pytorch 2.5.0.dev20240821+cu121
 - Datasets 2.21.0

logs/attn_loss_fn=raw_mse, attn_weight=10.0, projector=identity/events.out.tfevents.1724319775.e3f806ea38c9

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:876c84b91a2986e18efa4ac943507bf521e1b7dd0c5d9b651fa646cb8d3b1896
+size 588