End of training

README.md

@@ -44,36 +44,36 @@ More information needed
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
 | **teacher eval** |  | 43.25 | 61.25 |  |  |  |  | 11.6875 | 19.125 |
-| 0 | 0 | 850403524608.0 | 85212151152640.0 | 21.0960 | 24.9736 | 100.106 | 12.533 | 2952790016.0 | 25013889531904.0 |
-| 2500 | 0.0404 | 748.0 | 6560.0 | 2.6187 | 25.0229 | 99.909 | 12.509 | 464.0 | 2656.0 |
-| 5000 | 0.0808 | 324.0 | 1416.0 | 1.9058 | 24.9884 | 100.046 | 12.526 | 249.0 | 300.0 |
-| 7500 | 0.1212 | 217.0 | 752.0 | 1.6210 | 24.9597 | 100.161 | 12.54 | 181.0 | 190.0 |
-| 10000 | 0.1616 | 172.0 | 716.0 | 1.4406 | 25.0292 | 99.883 | 12.505 | 151.0 | 170.0 |
-| 12500 | 0.2020 | 124.0 | 458.0 | 1.2049 | 25.0192 | 99.923 | 12.51 | 104.0 | 152.0 |
-| 15000 | 0.2424 | 104.5 | 412.0 | 1.0694 | 24.9961 | 100.016 | 12.522 | 88.5 | 145.0 |
-| 17500 | 0.2828 | 92.0 | 346.0 | 0.9815 | 25.0156 | 99.937 | 12.512 | 82.0 | 100.0 |
-| 20000 | 0.3232 | 83.0 | 314.0 | 0.8990 | 25.0162 | 99.935 | 12.512 | 68.0 | 105.0 |
-| 22500 | 0.3636 | 70.5 | 230.0 | 0.7774 | 24.9804 | 100.079 | 12.53 | 57.5 | 72.5 |
-| 25000 | 0.4040 | 65.0 | 222.0 | 0.7207 | 25.0102 | 99.959 | 12.515 | 51.75 | 96.5 |
-| 27500 | 0.4444 | 64.5 | 202.0 | 0.6891 | 25.0139 | 99.945 | 12.513 | 49.25 | 80.0 |
-| 30000 | 0.4848 | 62.0 | 200.0 | 0.6958 | 24.9984 | 100.006 | 12.521 | 48.25 | 66.0 |
-| 32500 | 0.5253 | 65.0 | 221.0 | 0.6815 | 25.0096 | 99.962 | 12.515 | 47.5 | 402.0 |
-| 35000 | 0.5657 | 60.25 | 189.0 | 0.6250 | 25.0328 | 99.869 | 12.504 | 43.25 | 95.5 |
-| 37500 | 0.6061 | 59.0 | 165.0 | 0.6019 | 25.0027 | 99.989 | 12.519 | 43.5 | 72.5 |
-| 40000 | 0.6465 | 57.5 | 155.0 | 0.5777 | 24.9823 | 100.071 | 12.529 | 39.0 | 90.5 |
-| 42500 | 0.6869 | 59.25 | 161.0 | 0.5669 | 25.0242 | 99.903 | 12.508 | 39.5 | 57.0 |
-| 45000 | 0.7273 | 53.0 | 149.0 | 0.4764 | 25.0157 | 99.937 | 12.512 | 34.0 | 42.25 |
-| 47500 | 0.7677 | 50.5 | 130.0 | 0.4510 | 24.979 | 100.084 | 12.531 | 33.0 | 40.0 |
-| 50000 | 0.8081 | 50.5 | 129.0 | 0.4420 | 24.9761 | 100.096 | 12.532 | 31.75 | 37.0 |
-| 52500 | 0.8485 | 50.25 | 134.0 | 0.4333 | 24.9616 | 100.154 | 12.539 | 31.625 | 37.0 |
-| 55000 | 0.8889 | 49.25 | 124.0 | 0.4170 | 24.9558 | 100.177 | 12.542 | 30.375 | 35.0 |
-| 57500 | 0.9293 | 49.0 | 124.5 | 0.4120 | 24.9149 | 100.342 | 12.563 | 30.25 | 34.75 |
-| 60000 | 0.9697 | 49.0 | 123.5 | 0.4089 | 24.9811 | 100.076 | 12.529 | 30.125 | 34.25 |
-| 61875 | 1.0 | 49.0 | 123.0 | 0.4086 | 24.9186 | 100.327 | 12.561 | 30.125 | 34.25 |
+| 0 | 0 | 2473901162496.0 | 170424302305280.0 | 30.7740 | 25.2349 | 99.069 | 12.403 | 4060086272.0 | 71468255805440.0 |
+| 2500 | 0.0404 | 1184.0 | 11776.0 | 9.8284 | 25.2487 | 99.015 | 12.397 | 784.0 | 12800.0 |
+| 5000 | 0.0808 | 412.0 | 2272.0 | 8.3993 | 25.2618 | 98.964 | 12.39 | 290.0 | 434.0 |
+| 7500 | 0.1212 | 245.0 | 916.0 | 7.6586 | 25.2887 | 98.858 | 12.377 | 218.0 | 195.0 |
+| 10000 | 0.1616 | 182.0 | 676.0 | 7.2415 | 25.2556 | 98.988 | 12.393 | 164.0 | 190.0 |
+| 12500 | 0.2020 | 131.0 | 504.0 | 6.6883 | 25.2962 | 98.829 | 12.373 | 115.5 | 158.0 |
+| 15000 | 0.2424 | 112.5 | 432.0 | 6.4127 | 25.2743 | 98.915 | 12.384 | 89.5 | 144.0 |
+| 17500 | 0.2828 | 93.5 | 344.0 | 6.1979 | 25.214 | 99.151 | 12.414 | 70.5 | 127.0 |
+| 20000 | 0.3232 | 75.0 | 270.0 | 5.9310 | 25.2265 | 99.102 | 12.408 | 63.75 | 128.0 |
+| 22500 | 0.3636 | 67.0 | 209.0 | 5.6634 | 25.2495 | 99.012 | 12.396 | 49.75 | 83.5 |
+| 25000 | 0.4040 | 63.5 | 192.0 | 5.5561 | 25.2476 | 99.019 | 12.397 | 44.25 | 86.0 |
+| 27500 | 0.4444 | 58.0 | 192.0 | 5.4855 | 25.2834 | 98.879 | 12.38 | 40.25 | 70.5 |
+| 30000 | 0.4848 | 58.75 | 195.0 | 5.4646 | 25.2547 | 98.992 | 12.394 | 41.75 | 65.0 |
+| 32500 | 0.5253 | 58.5 | 171.0 | 5.4511 | 25.2 | 99.206 | 12.421 | 40.0 | 60.0 |
+| 35000 | 0.5657 | 57.0 | 165.0 | 5.3711 | 25.2873 | 98.864 | 12.378 | 36.75 | 49.25 |
+| 37500 | 0.6061 | 57.75 | 155.0 | 5.3390 | 25.2952 | 98.833 | 12.374 | 37.75 | 54.0 |
+| 40000 | 0.6465 | 55.75 | 154.0 | 5.3225 | 25.2919 | 98.846 | 12.376 | 34.5 | 57.0 |
+| 42500 | 0.6869 | 54.75 | 146.0 | 5.2939 | 25.2713 | 98.926 | 12.386 | 35.5 | 49.0 |
+| 45000 | 0.7273 | 50.75 | 133.0 | 5.1563 | 25.2812 | 98.888 | 12.381 | 30.0 | 48.0 |
+| 47500 | 0.7677 | 50.75 | 124.5 | 5.1271 | 25.3128 | 98.764 | 12.365 | 29.375 | 35.25 |
+| 50000 | 0.8081 | 49.75 | 123.0 | 5.1093 | 25.2369 | 99.061 | 12.402 | 28.75 | 37.5 |
+| 52500 | 0.8485 | 48.5 | 119.5 | 5.0960 | 25.2934 | 98.84 | 12.375 | 28.875 | 34.5 |
+| 55000 | 0.8889 | 48.5 | 118.0 | 5.0747 | 25.23 | 99.088 | 12.406 | 28.0 | 33.25 |
+| 57500 | 0.9293 | 48.0 | 117.0 | 5.0698 | 25.2235 | 99.114 | 12.409 | 27.75 | 32.0 |
+| 60000 | 0.9697 | 48.0 | 117.0 | 5.0651 | 25.2107 | 99.164 | 12.415 | 27.75 | 31.875 |
+| 61875 | 1.0 | 48.0 | 117.5 | 5.0643 | 25.1856 | 99.263 | 12.428 | 27.625 | 32.0 |
 
 # Resource Usage Comparison
 
-- VRAM Use: 7.7831 GB
+- VRAM Use: 7.7843 GB
 
 # Distillation (Teacher -> Student) Architecture Difference:
 
@@ -93,7 +93,7 @@ More information needed
 <br/>
 
 # Train Dataset
-Trained on 145,731,638 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `247,500`
 - Subset: `20231101.en`
@@ -103,7 +103,7 @@ Trained on 145,731,638 tokens from the [wikimedia/wikipedia](https://huggingface
 # Training Objective
 
 ```
-DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))
+DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))
 ```
 
 # Hyperparameters
@@ -120,9 +120,9 @@ The following hyperparameters were used during training:
 - lr_scheduler_type: `linear`
 - lr_scheduler_warmup_ratio: `0.5`
 - num_epochs: `1.0`
-- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=raw_mse, layer_mapper=layer-2))`
+- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=cos, layer_mapper=layer-2))`
 - train_embeddings: `True`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f6f213d8430>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f6f2143c700>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `None`

logs/attn_loss_fn=cos, attn_weight=10.0, projector=identity/events.out.tfevents.1724327951.e3f806ea38c9

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9309b9cf44c77a5b34c6655145c99d44043ad6f6cb7f10c8c29258616fd20be2
+size 588