End of training

README.md

@@ -16,14 +16,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 2192.0
-- eval_frwikippl: 9344.0
-- eval_zhwikippl: 165888.0
-- eval_tinystoriesppl: 1544.0
-- eval_loss: 3.2467
-- eval_runtime: 12.9561
-- eval_samples_per_second: 46.31
-- eval_steps_per_second: 11.578
+- eval_enwikippl: 1272.0
+- eval_frwikippl: 5952.0
+- eval_zhwikippl: 45056.0
+- eval_tinystoriesppl: 816.0
+- eval_loss: 2.4585
+- eval_runtime: 12.9232
+- eval_samples_per_second: 46.428
+- eval_steps_per_second: 11.607
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -46,7 +46,7 @@ More information needed
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=1.0, loss_fn=kl, layer_mapper=uniform+last, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=1.0, loss_fn=mse, layer_mapper=all, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
 - train_embeddings: True
 - learning_rate: 0.0001
 - train_batch_size: 8
@@ -64,17 +64,17 @@ Peak GPU Memory: 8.0905 GB
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 43.75 | 61.75 |  |  |  |  | 11.8125 | 19.125 |
-| 0 | 0 | 1821066133504.0 | 158329674399744.0 | 25.4650 | 12.9114 | 46.47 | 11.618 | 12079595520.0 | 98956046499840.0 |
-| 750 | 0.1010 | 2192.0 | 9344.0 | 3.2467 | 12.9561 | 46.31 | 11.578 | 1544.0 | 165888.0 |
-| 1500 | 0.2020 | 776.0 | 4704.0 | 2.2915 | 12.9788 | 46.229 | 11.557 | 584.0 | 5152.0 |
-| 2250 | 0.3030 | 454.0 | 2800.0 | 1.9392 | 13.006 | 46.133 | 11.533 | 358.0 | 604.0 |
-| 3000 | 0.4040 | 320.0 | 1512.0 | 1.6692 | 12.9768 | 46.236 | 11.559 | 255.0 | 278.0 |
-| 3750 | 0.5051 | 254.0 | 896.0 | 1.4808 | 13.0015 | 46.148 | 11.537 | 199.0 | 364.0 |
-| 4500 | 0.6061 | 185.0 | 684.0 | 1.2715 | 13.0003 | 46.153 | 11.538 | 152.0 | 516.0 |
-| 5250 | 0.7071 | 148.0 | 572.0 | 1.1020 | 12.9814 | 46.22 | 11.555 | 114.5 | 200.0 |
-| 6000 | 0.8081 | 134.0 | 490.0 | 1.0245 | 12.9705 | 46.259 | 11.565 | 104.0 | 178.0 |
-| 6750 | 0.9091 | 126.5 | 464.0 | 0.9858 | 13.0461 | 45.991 | 11.498 | 97.5 | 168.0 |
-| 7425 | 1.0 | 124.5 | 462.0 | 0.9781 | 13.0812 | 45.867 | 11.467 | 96.5 | 163.0 |
+| 0 | 0 | 1821066133504.0 | 158329674399744.0 | 19.3254 | 12.8781 | 46.591 | 11.648 | 12079595520.0 | 98956046499840.0 |
+| 750 | 0.1010 | 1272.0 | 5952.0 | 2.4585 | 12.9232 | 46.428 | 11.607 | 816.0 | 45056.0 |
+| 1500 | 0.2020 | 506.0 | 3248.0 | 1.7942 | 12.9482 | 46.338 | 11.585 | 352.0 | 736.0 |
+| 2250 | 0.3030 | 338.0 | 1632.0 | 1.5656 | 13.0688 | 45.911 | 11.478 | 278.0 | 308.0 |
+| 3000 | 0.4040 | 240.0 | 916.0 | 1.3548 | 13.029 | 46.051 | 11.513 | 209.0 | 191.0 |
+| 3750 | 0.5051 | 199.0 | 624.0 | 1.2058 | 13.0119 | 46.111 | 11.528 | 169.0 | 173.0 |
+| 4500 | 0.6061 | 158.0 | 512.0 | 1.0598 | 12.9786 | 46.23 | 11.557 | 128.0 | 164.0 |
+| 5250 | 0.7071 | 128.0 | 460.0 | 0.9367 | 12.9927 | 46.18 | 11.545 | 105.5 | 135.0 |
+| 6000 | 0.8081 | 118.0 | 416.0 | 0.8779 | 13.0165 | 46.095 | 11.524 | 96.5 | 132.0 |
+| 6750 | 0.9091 | 112.5 | 398.0 | 0.8504 | 13.0993 | 45.804 | 11.451 | 92.5 | 130.0 |
+| 7425 | 1.0 | 112.0 | 394.0 | 0.8451 | 12.9624 | 46.288 | 11.572 | 91.0 | 129.0 |
 
 ### Framework versions
 - Distily 0.2.0

logs/hs_layer_mapper=all, hs_loss_fn=mse, hs_weight=1.0/events.out.tfevents.1724106144.f383272e719b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0df2c3b0af61c333143ddced6f9e88ddd1378b106fceecdf0683ffb44e97140
+size 578