End of training

README.md

@@ -0,0 +1,133 @@
+---
+base_model: gpt2
+library_name: Distily
+license: mit
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_gpt2_activation_loss_c
+  results: []
+---
+
+# distily_bench_gpt2_activation_loss_c
+
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 214.4708
+- eval_frwikippl: 1120.6917
+- eval_zhwikippl: 530.7340
+- eval_loss: 1.1946
+- eval_runtime: 83.9846
+- eval_samples_per_second: 59.535
+- eval_steps_per_second: 7.442
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- train_embeddings: True
+- learning_rate: 4e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 7.9371 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
+| 0 | 0 | 57579.4648 | 56634.7969 | 5.8161 | 83.5176 | 59.868 | 7.483 | 59731.6172 |
+| 1000 | 0.0162 | 722.1866 | 4337.8599 | 1.8558 | 84.2964 | 59.314 | 7.414 | 11505.3213 |
+| 2000 | 0.0323 | 509.9531 | 3158.9646 | 1.6720 | 83.9707 | 59.545 | 7.443 | 1535.8192 |
+| 3000 | 0.0485 | 426.4450 | 2554.9209 | 1.5690 | 83.88 | 59.609 | 7.451 | 1022.2861 |
+| 4000 | 0.0646 | 366.2093 | 2339.7170 | 1.4872 | 84.1255 | 59.435 | 7.429 | 988.9852 |
+| 5000 | 0.0808 | 318.6859 | 1895.2631 | 1.4137 | 83.9179 | 59.582 | 7.448 | 853.8780 |
+| 6000 | 0.0970 | 284.2188 | 1590.7739 | 1.3445 | 83.9081 | 59.589 | 7.449 | 738.2130 |
+| 7000 | 0.1131 | 255.0395 | 1379.4120 | 1.2879 | 83.8508 | 59.63 | 7.454 | 757.3848 |
+| 8000 | 0.1293 | 233.6866 | 1271.9808 | 1.2390 | 84.5663 | 59.125 | 7.391 | 652.2598 |
+| 9000 | 0.1455 | 214.4708 | 1120.6917 | 1.1946 | 83.9846 | 59.535 | 7.442 | 530.7340 |
+| 10000 | 0.1616 | 197.4782 | 1103.2870 | 1.1508 | 84.0448 | 59.492 | 7.437 | 636.5135 |
+| 11000 | 0.1778 | 182.4826 | 1116.1182 | 1.1095 | 83.8827 | 59.607 | 7.451 | 616.4374 |
+| 12000 | 0.1939 | 171.1188 | 1000.4317 | 1.0699 | 84.1267 | 59.434 | 7.429 | 670.2728 |
+| 13000 | 0.2101 | 158.5921 | 926.2961 | 1.0328 | 84.2534 | 59.345 | 7.418 | 632.1090 |
+| 14000 | 0.2263 | 151.5253 | 819.1207 | 1.0080 | 84.4702 | 59.192 | 7.399 | 606.3138 |
+| 15000 | 0.2424 | 145.7775 | 759.5986 | 0.9852 | 83.647 | 59.775 | 7.472 | 439.2366 |
+| 16000 | 0.2586 | 140.4984 | 774.4702 | 0.9718 | 83.7317 | 59.715 | 7.464 | 461.6688 |
+| 17000 | 0.2747 | 137.8507 | 711.1382 | 0.9582 | 84.0042 | 59.521 | 7.44 | 422.0445 |
+| 18000 | 0.2909 | 137.0289 | 709.6857 | 0.9461 | 84.1862 | 59.392 | 7.424 | 465.4448 |
+| 19000 | 0.3071 | 131.9026 | 673.5125 | 0.9349 | 84.1157 | 59.442 | 7.43 | 436.7800 |
+| 20000 | 0.3232 | 130.3853 | 649.6329 | 0.9254 | 83.8519 | 59.629 | 7.454 | 337.0057 |
+| 21000 | 0.3394 | 128.9655 | 670.7639 | 0.9189 | 83.8863 | 59.605 | 7.451 | 325.8972 |
+| 22000 | 0.3556 | 124.7593 | 654.3679 | 0.9088 | 83.8586 | 59.624 | 7.453 | 331.9586 |
+| 23000 | 0.3717 | 123.4583 | 693.6086 | 0.9066 | 83.9555 | 59.555 | 7.444 | 331.2501 |
+| 24000 | 0.3879 | 123.4583 | 657.0493 | 0.8998 | 83.9273 | 59.575 | 7.447 | 340.5343 |
+| 25000 | 0.4040 | 122.2752 | 625.8052 | 0.8943 | 83.8458 | 59.633 | 7.454 | 417.7828 |
+| 26000 | 0.4202 | 122.2183 | 657.2346 | 0.8931 | 84.4963 | 59.174 | 7.397 | 621.3131 |
+| 27000 | 0.4364 | 120.8686 | 631.9240 | 0.8893 | 83.9715 | 59.544 | 7.443 | 352.2357 |
+| 28000 | 0.4525 | 121.1317 | 636.7093 | 0.8845 | 83.9169 | 59.583 | 7.448 | 309.6069 |
+| 29000 | 0.4687 | 119.4132 | 628.9014 | 0.8832 | 83.8228 | 59.65 | 7.456 | 313.4341 |
+| 30000 | 0.4848 | 117.5456 | 634.5134 | 0.8788 | 83.8793 | 59.609 | 7.451 | 447.6450 |
+| 31000 | 0.5010 | 121.2258 | 630.5887 | 0.8781 | 83.9747 | 59.542 | 7.443 | 346.9143 |
+| 32000 | 0.5172 | 119.7382 | 649.1292 | 0.8732 | 83.9217 | 59.579 | 7.447 | 299.0420 |
+| 33000 | 0.5333 | 117.6826 | 597.3090 | 0.8709 | 84.0205 | 59.509 | 7.439 | 376.2052 |
+| 34000 | 0.5495 | 115.3931 | 648.1230 | 0.8693 | 84.0586 | 59.482 | 7.435 | 354.7850 |
+| 35000 | 0.5657 | 115.3573 | 604.2133 | 0.8687 | 83.9186 | 59.582 | 7.448 | 301.0854 |
+| 36000 | 0.5818 | 115.5097 | 655.6147 | 0.8642 | 83.8597 | 59.623 | 7.453 | 342.8156 |
+| 37000 | 0.5980 | 115.6533 | 596.4253 | 0.8617 | 83.9157 | 59.584 | 7.448 | 441.5301 |
+| 38000 | 0.6141 | 115.5187 | 585.1377 | 0.8599 | 83.8024 | 59.664 | 7.458 | 357.8301 |
+| 39000 | 0.6303 | 114.7854 | 604.0002 | 0.8576 | 83.8371 | 59.639 | 7.455 | 584.5303 |
+| 40000 | 0.6465 | 113.0692 | 590.0259 | 0.8554 | 83.851 | 59.63 | 7.454 | 326.7251 |
+| 41000 | 0.6626 | 114.2164 | 583.6541 | 0.8545 | 83.8543 | 59.627 | 7.453 | 314.1884 |
+| 42000 | 0.6788 | 114.7053 | 581.4363 | 0.8539 | 83.8887 | 59.603 | 7.45 | 315.9555 |
+| 43000 | 0.6949 | 113.6149 | 625.1437 | 0.8533 | 83.8964 | 59.597 | 7.45 | 385.3576 |
+| 44000 | 0.7111 | 112.8849 | 567.9040 | 0.8525 | 84.0503 | 59.488 | 7.436 | 334.0932 |
+| 45000 | 0.7273 | 114.1277 | 569.7891 | 0.8511 | 84.1155 | 59.442 | 7.43 | 255.3773 |
+| 46000 | 0.7434 | 112.0639 | 555.1952 | 0.8483 | 83.9445 | 59.563 | 7.445 | 334.2270 |
+| 47000 | 0.7596 | 112.9639 | 581.0264 | 0.8477 | 83.8652 | 59.62 | 7.452 | 334.5396 |
+| 48000 | 0.7758 | 112.9025 | 589.9427 | 0.8484 | 83.9109 | 59.587 | 7.448 | 346.3587 |
+| 49000 | 0.7919 | 110.6717 | 572.0029 | 0.8471 | 84.1959 | 59.385 | 7.423 | 331.0290 |
+| 50000 | 0.8081 | 112.7535 | 564.6300 | 0.8457 | 84.1925 | 59.388 | 7.423 | 295.1146 |
+| 51000 | 0.8242 | 111.4478 | 558.8870 | 0.8415 | 84.0779 | 59.469 | 7.434 | 282.3920 |
+| 52000 | 0.8404 | 112.9814 | 552.8126 | 0.8414 | 102.3824 | 48.837 | 6.105 | 362.7858 |
+| 53000 | 0.8566 | 111.2230 | 548.1937 | 0.8397 | 84.0481 | 59.49 | 7.436 | 341.8555 |
+| 54000 | 0.8727 | 110.5084 | 575.6036 | 0.8420 | 84.4892 | 59.179 | 7.397 | 396.7935 |
+| 55000 | 0.8889 | 112.1945 | 559.5179 | 0.8408 | 84.2987 | 59.313 | 7.414 | 603.7286 |
+| 56000 | 0.9051 | 110.6201 | 557.0381 | 0.8390 | 84.3563 | 59.272 | 7.409 | 280.8501 |
+| 57000 | 0.9212 | 111.2490 | 560.7422 | 0.8403 | 84.0925 | 59.458 | 7.432 | 242.3530 |
+| 58000 | 0.9374 | 111.9074 | 636.3951 | 0.8443 | 84.3426 | 59.282 | 7.41 | 1202.8456 |
+| 59000 | 0.9535 | 111.5430 | 534.7561 | 0.8396 | 84.1029 | 59.451 | 7.431 | 353.7442 |
+| 60000 | 0.9697 | 110.6287 | 566.9437 | 0.8417 | 84.411 | 59.234 | 7.404 | 330.7196 |
+| 61000 | 0.9859 | 110.7576 | 560.9399 | 0.8381 | 84.4585 | 59.201 | 7.4 | 279.1303 |
+| 61875 | 1.0 | 110.3713 | 541.5095 | 0.8372 | 84.1306 | 59.431 | 7.429 | 307.0541 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.21.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/events.out.tfevents.1723730974.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53b0c99635642b68455071ea47ece5f44e17e897865d6b06e527892340daaf17
+size 253