Hugging Face's logo

lapp0
/
distily_bench_gpt2_activation_loss_c

TensorBoard
Safetensors
Distily
gpt2
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Metrics Community
lapp0 commited on
Commit
fa14ba4
•
1 Parent(s): 8a10f67

End of training

Browse files
Files changed (2) hide show
  1. README.md +72 -72
  2. logs/hs_loss_fn=mse, hs_weight=2.0/events.out.tfevents.1723754495.5f530b1cf724 +3 -0
README.md CHANGED
@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
- - eval_enwikippl: 214.4708
20
- - eval_frwikippl: 1120.6917
21
- - eval_zhwikippl: 530.7340
22
- - eval_loss: 1.1946
23
- - eval_runtime: 83.9846
24
- - eval_samples_per_second: 59.535
25
- - eval_steps_per_second: 7.442
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
- - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
@@ -56,75 +56,75 @@ The following hyperparameters were used during training:
56
  - num_epochs: 1.0
57
 
58
  ### Resource Usage
59
- Peak GPU Memory: 7.9371 GB
60
 
61
  ### Eval-Phase Metrics
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
- | 0 | 0 | 57579.4648 | 56634.7969 | 5.8161 | 83.5176 | 59.868 | 7.483 | 59731.6172 |
66
- | 1000 | 0.0162 | 722.1866 | 4337.8599 | 1.8558 | 84.2964 | 59.314 | 7.414 | 11505.3213 |
67
- | 2000 | 0.0323 | 509.9531 | 3158.9646 | 1.6720 | 83.9707 | 59.545 | 7.443 | 1535.8192 |
68
- | 3000 | 0.0485 | 426.4450 | 2554.9209 | 1.5690 | 83.88 | 59.609 | 7.451 | 1022.2861 |
69
- | 4000 | 0.0646 | 366.2093 | 2339.7170 | 1.4872 | 84.1255 | 59.435 | 7.429 | 988.9852 |
70
- | 5000 | 0.0808 | 318.6859 | 1895.2631 | 1.4137 | 83.9179 | 59.582 | 7.448 | 853.8780 |
71
- | 6000 | 0.0970 | 284.2188 | 1590.7739 | 1.3445 | 83.9081 | 59.589 | 7.449 | 738.2130 |
72
- | 7000 | 0.1131 | 255.0395 | 1379.4120 | 1.2879 | 83.8508 | 59.63 | 7.454 | 757.3848 |
73
- | 8000 | 0.1293 | 233.6866 | 1271.9808 | 1.2390 | 84.5663 | 59.125 | 7.391 | 652.2598 |
74
- | 9000 | 0.1455 | 214.4708 | 1120.6917 | 1.1946 | 83.9846 | 59.535 | 7.442 | 530.7340 |
75
- | 10000 | 0.1616 | 197.4782 | 1103.2870 | 1.1508 | 84.0448 | 59.492 | 7.437 | 636.5135 |
76
- | 11000 | 0.1778 | 182.4826 | 1116.1182 | 1.1095 | 83.8827 | 59.607 | 7.451 | 616.4374 |
77
- | 12000 | 0.1939 | 171.1188 | 1000.4317 | 1.0699 | 84.1267 | 59.434 | 7.429 | 670.2728 |
78
- | 13000 | 0.2101 | 158.5921 | 926.2961 | 1.0328 | 84.2534 | 59.345 | 7.418 | 632.1090 |
79
- | 14000 | 0.2263 | 151.5253 | 819.1207 | 1.0080 | 84.4702 | 59.192 | 7.399 | 606.3138 |
80
- | 15000 | 0.2424 | 145.7775 | 759.5986 | 0.9852 | 83.647 | 59.775 | 7.472 | 439.2366 |
81
- | 16000 | 0.2586 | 140.4984 | 774.4702 | 0.9718 | 83.7317 | 59.715 | 7.464 | 461.6688 |
82
- | 17000 | 0.2747 | 137.8507 | 711.1382 | 0.9582 | 84.0042 | 59.521 | 7.44 | 422.0445 |
83
- | 18000 | 0.2909 | 137.0289 | 709.6857 | 0.9461 | 84.1862 | 59.392 | 7.424 | 465.4448 |
84
- | 19000 | 0.3071 | 131.9026 | 673.5125 | 0.9349 | 84.1157 | 59.442 | 7.43 | 436.7800 |
85
- | 20000 | 0.3232 | 130.3853 | 649.6329 | 0.9254 | 83.8519 | 59.629 | 7.454 | 337.0057 |
86
- | 21000 | 0.3394 | 128.9655 | 670.7639 | 0.9189 | 83.8863 | 59.605 | 7.451 | 325.8972 |
87
- | 22000 | 0.3556 | 124.7593 | 654.3679 | 0.9088 | 83.8586 | 59.624 | 7.453 | 331.9586 |
88
- | 23000 | 0.3717 | 123.4583 | 693.6086 | 0.9066 | 83.9555 | 59.555 | 7.444 | 331.2501 |
89
- | 24000 | 0.3879 | 123.4583 | 657.0493 | 0.8998 | 83.9273 | 59.575 | 7.447 | 340.5343 |
90
- | 25000 | 0.4040 | 122.2752 | 625.8052 | 0.8943 | 83.8458 | 59.633 | 7.454 | 417.7828 |
91
- | 26000 | 0.4202 | 122.2183 | 657.2346 | 0.8931 | 84.4963 | 59.174 | 7.397 | 621.3131 |
92
- | 27000 | 0.4364 | 120.8686 | 631.9240 | 0.8893 | 83.9715 | 59.544 | 7.443 | 352.2357 |
93
- | 28000 | 0.4525 | 121.1317 | 636.7093 | 0.8845 | 83.9169 | 59.583 | 7.448 | 309.6069 |
94
- | 29000 | 0.4687 | 119.4132 | 628.9014 | 0.8832 | 83.8228 | 59.65 | 7.456 | 313.4341 |
95
- | 30000 | 0.4848 | 117.5456 | 634.5134 | 0.8788 | 83.8793 | 59.609 | 7.451 | 447.6450 |
96
- | 31000 | 0.5010 | 121.2258 | 630.5887 | 0.8781 | 83.9747 | 59.542 | 7.443 | 346.9143 |
97
- | 32000 | 0.5172 | 119.7382 | 649.1292 | 0.8732 | 83.9217 | 59.579 | 7.447 | 299.0420 |
98
- | 33000 | 0.5333 | 117.6826 | 597.3090 | 0.8709 | 84.0205 | 59.509 | 7.439 | 376.2052 |
99
- | 34000 | 0.5495 | 115.3931 | 648.1230 | 0.8693 | 84.0586 | 59.482 | 7.435 | 354.7850 |
100
- | 35000 | 0.5657 | 115.3573 | 604.2133 | 0.8687 | 83.9186 | 59.582 | 7.448 | 301.0854 |
101
- | 36000 | 0.5818 | 115.5097 | 655.6147 | 0.8642 | 83.8597 | 59.623 | 7.453 | 342.8156 |
102
- | 37000 | 0.5980 | 115.6533 | 596.4253 | 0.8617 | 83.9157 | 59.584 | 7.448 | 441.5301 |
103
- | 38000 | 0.6141 | 115.5187 | 585.1377 | 0.8599 | 83.8024 | 59.664 | 7.458 | 357.8301 |
104
- | 39000 | 0.6303 | 114.7854 | 604.0002 | 0.8576 | 83.8371 | 59.639 | 7.455 | 584.5303 |
105
- | 40000 | 0.6465 | 113.0692 | 590.0259 | 0.8554 | 83.851 | 59.63 | 7.454 | 326.7251 |
106
- | 41000 | 0.6626 | 114.2164 | 583.6541 | 0.8545 | 83.8543 | 59.627 | 7.453 | 314.1884 |
107
- | 42000 | 0.6788 | 114.7053 | 581.4363 | 0.8539 | 83.8887 | 59.603 | 7.45 | 315.9555 |
108
- | 43000 | 0.6949 | 113.6149 | 625.1437 | 0.8533 | 83.8964 | 59.597 | 7.45 | 385.3576 |
109
- | 44000 | 0.7111 | 112.8849 | 567.9040 | 0.8525 | 84.0503 | 59.488 | 7.436 | 334.0932 |
110
- | 45000 | 0.7273 | 114.1277 | 569.7891 | 0.8511 | 84.1155 | 59.442 | 7.43 | 255.3773 |
111
- | 46000 | 0.7434 | 112.0639 | 555.1952 | 0.8483 | 83.9445 | 59.563 | 7.445 | 334.2270 |
112
- | 47000 | 0.7596 | 112.9639 | 581.0264 | 0.8477 | 83.8652 | 59.62 | 7.452 | 334.5396 |
113
- | 48000 | 0.7758 | 112.9025 | 589.9427 | 0.8484 | 83.9109 | 59.587 | 7.448 | 346.3587 |
114
- | 49000 | 0.7919 | 110.6717 | 572.0029 | 0.8471 | 84.1959 | 59.385 | 7.423 | 331.0290 |
115
- | 50000 | 0.8081 | 112.7535 | 564.6300 | 0.8457 | 84.1925 | 59.388 | 7.423 | 295.1146 |
116
- | 51000 | 0.8242 | 111.4478 | 558.8870 | 0.8415 | 84.0779 | 59.469 | 7.434 | 282.3920 |
117
- | 52000 | 0.8404 | 112.9814 | 552.8126 | 0.8414 | 102.3824 | 48.837 | 6.105 | 362.7858 |
118
- | 53000 | 0.8566 | 111.2230 | 548.1937 | 0.8397 | 84.0481 | 59.49 | 7.436 | 341.8555 |
119
- | 54000 | 0.8727 | 110.5084 | 575.6036 | 0.8420 | 84.4892 | 59.179 | 7.397 | 396.7935 |
120
- | 55000 | 0.8889 | 112.1945 | 559.5179 | 0.8408 | 84.2987 | 59.313 | 7.414 | 603.7286 |
121
- | 56000 | 0.9051 | 110.6201 | 557.0381 | 0.8390 | 84.3563 | 59.272 | 7.409 | 280.8501 |
122
- | 57000 | 0.9212 | 111.2490 | 560.7422 | 0.8403 | 84.0925 | 59.458 | 7.432 | 242.3530 |
123
- | 58000 | 0.9374 | 111.9074 | 636.3951 | 0.8443 | 84.3426 | 59.282 | 7.41 | 1202.8456 |
124
- | 59000 | 0.9535 | 111.5430 | 534.7561 | 0.8396 | 84.1029 | 59.451 | 7.431 | 353.7442 |
125
- | 60000 | 0.9697 | 110.6287 | 566.9437 | 0.8417 | 84.411 | 59.234 | 7.404 | 330.7196 |
126
- | 61000 | 0.9859 | 110.7576 | 560.9399 | 0.8381 | 84.4585 | 59.201 | 7.4 | 279.1303 |
127
- | 61875 | 1.0 | 110.3713 | 541.5095 | 0.8372 | 84.1306 | 59.431 | 7.429 | 307.0541 |
128
 
129
  ### Framework versions
130
  - Distily 0.2.0
 
16
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
17
 
18
  It achieves the following results on the evaluation set:
19
+ - eval_enwikippl: 215.5059
20
+ - eval_frwikippl: 1193.6056
21
+ - eval_zhwikippl: 627.1483
22
+ - eval_loss: 1.2009
23
+ - eval_runtime: 85.3591
24
+ - eval_samples_per_second: 58.576
25
+ - eval_steps_per_second: 7.322
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
 
45
  ### Training hyperparameters
46
 
47
  The following hyperparameters were used during training:
48
+ - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=2.0, loss_fn=mse, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
49
  - train_embeddings: True
50
  - learning_rate: 4e-05
51
  - train_batch_size: 8
 
56
  - num_epochs: 1.0
57
 
58
  ### Resource Usage
59
+ Peak GPU Memory: 8.0873 GB
60
 
61
  ### Eval-Phase Metrics
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 30.2086 | 57.2728 | | | | | 18.1784 |
65
+ | 0 | 0 | 56314.7695 | 59887.2773 | 5.8256 | 85.6711 | 58.363 | 7.295 | 59033.8086 |
66
+ | 1000 | 0.0162 | 703.3142 | 4236.9004 | 1.8490 | 85.6638 | 58.368 | 7.296 | 11133.5088 |
67
+ | 2000 | 0.0323 | 504.8312 | 3192.5520 | 1.6764 | 85.4461 | 58.516 | 7.315 | 1842.1659 |
68
+ | 3000 | 0.0485 | 421.6048 | 2827.9453 | 1.5711 | 85.6065 | 58.407 | 7.301 | 841.4271 |
69
+ | 4000 | 0.0646 | 359.8385 | 2300.7822 | 1.4898 | 85.5248 | 58.463 | 7.308 | 1321.4115 |
70
+ | 5000 | 0.0808 | 320.1989 | 1782.2493 | 1.4134 | 85.6041 | 58.408 | 7.301 | 921.9020 |
71
+ | 6000 | 0.0970 | 279.3613 | 1572.2640 | 1.3457 | 85.4507 | 58.513 | 7.314 | 775.6033 |
72
+ | 7000 | 0.1131 | 252.3406 | 1452.0632 | 1.2901 | 85.4137 | 58.539 | 7.317 | 675.1237 |
73
+ | 8000 | 0.1293 | 230.5502 | 1345.9784 | 1.2423 | 85.9632 | 58.164 | 7.271 | 594.2899 |
74
+ | 9000 | 0.1455 | 215.5059 | 1193.6056 | 1.2009 | 85.3591 | 58.576 | 7.322 | 627.1483 |
75
+ | 10000 | 0.1616 | 194.5708 | 1147.0729 | 1.1501 | 85.2878 | 58.625 | 7.328 | 681.0092 |
76
+ | 11000 | 0.1778 | 179.9636 | 1066.1221 | 1.1062 | 85.2181 | 58.673 | 7.334 | 556.0541 |
77
+ | 12000 | 0.1939 | 165.4222 | 900.6642 | 1.0627 | 85.2275 | 58.667 | 7.333 | 517.4376 |
78
+ | 13000 | 0.2101 | 155.7605 | 880.5709 | 1.0328 | 85.4983 | 58.481 | 7.31 | 504.9460 |
79
+ | 14000 | 0.2263 | 148.5429 | 820.9711 | 1.0057 | 85.4522 | 58.512 | 7.314 | 432.5430 |
80
+ | 15000 | 0.2424 | 142.2881 | 752.3494 | 0.9840 | 85.291 | 58.623 | 7.328 | 371.8599 |
81
+ | 16000 | 0.2586 | 138.4622 | 756.4453 | 0.9709 | 85.5234 | 58.464 | 7.308 | 645.2426 |
82
+ | 17000 | 0.2747 | 136.4131 | 709.4854 | 0.9606 | 85.2257 | 58.668 | 7.333 | 653.3060 |
83
+ | 18000 | 0.2909 | 133.8840 | 722.0003 | 0.9493 | 85.2137 | 58.676 | 7.335 | 538.2999 |
84
+ | 19000 | 0.3071 | 131.9743 | 726.1355 | 0.9435 | 85.2513 | 58.65 | 7.331 | 595.8792 |
85
+ | 20000 | 0.3232 | 129.7892 | 706.8889 | 0.9335 | 85.2873 | 58.625 | 7.328 | 420.4131 |
86
+ | 21000 | 0.3394 | 127.3829 | 659.1836 | 0.9238 | 85.0899 | 58.761 | 7.345 | 377.2113 |
87
+ | 22000 | 0.3556 | 125.8100 | 627.8823 | 0.9149 | 85.107 | 58.75 | 7.344 | 378.1191 |
88
+ | 23000 | 0.3717 | 124.3241 | 675.1288 | 0.9101 | 85.2843 | 58.627 | 7.328 | 407.0987 |
89
+ | 24000 | 0.3879 | 121.7446 | 648.3518 | 0.9012 | 85.2715 | 58.636 | 7.33 | 370.7195 |
90
+ | 25000 | 0.4040 | 121.8676 | 673.0380 | 0.8998 | 85.4414 | 58.52 | 7.315 | 401.9131 |
91
+ | 26000 | 0.4202 | 121.1881 | 598.3207 | 0.8906 | 85.1925 | 58.691 | 7.336 | 455.3015 |
92
+ | 27000 | 0.4364 | 119.8778 | 614.9578 | 0.8859 | 85.3813 | 58.561 | 7.32 | 291.9007 |
93
+ | 28000 | 0.4525 | 119.7104 | 589.9427 | 0.8831 | 85.3094 | 58.61 | 7.326 | 313.4760 |
94
+ | 29000 | 0.4687 | 118.6553 | 652.7549 | 0.8794 | 85.2629 | 58.642 | 7.33 | 299.0819 |
95
+ | 30000 | 0.4848 | 118.7475 | 602.5115 | 0.8775 | 85.4036 | 58.546 | 7.318 | 355.6388 |
96
+ | 31000 | 0.5010 | 118.1863 | 610.4652 | 0.8759 | 85.1743 | 58.703 | 7.338 | 275.1334 |
97
+ | 32000 | 0.5172 | 117.4726 | 628.6798 | 0.8750 | 85.2859 | 58.626 | 7.328 | 301.3671 |
98
+ | 33000 | 0.5333 | 115.1694 | 602.0021 | 0.8713 | 85.2629 | 58.642 | 7.33 | 277.0137 |
99
+ | 34000 | 0.5495 | 115.8600 | 574.1846 | 0.8689 | 85.1695 | 58.706 | 7.338 | 277.8658 |
100
+ | 35000 | 0.5657 | 114.0391 | 537.2504 | 0.8629 | 85.3032 | 58.614 | 7.327 | 307.7109 |
101
+ | 36000 | 0.5818 | 115.1694 | 602.9366 | 0.8660 | 85.3327 | 58.594 | 7.324 | 328.2996 |
102
+ | 37000 | 0.5980 | 113.9152 | 575.0357 | 0.8590 | 85.7449 | 58.313 | 7.289 | 332.3134 |
103
+ | 38000 | 0.6141 | 114.4739 | 573.7802 | 0.8618 | 85.7064 | 58.339 | 7.292 | 270.8683 |
104
+ | 39000 | 0.6303 | 112.6310 | 546.8427 | 0.8543 | 85.2884 | 58.625 | 7.328 | 289.1075 |
105
+ | 40000 | 0.6465 | 112.8762 | 570.1909 | 0.8537 | 85.2282 | 58.666 | 7.333 | 257.7758 |
106
+ | 41000 | 0.6626 | 112.9112 | 548.9287 | 0.8543 | 85.3272 | 58.598 | 7.325 | 325.8972 |
107
+ | 42000 | 0.6788 | 111.7424 | 549.7032 | 0.8534 | 85.5416 | 58.451 | 7.306 | 291.7448 |
108
+ | 43000 | 0.6949 | 112.2556 | 568.9060 | 0.8524 | 85.2667 | 58.64 | 7.33 | 310.9328 |
109
+ | 44000 | 0.7111 | 110.7490 | 603.5746 | 0.8525 | 85.3547 | 58.579 | 7.322 | 269.4612 |
110
+ | 45000 | 0.7273 | 112.0378 | 593.0288 | 0.8486 | 85.4267 | 58.53 | 7.316 | 378.8268 |
111
+ | 46000 | 0.7434 | 111.5950 | 589.0699 | 0.8492 | 85.0567 | 58.784 | 7.348 | 364.7776 |
112
+ | 47000 | 0.7596 | 112.7010 | 588.7380 | 0.8558 | 85.186 | 58.695 | 7.337 | 446.9284 |
113
+ | 48000 | 0.7758 | 114.3584 | 519.3724 | 0.8590 | 85.1156 | 58.744 | 7.343 | 2148.5159 |
114
+ | 49000 | 0.7919 | 115.1962 | 590.7754 | 0.8648 | 85.1589 | 58.714 | 7.339 | 430.9863 |
115
+ | 50000 | 0.8081 | 114.1809 | 614.9578 | 0.8597 | 85.2319 | 58.663 | 7.333 | 309.8965 |
116
+ | 51000 | 0.8242 | 112.0117 | 593.5725 | 0.8551 | 85.2128 | 58.677 | 7.335 | 423.6820 |
117
+ | 52000 | 0.8404 | 109.5515 | 563.6358 | 0.8457 | 85.2439 | 58.655 | 7.332 | 337.9521 |
118
+ | 53000 | 0.8566 | 109.7388 | 550.0908 | 0.8446 | 85.1744 | 58.703 | 7.338 | 412.3510 |
119
+ | 54000 | 0.8727 | 111.3959 | 551.1781 | 0.8453 | 85.2832 | 58.628 | 7.329 | 368.3508 |
120
+ | 55000 | 0.8889 | 111.1712 | 575.0760 | 0.8450 | 85.1663 | 58.709 | 7.339 | 283.8286 |
121
+ | 56000 | 0.9051 | 110.6545 | 557.3918 | 0.8454 | 85.4688 | 58.501 | 7.313 | 360.2272 |
122
+ | 57000 | 0.9212 | 110.4055 | 604.0854 | 0.8479 | 85.3777 | 58.563 | 7.32 | 420.6379 |
123
+ | 58000 | 0.9374 | 111.5257 | 635.6327 | 0.8466 | 85.6212 | 58.397 | 7.3 | 492.8218 |
124
+ | 59000 | 0.9535 | 109.2372 | 581.6412 | 0.8423 | 85.6034 | 58.409 | 7.301 | 366.9761 |
125
+ | 60000 | 0.9697 | 108.7379 | 565.1876 | 0.8362 | 85.2814 | 58.629 | 7.329 | 331.8257 |
126
+ | 61000 | 0.9859 | 108.9746 | 583.4484 | 0.8370 | 85.1729 | 58.704 | 7.338 | 399.4518 |
127
+ | 61875 | 1.0 | 109.4665 | 569.9899 | 0.8351 | 85.5074 | 58.474 | 7.309 | 290.9278 |
128
 
129
  ### Framework versions
130
  - Distily 0.2.0
logs/hs_loss_fn=mse, hs_weight=2.0/events.out.tfevents.1723754495.5f530b1cf724 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b095d411875b12c72e4ba66c3a4d371155ebcf13c62eb162da54e90dcb0d3b4
3
+ size 529