Hugging Face's logo

distily
/
distily_bitnet_gpt2

TensorBoard
Safetensors
Distily
gpt2
bitnet
1.58b
Generated from Trainer
Model card Files Metrics Community
lapp0 commited on
Commit
560b542
•
1 Parent(s): bc2e5d0

End of training

Browse files
Files changed (2) hide show
  1. README.md +73 -73
  2. logs/lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724181244.5f530b1cf724 +3 -0
README.md CHANGED
@@ -18,14 +18,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
18
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
19
 
20
  It achieves the following results on the evaluation set:
21
- - eval_enwikippl: 181.0
22
- - eval_frwikippl: 640.0
23
- - eval_zhwikippl: 174.0
24
- - eval_tinystoriesppl: 149.0
25
- - eval_loss: 1.1613
26
- - eval_runtime: 29.8849
27
- - eval_samples_per_second: 83.654
28
- - eval_steps_per_second: 10.474
29
 
30
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
31
  should probably proofread and complete it, then remove this comment.
@@ -55,80 +55,80 @@ The following hyperparameters were used during training:
55
  - eval_batch_size: 8
56
  - seed: 42
57
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
58
- - lr_scheduler_type: linear
59
  - lr_scheduler_warmup_ratio: 0.5
60
  - num_epochs: 1.0
61
 
62
  ### Resource Usage
63
- Peak GPU Memory: 7.5012 GB
64
 
65
  ### Eval-Phase Metrics
66
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
67
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
68
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
69
- | 0 | 0 | 1391569403904.0 | 152832116260864.0 | 21.1096 | 29.8681 | 83.701 | 10.479 | 2919235584.0 | 51951924412416.0 |
70
- | 1000 | 0.0162 | 8768.0 | 116224.0 | 3.8011 | 29.9472 | 83.48 | 10.452 | 4832.0 | 516096.0 |
71
- | 2000 | 0.0323 | 1424.0 | 9088.0 | 2.4971 | 29.9027 | 83.604 | 10.467 | 840.0 | 16512.0 |
72
- | 3000 | 0.0485 | 648.0 | 4128.0 | 1.9833 | 29.9333 | 83.519 | 10.457 | 424.0 | 1960.0 |
73
- | 4000 | 0.0646 | 438.0 | 2640.0 | 1.7696 | 29.8889 | 83.643 | 10.472 | 318.0 | 498.0 |
74
- | 5000 | 0.0808 | 332.0 | 1392.0 | 1.5375 | 29.9028 | 83.604 | 10.467 | 251.0 | 266.0 |
75
- | 6000 | 0.0970 | 258.0 | 1000.0 | 1.3761 | 29.996 | 83.344 | 10.435 | 211.0 | 188.0 |
76
- | 7000 | 0.1131 | 227.0 | 728.0 | 1.2930 | 29.952 | 83.467 | 10.45 | 179.0 | 189.0 |
77
- | 8000 | 0.1293 | 203.0 | 656.0 | 1.2303 | 29.8869 | 83.649 | 10.473 | 163.0 | 175.0 |
78
- | 9000 | 0.1455 | 181.0 | 640.0 | 1.1613 | 29.8849 | 83.654 | 10.474 | 149.0 | 174.0 |
79
- | 10000 | 0.1616 | 157.0 | 560.0 | 1.0615 | 29.9656 | 83.429 | 10.445 | 132.0 | 134.0 |
80
- | 11000 | 0.1778 | 136.0 | 502.0 | 0.9562 | 29.8901 | 83.64 | 10.472 | 108.0 | 185.0 |
81
- | 12000 | 0.1939 | 124.0 | 452.0 | 0.8822 | 29.954 | 83.461 | 10.449 | 96.5 | 155.0 |
82
- | 13000 | 0.2101 | 112.0 | 398.0 | 0.8152 | 30.0189 | 83.281 | 10.427 | 90.5 | 136.0 |
83
- | 14000 | 0.2263 | 105.0 | 404.0 | 0.7710 | 29.8719 | 83.691 | 10.478 | 78.5 | 123.0 |
84
- | 15000 | 0.2424 | 100.0 | 414.0 | 0.7459 | 29.8461 | 83.763 | 10.487 | 81.0 | 109.5 |
85
- | 16000 | 0.2586 | 97.0 | 374.0 | 0.7016 | 29.964 | 83.434 | 10.446 | 76.5 | 103.5 |
86
- | 17000 | 0.2747 | 89.0 | 328.0 | 0.6714 | 29.8419 | 83.775 | 10.489 | 71.5 | 115.0 |
87
- | 18000 | 0.2909 | 85.0 | 310.0 | 0.6407 | 29.9068 | 83.593 | 10.466 | 69.0 | 116.0 |
88
- | 19000 | 0.3071 | 87.0 | 342.0 | 0.6638 | 29.8426 | 83.773 | 10.488 | 72.5 | 194.0 |
89
- | 20000 | 0.3232 | 74.0 | 304.0 | 0.5963 | 29.879 | 83.671 | 10.476 | 64.0 | 107.0 |
90
- | 21000 | 0.3394 | 72.5 | 252.0 | 0.5502 | 29.8656 | 83.708 | 10.48 | 57.5 | 132.0 |
91
- | 22000 | 0.3556 | 70.0 | 229.0 | 0.5176 | 29.9093 | 83.586 | 10.465 | 55.0 | 115.0 |
92
- | 23000 | 0.3717 | 66.5 | 223.0 | 0.5025 | 29.8474 | 83.759 | 10.487 | 54.25 | 114.5 |
93
- | 24000 | 0.3879 | 66.5 | 222.0 | 0.4951 | 29.8676 | 83.703 | 10.48 | 51.75 | 89.5 |
94
- | 25000 | 0.4040 | 65.5 | 206.0 | 0.4796 | 29.8288 | 83.812 | 10.493 | 51.5 | 92.0 |
95
- | 26000 | 0.4202 | 65.0 | 207.0 | 0.4807 | 29.8664 | 83.706 | 10.48 | 49.0 | 91.0 |
96
- | 27000 | 0.4364 | 63.25 | 203.0 | 0.4638 | 29.8445 | 83.767 | 10.488 | 49.5 | 181.0 |
97
- | 28000 | 0.4525 | 64.5 | 208.0 | 0.4575 | 29.9002 | 83.611 | 10.468 | 46.25 | 107.0 |
98
- | 29000 | 0.4687 | 64.0 | 214.0 | 0.4679 | 29.8965 | 83.622 | 10.469 | 49.0 | 262.0 |
99
- | 30000 | 0.4848 | 64.0 | 208.0 | 0.4648 | 29.8926 | 83.633 | 10.471 | 50.75 | 83.5 |
100
- | 31000 | 0.5010 | 65.0 | 205.0 | 0.4666 | 29.8619 | 83.719 | 10.482 | 48.5 | 82.0 |
101
- | 32000 | 0.5172 | 64.0 | 193.0 | 0.4585 | 29.8674 | 83.703 | 10.48 | 48.75 | 106.5 |
102
- | 33000 | 0.5333 | 63.75 | 222.0 | 0.4543 | 29.8686 | 83.7 | 10.479 | 50.25 | 79.5 |
103
- | 34000 | 0.5495 | 61.5 | 189.0 | 0.4361 | 29.957 | 83.453 | 10.448 | 47.25 | 87.0 |
104
- | 35000 | 0.5657 | 61.75 | 180.0 | 0.4240 | 29.8662 | 83.707 | 10.48 | 45.25 | 77.5 |
105
- | 36000 | 0.5818 | 60.25 | 187.0 | 0.4193 | 29.9252 | 83.542 | 10.459 | 45.0 | 65.5 |
106
- | 37000 | 0.5980 | 61.75 | 190.0 | 0.4200 | 29.9729 | 83.409 | 10.443 | 46.75 | 103.5 |
107
- | 38000 | 0.6141 | 58.25 | 169.0 | 0.4123 | 30.0816 | 83.107 | 10.405 | 44.5 | 69.0 |
108
- | 39000 | 0.6303 | 58.0 | 178.0 | 0.4068 | 29.992 | 83.356 | 10.436 | 41.5 | 144.0 |
109
- | 40000 | 0.6465 | 59.0 | 181.0 | 0.4049 | 29.8901 | 83.64 | 10.472 | 41.25 | 130.0 |
110
- | 41000 | 0.6626 | 58.25 | 171.0 | 0.3950 | 29.8957 | 83.624 | 10.47 | 42.75 | 84.5 |
111
- | 42000 | 0.6788 | 58.0 | 188.0 | 0.3972 | 29.9022 | 83.606 | 10.467 | 42.25 | 83.5 |
112
- | 43000 | 0.6949 | 57.75 | 166.0 | 0.3874 | 29.9521 | 83.467 | 10.45 | 43.25 | 102.5 |
113
- | 44000 | 0.7111 | 55.5 | 159.0 | 0.3504 | 29.9299 | 83.529 | 10.458 | 38.5 | 48.5 |
114
- | 45000 | 0.7273 | 52.75 | 160.0 | 0.3309 | 29.8687 | 83.7 | 10.479 | 36.5 | 59.25 |
115
- | 46000 | 0.7434 | 52.25 | 146.0 | 0.3215 | 29.9781 | 83.394 | 10.441 | 36.0 | 47.75 |
116
- | 47000 | 0.7596 | 51.25 | 137.0 | 0.3141 | 29.8834 | 83.659 | 10.474 | 35.0 | 57.75 |
117
- | 48000 | 0.7758 | 52.0 | 137.0 | 0.3109 | 29.8929 | 83.632 | 10.471 | 34.5 | 43.25 |
118
- | 49000 | 0.7919 | 51.25 | 133.0 | 0.3059 | 29.8644 | 83.712 | 10.481 | 34.25 | 39.75 |
119
- | 50000 | 0.8081 | 51.5 | 136.0 | 0.3033 | 29.888 | 83.646 | 10.472 | 33.75 | 39.25 |
120
- | 51000 | 0.8242 | 50.5 | 134.0 | 0.3021 | 29.8722 | 83.69 | 10.478 | 33.0 | 53.5 |
121
- | 52000 | 0.8404 | 50.5 | 131.0 | 0.3001 | 29.894 | 83.629 | 10.47 | 34.25 | 51.25 |
122
- | 53000 | 0.8566 | 50.25 | 131.0 | 0.2951 | 29.8576 | 83.731 | 10.483 | 34.0 | 41.25 |
123
- | 54000 | 0.8727 | 50.5 | 131.0 | 0.2904 | 29.8721 | 83.69 | 10.478 | 33.25 | 36.0 |
124
- | 55000 | 0.8889 | 50.0 | 129.0 | 0.2872 | 29.8687 | 83.7 | 10.479 | 32.75 | 35.75 |
125
- | 56000 | 0.9051 | 50.25 | 128.0 | 0.2851 | 29.8853 | 83.653 | 10.473 | 32.5 | 36.75 |
126
- | 57000 | 0.9212 | 49.5 | 128.0 | 0.2845 | 29.8917 | 83.635 | 10.471 | 32.5 | 35.25 |
127
- | 58000 | 0.9374 | 49.75 | 128.0 | 0.2828 | 29.8968 | 83.621 | 10.469 | 32.75 | 34.75 |
128
- | 59000 | 0.9535 | 49.75 | 126.5 | 0.2816 | 29.8603 | 83.723 | 10.482 | 32.5 | 34.5 |
129
- | 60000 | 0.9697 | 49.75 | 127.0 | 0.2814 | 29.9969 | 83.342 | 10.434 | 32.5 | 34.5 |
130
- | 61000 | 0.9859 | 49.75 | 127.5 | 0.2812 | 29.8685 | 83.7 | 10.479 | 32.5 | 34.5 |
131
- | 61875 | 1.0 | 49.75 | 127.0 | 0.2812 | 30.0361 | 83.233 | 10.421 | 32.5 | 34.5 |
132
 
133
  ### Framework versions
134
  - Distily 0.2.0
 
18
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
19
 
20
  It achieves the following results on the evaluation set:
21
+ - eval_enwikippl: 184.0
22
+ - eval_frwikippl: 744.0
23
+ - eval_zhwikippl: 180.0
24
+ - eval_tinystoriesppl: 148.0
25
+ - eval_loss: 1.1860
26
+ - eval_runtime: 29.84
27
+ - eval_samples_per_second: 83.78
28
+ - eval_steps_per_second: 10.489
29
 
30
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
31
  should probably proofread and complete it, then remove this comment.
 
55
  - eval_batch_size: 8
56
  - seed: 42
57
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
58
+ - lr_scheduler_type: cosine
59
  - lr_scheduler_warmup_ratio: 0.5
60
  - num_epochs: 1.0
61
 
62
  ### Resource Usage
63
+ Peak GPU Memory: 7.5008 GB
64
 
65
  ### Eval-Phase Metrics
66
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
67
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
68
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
69
+ | 0 | 0 | 841813590016.0 | 42880953483264.0 | 19.1388 | 29.7619 | 84.0 | 10.517 | 2533359616.0 | 18691697672192.0 |
70
+ | 1000 | 0.0162 | 8256.0 | 104448.0 | 3.7570 | 29.792 | 83.915 | 10.506 | 4608.0 | 250880.0 |
71
+ | 2000 | 0.0323 | 1488.0 | 8576.0 | 2.5011 | 29.7856 | 83.933 | 10.508 | 828.0 | 40448.0 |
72
+ | 3000 | 0.0485 | 668.0 | 4384.0 | 2.0176 | 29.8589 | 83.727 | 10.483 | 442.0 | 1648.0 |
73
+ | 4000 | 0.0646 | 444.0 | 2304.0 | 1.7541 | 29.853 | 83.744 | 10.485 | 308.0 | 672.0 |
74
+ | 5000 | 0.0808 | 328.0 | 1288.0 | 1.5507 | 29.8272 | 83.816 | 10.494 | 258.0 | 242.0 |
75
+ | 6000 | 0.0970 | 266.0 | 1168.0 | 1.3948 | 29.9048 | 83.599 | 10.467 | 217.0 | 253.0 |
76
+ | 7000 | 0.1131 | 229.0 | 1048.0 | 1.3140 | 29.8053 | 83.878 | 10.501 | 181.0 | 189.0 |
77
+ | 8000 | 0.1293 | 202.0 | 760.0 | 1.2384 | 29.8461 | 83.763 | 10.487 | 166.0 | 187.0 |
78
+ | 9000 | 0.1455 | 184.0 | 744.0 | 1.1860 | 29.84 | 83.78 | 10.489 | 148.0 | 180.0 |
79
+ | 10000 | 0.1616 | 161.0 | 564.0 | 1.0820 | 29.8521 | 83.746 | 10.485 | 132.0 | 170.0 |
80
+ | 11000 | 0.1778 | 139.0 | 478.0 | 0.9691 | 29.7904 | 83.92 | 10.507 | 112.5 | 139.0 |
81
+ | 12000 | 0.1939 | 122.5 | 446.0 | 0.8903 | 29.8277 | 83.815 | 10.494 | 91.0 | 153.0 |
82
+ | 13000 | 0.2101 | 130.0 | 450.0 | 0.8290 | 29.8764 | 83.678 | 10.476 | 113.5 | 148.0 |
83
+ | 14000 | 0.2263 | 111.5 | 410.0 | 0.7867 | 29.8386 | 83.784 | 10.49 | 85.5 | 116.5 |
84
+ | 15000 | 0.2424 | 103.0 | 394.0 | 0.7550 | 29.7824 | 83.942 | 10.51 | 81.5 | 126.5 |
85
+ | 16000 | 0.2586 | 95.5 | 368.0 | 0.7130 | 29.8395 | 83.781 | 10.489 | 74.0 | 137.0 |
86
+ | 17000 | 0.2747 | 91.0 | 370.0 | 0.6869 | 29.8002 | 83.892 | 10.503 | 72.0 | 110.0 |
87
+ | 18000 | 0.2909 | 89.5 | 356.0 | 0.6569 | 29.8522 | 83.746 | 10.485 | 65.0 | 124.5 |
88
+ | 19000 | 0.3071 | 87.0 | 354.0 | 0.6839 | 29.8823 | 83.661 | 10.474 | 68.5 | 137.0 |
89
+ | 20000 | 0.3232 | 79.5 | 290.0 | 0.6065 | 29.8977 | 83.618 | 10.469 | 65.0 | 113.5 |
90
+ | 21000 | 0.3394 | 75.0 | 251.0 | 0.5674 | 29.8207 | 83.834 | 10.496 | 59.75 | 112.5 |
91
+ | 22000 | 0.3556 | 70.0 | 250.0 | 0.5363 | 29.8336 | 83.798 | 10.492 | 56.25 | 81.0 |
92
+ | 23000 | 0.3717 | 69.0 | 220.0 | 0.5125 | 29.8003 | 83.892 | 10.503 | 53.75 | 86.5 |
93
+ | 24000 | 0.3879 | 65.5 | 226.0 | 0.5047 | 29.8312 | 83.805 | 10.492 | 52.25 | 91.0 |
94
+ | 25000 | 0.4040 | 65.5 | 211.0 | 0.4917 | 29.8281 | 83.813 | 10.493 | 55.0 | 141.0 |
95
+ | 26000 | 0.4202 | 63.25 | 204.0 | 0.4817 | 29.8227 | 83.829 | 10.495 | 50.75 | 86.5 |
96
+ | 27000 | 0.4364 | 64.5 | 213.0 | 0.4738 | 29.9242 | 83.544 | 10.46 | 51.25 | 94.5 |
97
+ | 28000 | 0.4525 | 62.75 | 192.0 | 0.4619 | 29.9106 | 83.583 | 10.465 | 48.75 | 113.5 |
98
+ | 29000 | 0.4687 | 64.5 | 204.0 | 0.4840 | 29.8026 | 83.885 | 10.502 | 52.5 | 81.5 |
99
+ | 30000 | 0.4848 | 65.0 | 217.0 | 0.4796 | 29.8897 | 83.641 | 10.472 | 49.25 | 140.0 |
100
+ | 31000 | 0.5010 | 63.5 | 206.0 | 0.4689 | 29.8072 | 83.872 | 10.501 | 48.25 | 141.0 |
101
+ | 32000 | 0.5172 | 63.25 | 217.0 | 0.4726 | 29.8682 | 83.701 | 10.479 | 46.25 | 112.5 |
102
+ | 33000 | 0.5333 | 66.5 | 231.0 | 0.4654 | 29.7912 | 83.917 | 10.506 | 51.25 | 87.5 |
103
+ | 34000 | 0.5495 | 62.75 | 200.0 | 0.4547 | 29.8255 | 83.821 | 10.494 | 49.75 | 89.5 |
104
+ | 35000 | 0.5657 | 63.75 | 196.0 | 0.4552 | 29.8185 | 83.841 | 10.497 | 49.25 | 83.5 |
105
+ | 36000 | 0.5818 | 63.75 | 215.0 | 0.4588 | 29.8868 | 83.649 | 10.473 | 46.0 | 113.5 |
106
+ | 37000 | 0.5980 | 61.5 | 193.0 | 0.4382 | 29.825 | 83.822 | 10.495 | 46.25 | 130.0 |
107
+ | 38000 | 0.6141 | 61.5 | 193.0 | 0.4237 | 29.8213 | 83.833 | 10.496 | 45.75 | 75.5 |
108
+ | 39000 | 0.6303 | 61.5 | 187.0 | 0.4218 | 29.8194 | 83.838 | 10.497 | 44.0 | 82.5 |
109
+ | 40000 | 0.6465 | 59.75 | 178.0 | 0.4127 | 29.8348 | 83.795 | 10.491 | 42.75 | 100.5 |
110
+ | 41000 | 0.6626 | 58.0 | 184.0 | 0.4133 | 29.778 | 83.955 | 10.511 | 42.25 | 119.0 |
111
+ | 42000 | 0.6788 | 56.75 | 184.0 | 0.4072 | 29.8696 | 83.697 | 10.479 | 40.75 | 109.0 |
112
+ | 43000 | 0.6949 | 57.75 | 184.0 | 0.3986 | 29.8393 | 83.782 | 10.49 | 41.75 | 87.0 |
113
+ | 44000 | 0.7111 | 58.0 | 180.0 | 0.4014 | 29.8433 | 83.771 | 10.488 | 40.5 | 101.0 |
114
+ | 45000 | 0.7273 | 55.75 | 158.0 | 0.3611 | 29.8497 | 83.753 | 10.486 | 38.25 | 67.0 |
115
+ | 46000 | 0.7434 | 55.0 | 148.0 | 0.3377 | 29.8619 | 83.719 | 10.482 | 36.0 | 63.75 |
116
+ | 47000 | 0.7596 | 52.25 | 143.0 | 0.3271 | 29.8199 | 83.837 | 10.496 | 35.0 | 50.75 |
117
+ | 48000 | 0.7758 | 52.0 | 141.0 | 0.3185 | 29.8125 | 83.857 | 10.499 | 34.0 | 49.25 |
118
+ | 49000 | 0.7919 | 52.5 | 142.0 | 0.3146 | 29.9037 | 83.602 | 10.467 | 33.5 | 43.5 |
119
+ | 50000 | 0.8081 | 51.25 | 134.0 | 0.3096 | 29.8931 | 83.631 | 10.471 | 33.25 | 46.25 |
120
+ | 51000 | 0.8242 | 51.25 | 133.0 | 0.3025 | 30.0212 | 83.274 | 10.426 | 32.5 | 40.0 |
121
+ | 52000 | 0.8404 | 51.5 | 132.0 | 0.2984 | 29.8459 | 83.764 | 10.487 | 32.5 | 39.5 |
122
+ | 53000 | 0.8566 | 50.5 | 131.0 | 0.2951 | 29.8292 | 83.81 | 10.493 | 32.5 | 36.0 |
123
+ | 54000 | 0.8727 | 50.5 | 132.0 | 0.2934 | 29.9146 | 83.571 | 10.463 | 32.25 | 37.75 |
124
+ | 55000 | 0.8889 | 50.0 | 131.0 | 0.2918 | 29.8217 | 83.831 | 10.496 | 32.5 | 35.75 |
125
+ | 56000 | 0.9051 | 50.0 | 130.0 | 0.2911 | 29.8366 | 83.79 | 10.49 | 32.25 | 35.5 |
126
+ | 57000 | 0.9212 | 50.0 | 130.0 | 0.2903 | 29.8261 | 83.819 | 10.494 | 32.25 | 35.5 |
127
+ | 58000 | 0.9374 | 50.0 | 130.0 | 0.2901 | 29.8639 | 83.713 | 10.481 | 32.25 | 35.5 |
128
+ | 59000 | 0.9535 | 50.0 | 130.0 | 0.2900 | 29.8256 | 83.821 | 10.494 | 32.25 | 35.25 |
129
+ | 60000 | 0.9697 | 50.0 | 130.0 | 0.2899 | 29.8767 | 83.677 | 10.476 | 32.25 | 35.25 |
130
+ | 61000 | 0.9859 | 50.0 | 130.0 | 0.2900 | 29.8188 | 83.84 | 10.497 | 32.25 | 35.25 |
131
+ | 61875 | 1.0 | 50.0 | 130.0 | 0.2899 | 29.869 | 83.699 | 10.479 | 32.25 | 35.25 |
132
 
133
  ### Framework versions
134
  - Distily 0.2.0
logs/lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724181244.5f530b1cf724 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8b2a5514ca6b1c6850da55776e8f5fb3de0a74575b0b252791613944102570b2
3
+ size 312