Hugging Face's logo

distily
/
distily_bitnet_gpt2

TensorBoard
Safetensors
Distily
gpt2
bitnet
1.58b
Generated from Trainer
Model card Files Metrics Community
lapp0 commited on
Commit
2ba6a2b
•
1 Parent(s): 499676b

End of training

Browse files
Files changed (2) hide show
  1. README.md +73 -73
  2. logs/attn_layer_mapper=layer-2, attn_loss_fn=cos, attn_weight=1.0, lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724212603.5f530b1cf724 +3 -0
README.md CHANGED
@@ -18,14 +18,14 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
18
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
19
 
20
  It achieves the following results on the evaluation set:
21
- - eval_enwikippl: 181.0
22
- - eval_frwikippl: 640.0
23
- - eval_zhwikippl: 173.0
24
- - eval_tinystoriesppl: 149.0
25
- - eval_loss: 1.1615
26
- - eval_runtime: 30.3849
27
- - eval_samples_per_second: 82.278
28
- - eval_steps_per_second: 10.301
29
 
30
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
31
  should probably proofread and complete it, then remove this comment.
@@ -48,7 +48,7 @@ More information needed
48
  ### Training hyperparameters
49
 
50
  The following hyperparameters were used during training:
51
- - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=1.0, loss_fn=mse, layer_mapper=last, projector=None))
52
  - train_embeddings: True
53
  - learning_rate: 0.0001
54
  - train_batch_size: 4
@@ -60,75 +60,75 @@ The following hyperparameters were used during training:
60
  - num_epochs: 1.0
61
 
62
  ### Resource Usage
63
- Peak GPU Memory: 7.7843 GB
64
 
65
  ### Eval-Phase Metrics
66
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
67
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
68
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
69
- | 0 | 0 | 1391569403904.0 | 152832116260864.0 | 21.1096 | 30.3778 | 82.297 | 10.304 | 2919235584.0 | 51951924412416.0 |
70
- | 1000 | 0.0162 | 8768.0 | 116224.0 | 3.8006 | 30.3959 | 82.248 | 10.297 | 4832.0 | 516096.0 |
71
- | 2000 | 0.0323 | 1424.0 | 9088.0 | 2.4969 | 30.3684 | 82.322 | 10.307 | 840.0 | 16512.0 |
72
- | 3000 | 0.0485 | 648.0 | 4096.0 | 1.9833 | 30.3429 | 82.392 | 10.315 | 424.0 | 1960.0 |
73
- | 4000 | 0.0646 | 438.0 | 2640.0 | 1.7698 | 30.4625 | 82.068 | 10.275 | 318.0 | 496.0 |
74
- | 5000 | 0.0808 | 332.0 | 1392.0 | 1.5383 | 30.3813 | 82.287 | 10.302 | 250.0 | 266.0 |
75
- | 6000 | 0.0970 | 258.0 | 1000.0 | 1.3764 | 30.3609 | 82.343 | 10.309 | 211.0 | 188.0 |
76
- | 7000 | 0.1131 | 227.0 | 728.0 | 1.2934 | 30.3509 | 82.37 | 10.313 | 179.0 | 190.0 |
77
- | 8000 | 0.1293 | 203.0 | 656.0 | 1.2303 | 30.3602 | 82.345 | 10.31 | 163.0 | 175.0 |
78
- | 9000 | 0.1455 | 181.0 | 640.0 | 1.1615 | 30.3849 | 82.278 | 10.301 | 149.0 | 173.0 |
79
- | 10000 | 0.1616 | 157.0 | 556.0 | 1.0616 | 30.3811 | 82.288 | 10.302 | 132.0 | 134.0 |
80
- | 11000 | 0.1778 | 136.0 | 502.0 | 0.9560 | 30.3896 | 82.265 | 10.3 | 108.5 | 184.0 |
81
- | 12000 | 0.1939 | 123.5 | 450.0 | 0.8820 | 30.3801 | 82.291 | 10.303 | 96.0 | 155.0 |
82
- | 13000 | 0.2101 | 112.0 | 398.0 | 0.8150 | 30.4027 | 82.23 | 10.295 | 90.5 | 137.0 |
83
- | 14000 | 0.2263 | 105.0 | 402.0 | 0.7710 | 30.3773 | 82.298 | 10.304 | 78.5 | 122.0 |
84
- | 15000 | 0.2424 | 100.0 | 410.0 | 0.7460 | 30.383 | 82.283 | 10.302 | 81.0 | 108.0 |
85
- | 16000 | 0.2586 | 96.5 | 370.0 | 0.7014 | 30.3917 | 82.259 | 10.299 | 76.0 | 104.0 |
86
- | 17000 | 0.2747 | 88.5 | 326.0 | 0.6711 | 30.3819 | 82.286 | 10.302 | 71.0 | 114.5 |
87
- | 18000 | 0.2909 | 85.0 | 308.0 | 0.6404 | 30.343 | 82.391 | 10.315 | 68.5 | 115.5 |
88
- | 19000 | 0.3071 | 86.5 | 336.0 | 0.6613 | 30.3361 | 82.41 | 10.318 | 72.5 | 238.0 |
89
- | 20000 | 0.3232 | 73.5 | 296.0 | 0.5964 | 30.3436 | 82.39 | 10.315 | 64.0 | 115.5 |
90
- | 21000 | 0.3394 | 74.0 | 251.0 | 0.5524 | 30.3214 | 82.45 | 10.323 | 58.25 | 112.5 |
91
- | 22000 | 0.3556 | 70.0 | 229.0 | 0.5193 | 30.3514 | 82.369 | 10.313 | 55.0 | 95.0 |
92
- | 23000 | 0.3717 | 67.5 | 220.0 | 0.5032 | 30.3833 | 82.282 | 10.302 | 52.75 | 113.5 |
93
- | 24000 | 0.3879 | 66.5 | 221.0 | 0.4939 | 30.3276 | 82.433 | 10.321 | 52.75 | 85.0 |
94
- | 25000 | 0.4040 | 64.5 | 209.0 | 0.4789 | 30.3465 | 82.382 | 10.314 | 51.5 | 109.0 |
95
- | 26000 | 0.4202 | 64.5 | 198.0 | 0.4777 | 30.3465 | 82.382 | 10.314 | 48.75 | 99.5 |
96
- | 27000 | 0.4364 | 62.5 | 211.0 | 0.4654 | 30.3557 | 82.357 | 10.311 | 49.75 | 148.0 |
97
- | 28000 | 0.4525 | 63.75 | 205.0 | 0.4571 | 30.4463 | 82.112 | 10.28 | 46.5 | 75.0 |
98
- | 29000 | 0.4687 | 64.5 | 214.0 | 0.4744 | 30.3393 | 82.401 | 10.317 | 49.75 | 128.0 |
99
- | 30000 | 0.4848 | 63.5 | 213.0 | 0.4648 | 30.3973 | 82.244 | 10.297 | 50.25 | 91.0 |
100
- | 31000 | 0.5010 | 63.5 | 203.0 | 0.4676 | 30.3762 | 82.301 | 10.304 | 47.25 | 70.0 |
101
- | 32000 | 0.5172 | 63.0 | 191.0 | 0.4571 | 30.3628 | 82.338 | 10.309 | 48.5 | 90.0 |
102
- | 33000 | 0.5333 | 65.0 | 224.0 | 0.4568 | 30.3525 | 82.365 | 10.312 | 48.0 | 78.5 |
103
- | 34000 | 0.5495 | 63.0 | 206.0 | 0.4476 | 30.3653 | 82.331 | 10.308 | 50.5 | 132.0 |
104
- | 35000 | 0.5657 | 65.0 | 193.0 | 0.4482 | 30.365 | 82.332 | 10.308 | 46.5 | 84.0 |
105
- | 36000 | 0.5818 | 63.75 | 207.0 | 0.4406 | 30.3594 | 82.347 | 10.31 | 47.0 | 108.0 |
106
- | 37000 | 0.5980 | 64.0 | 206.0 | 0.4353 | 30.3617 | 82.341 | 10.309 | 49.0 | 149.0 |
107
- | 38000 | 0.6141 | 58.25 | 175.0 | 0.4183 | 30.3839 | 82.28 | 10.301 | 45.75 | 78.5 |
108
- | 39000 | 0.6303 | 58.75 | 179.0 | 0.4164 | 30.368 | 82.324 | 10.307 | 43.0 | 128.0 |
109
- | 40000 | 0.6465 | 59.0 | 190.0 | 0.4120 | 30.3866 | 82.273 | 10.301 | 41.75 | 91.5 |
110
- | 41000 | 0.6626 | 57.25 | 177.0 | 0.4045 | 30.3663 | 82.328 | 10.307 | 42.75 | 79.0 |
111
- | 42000 | 0.6788 | 57.75 | 200.0 | 0.3997 | 30.3611 | 82.342 | 10.309 | 42.25 | 93.0 |
112
- | 43000 | 0.6949 | 57.25 | 164.0 | 0.3936 | 30.3791 | 82.293 | 10.303 | 41.5 | 61.0 |
113
- | 44000 | 0.7111 | 58.5 | 184.0 | 0.3924 | 30.3499 | 82.372 | 10.313 | 41.75 | 79.0 |
114
- | 45000 | 0.7273 | 55.0 | 169.0 | 0.3546 | 30.4012 | 82.234 | 10.296 | 37.75 | 83.5 |
115
- | 46000 | 0.7434 | 53.5 | 150.0 | 0.3326 | 30.3486 | 82.376 | 10.313 | 36.25 | 48.5 |
116
- | 47000 | 0.7596 | 52.25 | 138.0 | 0.3211 | 30.3762 | 82.301 | 10.304 | 35.0 | 43.5 |
117
- | 48000 | 0.7758 | 52.5 | 138.0 | 0.3136 | 30.3711 | 82.315 | 10.306 | 34.5 | 46.75 |
118
- | 49000 | 0.7919 | 51.75 | 134.0 | 0.3075 | 30.3569 | 82.354 | 10.311 | 34.25 | 39.25 |
119
- | 50000 | 0.8081 | 52.0 | 138.0 | 0.3038 | 30.3804 | 82.29 | 10.303 | 33.75 | 38.5 |
120
- | 51000 | 0.8242 | 50.5 | 135.0 | 0.2979 | 30.3749 | 82.305 | 10.305 | 33.0 | 39.5 |
121
- | 52000 | 0.8404 | 50.0 | 132.0 | 0.2923 | 30.3713 | 82.315 | 10.306 | 33.0 | 37.25 |
122
- | 53000 | 0.8566 | 50.0 | 134.0 | 0.2901 | 30.3482 | 82.377 | 10.314 | 32.75 | 37.75 |
123
- | 54000 | 0.8727 | 50.0 | 136.0 | 0.2885 | 30.3375 | 82.406 | 10.317 | 32.5 | 37.25 |
124
- | 55000 | 0.8889 | 50.0 | 134.0 | 0.2862 | 30.3588 | 82.348 | 10.31 | 32.5 | 36.0 |
125
- | 56000 | 0.9051 | 50.0 | 133.0 | 0.2854 | 30.362 | 82.34 | 10.309 | 32.25 | 36.0 |
126
- | 57000 | 0.9212 | 50.0 | 134.0 | 0.2852 | 30.3411 | 82.396 | 10.316 | 32.25 | 36.25 |
127
- | 58000 | 0.9374 | 50.0 | 133.0 | 0.2848 | 30.3519 | 82.367 | 10.312 | 32.25 | 36.0 |
128
- | 59000 | 0.9535 | 50.0 | 133.0 | 0.2846 | 30.3634 | 82.336 | 10.308 | 32.25 | 36.0 |
129
- | 60000 | 0.9697 | 50.0 | 133.0 | 0.2845 | 30.3315 | 82.423 | 10.319 | 32.25 | 36.0 |
130
- | 61000 | 0.9859 | 50.0 | 133.0 | 0.2846 | 30.4113 | 82.206 | 10.292 | 32.25 | 36.0 |
131
- | 61875 | 1.0 | 50.0 | 133.0 | 0.2845 | 30.3935 | 82.254 | 10.298 | 32.25 | 36.0 |
132
 
133
  ### Framework versions
134
  - Distily 0.2.0
 
18
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
19
 
20
  It achieves the following results on the evaluation set:
21
+ - eval_enwikippl: 179.0
22
+ - eval_frwikippl: 624.0
23
+ - eval_zhwikippl: 166.0
24
+ - eval_tinystoriesppl: 159.0
25
+ - eval_loss: 1.8254
26
+ - eval_runtime: 30.606
27
+ - eval_samples_per_second: 81.683
28
+ - eval_steps_per_second: 10.227
29
 
30
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
31
  should probably proofread and complete it, then remove this comment.
 
48
  ### Training hyperparameters
49
 
50
  The following hyperparameters were used during training:
51
+ - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=1.0, loss_fn=cos, layer_mapper=layer-2, projector=None))
52
  - train_embeddings: True
53
  - learning_rate: 0.0001
54
  - train_batch_size: 4
 
60
  - num_epochs: 1.0
61
 
62
  ### Resource Usage
63
+ Peak GPU Memory: 7.7840 GB
64
 
65
  ### Eval-Phase Metrics
66
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
67
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
68
  | **teacher eval** | | 43.25 | 61.25 | | | | | 11.6875 | 19.125 |
69
+ | 0 | 0 | 1082331758592.0 | 57174604644352.0 | 19.3008 | 30.4471 | 82.11 | 10.28 | 5268045824.0 | 25013889531904.0 |
70
+ | 1000 | 0.0162 | 8320.0 | 67072.0 | 4.7159 | 30.4361 | 82.139 | 10.284 | 4640.0 | 399360.0 |
71
+ | 2000 | 0.0323 | 1352.0 | 6688.0 | 3.3630 | 30.4661 | 82.058 | 10.274 | 856.0 | 42240.0 |
72
+ | 3000 | 0.0485 | 644.0 | 4192.0 | 2.7949 | 30.5076 | 81.947 | 10.26 | 410.0 | 2160.0 |
73
+ | 4000 | 0.0646 | 452.0 | 2672.0 | 2.5266 | 30.4909 | 81.992 | 10.265 | 308.0 | 592.0 |
74
+ | 5000 | 0.0808 | 348.0 | 1504.0 | 2.2655 | 30.4813 | 82.017 | 10.269 | 262.0 | 288.0 |
75
+ | 6000 | 0.0970 | 270.0 | 1072.0 | 2.0833 | 30.4751 | 82.034 | 10.271 | 223.0 | 225.0 |
76
+ | 7000 | 0.1131 | 226.0 | 944.0 | 1.9806 | 30.4982 | 81.972 | 10.263 | 189.0 | 182.0 |
77
+ | 8000 | 0.1293 | 198.0 | 752.0 | 1.8914 | 30.4573 | 82.082 | 10.277 | 172.0 | 159.0 |
78
+ | 9000 | 0.1455 | 179.0 | 624.0 | 1.8254 | 30.606 | 81.683 | 10.227 | 159.0 | 166.0 |
79
+ | 10000 | 0.1616 | 172.0 | 568.0 | 1.7317 | 30.4969 | 81.976 | 10.263 | 138.0 | 141.0 |
80
+ | 11000 | 0.1778 | 139.0 | 486.0 | 1.6030 | 30.4606 | 82.073 | 10.276 | 111.5 | 147.0 |
81
+ | 12000 | 0.1939 | 124.5 | 460.0 | 1.5229 | 30.4525 | 82.095 | 10.278 | 98.0 | 120.0 |
82
+ | 13000 | 0.2101 | 112.0 | 432.0 | 1.4591 | 30.4583 | 82.079 | 10.276 | 87.0 | 116.0 |
83
+ | 14000 | 0.2263 | 102.5 | 400.0 | 1.4049 | 30.4283 | 82.16 | 10.286 | 83.0 | 110.5 |
84
+ | 15000 | 0.2424 | 102.5 | 408.0 | 1.3847 | 30.4854 | 82.007 | 10.267 | 82.5 | 162.0 |
85
+ | 16000 | 0.2586 | 94.5 | 382.0 | 1.3539 | 30.5028 | 81.96 | 10.261 | 73.5 | 120.5 |
86
+ | 17000 | 0.2747 | 86.5 | 330.0 | 1.3206 | 30.5072 | 81.948 | 10.26 | 72.0 | 125.5 |
87
+ | 18000 | 0.2909 | 86.5 | 314.0 | 1.2763 | 30.4803 | 82.02 | 10.269 | 67.5 | 121.5 |
88
+ | 19000 | 0.3071 | 89.0 | 342.0 | 1.2988 | 30.498 | 81.973 | 10.263 | 74.0 | 130.0 |
89
+ | 20000 | 0.3232 | 78.0 | 310.0 | 1.2192 | 30.473 | 82.04 | 10.271 | 60.75 | 106.5 |
90
+ | 21000 | 0.3394 | 75.5 | 284.0 | 1.1612 | 30.4776 | 82.028 | 10.27 | 56.75 | 121.0 |
91
+ | 22000 | 0.3556 | 73.5 | 233.0 | 1.1233 | 30.4472 | 82.109 | 10.28 | 57.5 | 124.0 |
92
+ | 23000 | 0.3717 | 68.0 | 231.0 | 1.0997 | 30.4635 | 82.065 | 10.275 | 55.75 | 93.5 |
93
+ | 24000 | 0.3879 | 65.5 | 213.0 | 1.0867 | 30.4576 | 82.081 | 10.277 | 52.0 | 102.0 |
94
+ | 25000 | 0.4040 | 63.0 | 207.0 | 1.0647 | 30.4272 | 82.163 | 10.287 | 50.25 | 109.0 |
95
+ | 26000 | 0.4202 | 65.0 | 197.0 | 1.0463 | 30.5134 | 81.931 | 10.258 | 52.5 | 98.5 |
96
+ | 27000 | 0.4364 | 65.5 | 211.0 | 1.0503 | 30.4982 | 81.972 | 10.263 | 48.75 | 97.0 |
97
+ | 28000 | 0.4525 | 62.25 | 199.0 | 1.0304 | 30.4658 | 82.059 | 10.274 | 47.5 | 104.5 |
98
+ | 29000 | 0.4687 | 66.0 | 214.0 | 1.0459 | 30.4838 | 82.011 | 10.268 | 49.75 | 94.0 |
99
+ | 30000 | 0.4848 | 63.0 | 209.0 | 1.0378 | 30.4635 | 82.065 | 10.275 | 49.75 | 90.5 |
100
+ | 31000 | 0.5010 | 63.5 | 218.0 | 1.0314 | 30.5768 | 81.761 | 10.237 | 48.5 | 96.5 |
101
+ | 32000 | 0.5172 | 62.25 | 204.0 | 1.0225 | 30.4582 | 82.08 | 10.276 | 45.5 | 68.5 |
102
+ | 33000 | 0.5333 | 64.5 | 202.0 | 1.0209 | 30.4807 | 82.019 | 10.269 | 48.0 | 93.5 |
103
+ | 34000 | 0.5495 | 64.0 | 196.0 | 1.0165 | 30.4726 | 82.041 | 10.272 | 47.0 | 102.0 |
104
+ | 35000 | 0.5657 | 64.5 | 210.0 | 1.0109 | 30.4775 | 82.028 | 10.27 | 45.5 | 97.5 |
105
+ | 36000 | 0.5818 | 64.0 | 188.0 | 1.0084 | 30.4363 | 82.139 | 10.284 | 45.5 | 96.0 |
106
+ | 37000 | 0.5980 | 60.75 | 184.0 | 0.9871 | 30.479 | 82.024 | 10.269 | 44.25 | 77.5 |
107
+ | 38000 | 0.6141 | 59.75 | 190.0 | 0.9688 | 30.5442 | 81.849 | 10.247 | 43.5 | 74.0 |
108
+ | 39000 | 0.6303 | 59.0 | 172.0 | 0.9639 | 30.551 | 81.83 | 10.245 | 43.0 | 61.25 |
109
+ | 40000 | 0.6465 | 58.25 | 172.0 | 0.9537 | 30.489 | 81.997 | 10.266 | 42.0 | 69.0 |
110
+ | 41000 | 0.6626 | 60.5 | 176.0 | 0.9559 | 30.4953 | 81.98 | 10.264 | 41.0 | 60.5 |
111
+ | 42000 | 0.6788 | 57.75 | 184.0 | 0.9516 | 30.488 | 81.999 | 10.266 | 40.5 | 56.5 |
112
+ | 43000 | 0.6949 | 57.25 | 191.0 | 0.9369 | 30.4691 | 82.05 | 10.273 | 41.0 | 53.25 |
113
+ | 44000 | 0.7111 | 58.0 | 178.0 | 0.9370 | 30.4852 | 82.007 | 10.267 | 39.5 | 54.25 |
114
+ | 45000 | 0.7273 | 54.75 | 157.0 | 0.8904 | 30.4948 | 81.981 | 10.264 | 36.75 | 64.5 |
115
+ | 46000 | 0.7434 | 52.0 | 148.0 | 0.8656 | 30.5027 | 81.96 | 10.261 | 35.0 | 49.25 |
116
+ | 47000 | 0.7596 | 53.0 | 144.0 | 0.8540 | 30.4803 | 82.02 | 10.269 | 34.25 | 44.0 |
117
+ | 48000 | 0.7758 | 51.25 | 139.0 | 0.8430 | 30.452 | 82.097 | 10.278 | 33.0 | 44.25 |
118
+ | 49000 | 0.7919 | 52.0 | 146.0 | 0.8397 | 30.4747 | 82.035 | 10.271 | 33.5 | 41.75 |
119
+ | 50000 | 0.8081 | 51.75 | 140.0 | 0.8340 | 30.4359 | 82.14 | 10.284 | 33.5 | 40.0 |
120
+ | 51000 | 0.8242 | 50.25 | 136.0 | 0.8262 | 30.4828 | 82.013 | 10.268 | 32.5 | 40.25 |
121
+ | 52000 | 0.8404 | 50.5 | 138.0 | 0.8209 | 30.4471 | 82.11 | 10.28 | 32.25 | 37.5 |
122
+ | 53000 | 0.8566 | 50.25 | 140.0 | 0.8173 | 30.4685 | 82.052 | 10.273 | 32.0 | 36.0 |
123
+ | 54000 | 0.8727 | 49.75 | 136.0 | 0.8153 | 30.4908 | 81.992 | 10.265 | 31.875 | 36.5 |
124
+ | 55000 | 0.8889 | 50.0 | 138.0 | 0.8132 | 30.4798 | 82.022 | 10.269 | 31.75 | 36.75 |
125
+ | 56000 | 0.9051 | 50.0 | 136.0 | 0.8123 | 30.4915 | 81.99 | 10.265 | 31.75 | 36.5 |
126
+ | 57000 | 0.9212 | 50.0 | 135.0 | 0.8117 | 30.5178 | 81.919 | 10.256 | 31.75 | 36.25 |
127
+ | 58000 | 0.9374 | 49.75 | 134.0 | 0.8113 | 30.4452 | 82.115 | 10.281 | 31.625 | 36.25 |
128
+ | 59000 | 0.9535 | 50.0 | 134.0 | 0.8111 | 30.5075 | 81.947 | 10.26 | 31.75 | 36.25 |
129
+ | 60000 | 0.9697 | 50.0 | 134.0 | 0.8112 | 30.4961 | 81.978 | 10.264 | 31.75 | 36.25 |
130
+ | 61000 | 0.9859 | 50.0 | 134.0 | 0.8112 | 30.5439 | 81.849 | 10.248 | 31.75 | 36.25 |
131
+ | 61875 | 1.0 | 50.0 | 134.0 | 0.8112 | 30.6307 | 81.618 | 10.219 | 31.75 | 36.25 |
132
 
133
  ### Framework versions
134
  - Distily 0.2.0
logs/attn_layer_mapper=layer-2, attn_loss_fn=cos, attn_weight=1.0, lr_scheduler_type=cosine, warmup_ratio=0.5/events.out.tfevents.1724212603.5f530b1cf724 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1498dffc28a5386a4728bb0994dfc41e9fccc77b03f8fe7a5d71860a11615e78
3
+ size 312