Hugging Face's logo

lapp0
/
distily_bench_obj_cross

TensorBoard
Safetensors
Distily
gpt_neo
Generated from Trainer
8-bit precision
bitsandbytes
Model card Files Metrics Community
lapp0 commited on
Commit
4eed482
·
verified ·
1 Parent(s): a260409

Training in progress, step 61875

Browse files
Files changed (5) hide show
  1. README.md +32 -33
  2. logs/attn_loss_fn=mse, attn_weight=10.0, hs_loss_fn=raw_mse, hs_weight=10.0, learning_rate=0.001, warmup_ratio=0.1/events.out.tfevents.1723783997.5f530b1cf724 +3 -0
  3. logs/attn_loss_fn=mse, attn_weight=10.0, hs_loss_fn=raw_mse, hs_weight=10.0, learning_rate=0.001, warmup_ratio=0/events.out.tfevents.1723783790.5f530b1cf724 +2 -2
  4. model.safetensors +1 -1
  5. training_args.bin +1 -1
README.md CHANGED
@@ -15,14 +15,14 @@ This student model is distilled from the teacher model [roneneldan/TinyStories-3
15
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
16
 
17
  It achieves the following results on the evaluation set:
18
- - eval_enwikippl: 3944.9531
19
- - eval_frwikippl: 30197.7344
20
- - eval_zhwikippl: 52496.3438
21
- - eval_tinystoriesppl: 1385.5492
22
- - eval_loss: 16.6107
23
- - eval_runtime: 66.9937
24
- - eval_samples_per_second: 74.634
25
- - eval_steps_per_second: 9.329
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
@@ -47,44 +47,43 @@ More information needed
47
  The following hyperparameters were used during training:
48
  - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10.0, loss_fn=raw_mse, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=mse, layer_mapper=None, projector=None))
49
  - train_embeddings: True
50
- - learning_rate: 0.0004
51
  - train_batch_size: 8
52
  - eval_batch_size: 8
53
  - seed: 42
54
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
55
  - lr_scheduler_type: constant
56
- - lr_scheduler_warmup_ratio: 0.1
57
  - num_epochs: 1.0
58
 
59
  ### Resource Usage
60
- Peak GPU Memory: 8.2677 GB
61
 
62
  ### Eval-Phase Metrics
63
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
64
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
65
  | **teacher eval** | | 169.9865 | 47377.9414 | | | | | 3.9789 | 4998.1294 |
66
- | 0 | 0 | 21397.4785 | 57946.0117 | 18.3162 | 67.1093 | 74.505 | 9.313 | 12321.8145 | 60955.8008 |
67
- | 3000 | 0.0485 | 3940.0654 | 30197.7344 | 16.6107 | 67.006 | 74.62 | 9.328 | 1383.7178 | 52496.3438 |
68
- | 6000 | 0.0970 | 3937.6238 | 30180.7188 | 16.6119 | 67.1095 | 74.505 | 9.313 | 1383.9467 | 52496.3438 |
69
- | 9000 | 0.1455 | 3944.9531 | 30197.7344 | 16.6107 | 66.9937 | 74.634 | 9.329 | 1385.5492 | 52496.3438 |
70
- | 12000 | 0.1939 | 3944.9531 | 30197.7344 | 16.6115 | 67.0666 | 74.553 | 9.319 | 1384.8617 | 52496.3438 |
71
- | 15000 | 0.2424 | 3937.6238 | 30180.7188 | 16.6121 | 66.6143 | 75.059 | 9.382 | 1385.3201 | 52524.3359 |
72
- | 18000 | 0.2909 | 3937.6238 | 30180.7188 | 16.6117 | 66.7575 | 74.898 | 9.362 | 1384.1750 | 52552.3945 |
73
- | 21000 | 0.3394 | 3944.9531 | 30197.7344 | 16.6115 | 66.679 | 74.986 | 9.373 | 1384.4041 | 52496.3438 |
74
- | 24000 | 0.3879 | 3944.9531 | 30197.7344 | 16.6121 | 66.8908 | 74.749 | 9.344 | 1384.4041 | 52468.3164 |
75
- | 27000 | 0.4364 | 3942.5085 | 30197.7344 | 16.6117 | 66.4311 | 75.266 | 9.408 | 1383.0317 | 52496.3438 |
76
- | 30000 | 0.4848 | 3940.0654 | 30180.7188 | 16.6107 | 66.4762 | 75.215 | 9.402 | 1383.2599 | 52496.3438 |
77
- | 33000 | 0.5333 | 3937.6238 | 30197.7344 | 16.6107 | 66.4814 | 75.209 | 9.401 | 1382.8029 | 52496.3438 |
78
- | 36000 | 0.5818 | 3942.5085 | 30180.7188 | 16.6111 | 67.3001 | 74.294 | 9.287 | 1385.3201 | 52496.3438 |
79
- | 39000 | 0.6303 | 3937.6238 | 30180.7188 | 16.6115 | 67.0065 | 74.62 | 9.327 | 1383.4888 | 52496.3438 |
80
- | 42000 | 0.6788 | 3942.5085 | 30197.7344 | 16.6109 | 66.7444 | 74.913 | 9.364 | 1384.1750 | 52496.3438 |
81
- | 45000 | 0.7273 | 3941.2869 | 30197.7344 | 16.6115 | 67.1516 | 74.458 | 9.307 | 1382.8029 | 52496.3438 |
82
- | 48000 | 0.7758 | 3944.9531 | 30180.7188 | 16.6107 | 66.7762 | 74.877 | 9.36 | 1386.6947 | 52524.3359 |
83
- | 51000 | 0.8242 | 3942.5085 | 30197.7344 | 16.6111 | 67.2623 | 74.336 | 9.292 | 1384.8617 | 52496.3438 |
84
- | 54000 | 0.8727 | 3944.9531 | 30180.7188 | 16.6107 | 66.724 | 74.936 | 9.367 | 1385.3201 | 52496.3438 |
85
- | 57000 | 0.9212 | 3941.2869 | 30197.7344 | 16.6115 | 67.0602 | 74.56 | 9.32 | 1382.8029 | 52468.3164 |
86
- | 60000 | 0.9697 | 3942.5085 | 30197.7344 | 16.6119 | 67.4137 | 74.169 | 9.271 | 1382.8029 | 52468.3164 |
87
- | 61875 | 1.0 | 3937.6238 | 30180.7188 | 16.6119 | 67.1794 | 74.428 | 9.303 | 1383.7178 | 52496.3438 |
88
 
89
  ### Framework versions
90
  - Distily 0.2.0
 
15
  The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
16
 
17
  It achieves the following results on the evaluation set:
18
+ - eval_enwikippl: 198.0270
19
+ - eval_frwikippl: 17127.3379
20
+ - eval_zhwikippl: 63614.1797
21
+ - eval_tinystoriesppl: 21.7514
22
+ - eval_loss: 12.7123
23
+ - eval_runtime: 65.1143
24
+ - eval_samples_per_second: 76.788
25
+ - eval_steps_per_second: 9.599
26
 
27
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
28
  should probably proofread and complete it, then remove this comment.
 
47
  The following hyperparameters were used during training:
48
  - distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10.0, loss_fn=raw_mse, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=mse, layer_mapper=None, projector=None))
49
  - train_embeddings: True
50
+ - learning_rate: 0.001
51
  - train_batch_size: 8
52
  - eval_batch_size: 8
53
  - seed: 42
54
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
55
  - lr_scheduler_type: constant
 
56
  - num_epochs: 1.0
57
 
58
  ### Resource Usage
59
+ Peak GPU Memory: 8.2666 GB
60
 
61
  ### Eval-Phase Metrics
62
  | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
63
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
  | **teacher eval** | | 169.9865 | 47377.9414 | | | | | 3.9789 | 4998.1294 |
65
+ | 0 | 0 | 14504.7236 | 73076.2578 | 17.7824 | 65.8473 | 75.933 | 9.492 | 6091.4858 | 69506.0391 |
66
+ | 3000 | 0.0485 | 198.0270 | 17127.3379 | 12.7127 | 65.1257 | 76.775 | 9.597 | 21.7514 | 63648.1641 |
67
+ | 6000 | 0.0970 | 197.0935 | 17088.7852 | 12.7124 | 65.0473 | 76.867 | 9.608 | 21.6429 | 63750.0977 |
68
+ | 9000 | 0.1455 | 198.0270 | 17127.3379 | 12.7123 | 65.1143 | 76.788 | 9.599 | 21.7514 | 63614.1797 |
69
+ | 12000 | 0.1939 | 197.4985 | 17098.4199 | 12.7128 | 65.0689 | 76.842 | 9.605 | 21.7173 | 63886.3086 |
70
+ | 15000 | 0.2424 | 197.3838 | 17122.5215 | 12.7128 | 65.1155 | 76.787 | 9.598 | 21.6850 | 63954.5195 |
71
+ | 18000 | 0.2909 | 197.9733 | 17108.0586 | 12.7130 | 65.0798 | 76.829 | 9.604 | 21.7865 | 63818.1680 |
72
+ | 21000 | 0.3394 | 197.1698 | 17103.2305 | 12.7134 | 65.0799 | 76.829 | 9.604 | 21.6322 | 64022.8672 |
73
+ | 24000 | 0.3879 | 197.8507 | 17117.7051 | 12.7131 | 65.1719 | 76.72 | 9.59 | 21.7757 | 63716.1211 |
74
+ | 27000 | 0.4364 | 197.6975 | 17127.3379 | 12.7131 | 65.1124 | 76.79 | 9.599 | 21.7191 | 63954.5195 |
75
+ | 30000 | 0.4848 | 197.3150 | 17079.1562 | 12.7131 | 65.1085 | 76.795 | 9.599 | 21.6904 | 63920.4375 |
76
+ | 33000 | 0.5333 | 197.6209 | 17103.2305 | 12.7129 | 65.2897 | 76.582 | 9.573 | 21.7191 | 63750.0977 |
77
+ | 36000 | 0.5818 | 198.3110 | 17127.3379 | 12.7122 | 65.5537 | 76.273 | 9.534 | 21.7883 | 63614.1797 |
78
+ | 39000 | 0.6303 | 198.2802 | 17127.3379 | 12.7128 | 65.2001 | 76.687 | 9.586 | 21.7721 | 63648.1641 |
79
+ | 42000 | 0.6788 | 197.9580 | 17127.3379 | 12.7130 | 65.4586 | 76.384 | 9.548 | 21.7433 | 63512.4648 |
80
+ | 45000 | 0.7273 | 198.2802 | 17108.0586 | 12.7129 | 65.2819 | 76.591 | 9.574 | 21.8009 | 63614.1797 |
81
+ | 48000 | 0.7758 | 197.5979 | 17098.4199 | 12.7125 | 65.1997 | 76.688 | 9.586 | 21.6940 | 63648.1641 |
82
+ | 51000 | 0.8242 | 198.2802 | 17127.3379 | 12.7120 | 65.5503 | 76.277 | 9.535 | 21.7892 | 63512.4648 |
83
+ | 54000 | 0.8727 | 198.2189 | 17127.3379 | 12.7129 | 65.3863 | 76.469 | 9.559 | 21.7811 | 63716.1211 |
84
+ | 57000 | 0.9212 | 199.1886 | 17136.9941 | 12.7133 | 65.1649 | 76.728 | 9.591 | 21.8759 | 63343.2148 |
85
+ | 60000 | 0.9697 | 197.3226 | 17122.5215 | 12.7127 | 65.2079 | 76.678 | 9.585 | 21.6886 | 63648.1641 |
86
+ | 61875 | 1.0 | 198.9419 | 17127.3379 | 12.7117 | 65.2766 | 76.597 | 9.575 | 21.8469 | 63648.1641 |
87
 
88
  ### Framework versions
89
  - Distily 0.2.0
logs/attn_loss_fn=mse, attn_weight=10.0, hs_loss_fn=raw_mse, hs_weight=10.0, learning_rate=0.001, warmup_ratio=0.1/events.out.tfevents.1723783997.5f530b1cf724 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:16e4ccf71b66127157a6775e8be200fb55cab2d7036f2472dc2067e3e182848d
3
+ size 16923356
logs/attn_loss_fn=mse, attn_weight=10.0, hs_loss_fn=raw_mse, hs_weight=10.0, learning_rate=0.001, warmup_ratio=0/events.out.tfevents.1723783790.5f530b1cf724 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:cf049bc8c5609dd342a0f94201a27d4f72d1883217a827d7aa47a5c5bc50a784
3
- size 312
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93679d6a3447c700947aad74a1a9b7e000b529808fc8824df22707cc5e201b9e
3
+ size 588
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:63548a6cc02bc7282675ac8003c55707aad063768454af1051d91b299d517c87
3
  size 137033984
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2658a5220385cffc81b92e31dc9da6ebb2722bea1f49ceeca4a1a62234bf776f
3
  size 137033984
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ad4d38544df7551ce00e936ef8f18583e06cde197b6b3130d03cbdef91d94057
3
  size 1017948104
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9595c7cbd53d3be0b11c43459f912b16bc77ebd4289706d42a11a77f806711d
3
  size 1017948104