lapp0's picture
End of training
d81becf verified
metadata
base_model: roneneldan/TinyStories-33M
library_name: Distily
tags:
  - generated_from_trainer
model-index:
  - name: distily_bench_obj_cross
    results: []

distily_bench_obj_cross

This student model is distilled from the teacher model roneneldan/TinyStories-33M using the dataset (unspecified).

The Distily library was used for this distillation.

It achieves the following results on the evaluation set:

  • eval_enwikippl: 148.8680
  • eval_frwikippl: 21987.7637
  • eval_zhwikippl: 181662.0469
  • eval_tinystoriesppl: 12.2941
  • eval_loss: 25.4402
  • eval_runtime: 66.3462
  • eval_samples_per_second: 75.362
  • eval_steps_per_second: 9.42

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10.0, loss_fn=kl, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=10.0, loss_fn=mse, layer_mapper=None, projector=None))
  • train_embeddings: True
  • learning_rate: 0.004
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • num_epochs: 1.0

Resource Usage

Peak GPU Memory: 8.2666 GB

Eval-Phase Metrics

step epoch enwikippl frwikippl loss runtime samples_per_second steps_per_second tinystoriesppl zhwikippl
teacher eval 169.9865 47377.9414 3.9789 4998.1294
0 0 8167.3613 48488.5742 38.4688 65.8686 75.909 9.489 3345.3254 73944.1484
3000 0.0485 145.5666 22094.8828 25.4406 65.8501 75.93 9.491 11.9420 179685.7344
6000 0.0970 146.8635 22376.7676 25.4394 66.4135 75.286 9.411 11.9667 183024.1719
9000 0.1455 148.8680 21987.7637 25.4402 66.3462 75.362 9.42 12.2941 181662.0469
12000 0.1939 151.0636 22504.7676 25.4400 66.2246 75.501 9.438 12.5052 181759.0938
15000 0.2424 146.5339 22604.8535 25.4392 66.1192 75.621 9.453 11.8540 189888.4375
18000 0.2909 147.3192 22481.0215 25.4400 66.2457 75.477 9.435 12.0058 183905.2969
21000 0.3394 150.9525 22555.5625 25.4390 66.2661 75.453 9.432 12.4310 188575.7344
24000 0.3879 149.8920 22155.6523 25.4404 66.2363 75.487 9.436 12.4593 177493.9531
27000 0.4364 147.1653 22531.7402 25.4398 66.3823 75.321 9.415 11.9514 183905.2969
30000 0.4848 150.4855 22580.9805 25.4400 66.2281 75.497 9.437 12.4172 183513.1875
33000 0.5333 145.7359 22307.5195 25.4400 66.4448 75.25 9.406 11.9159 180165.8438
36000 0.5818 148.7297 22495.2617 25.4396 66.2715 75.447 9.431 12.1426 186574.0156
39000 0.6303 147.5820 22807.9492 25.4406 66.6342 75.037 9.38 11.9944 187372.1406
42000 0.6788 150.2292 22193.125 25.4402 66.5873 75.089 9.386 12.5202 182050.1875
45000 0.7273 146.7725 22207.2051 25.4400 66.1476 75.589 9.449 11.9890 181468.2812
48000 0.7758 146.3014 22194.6914 25.4398 66.4166 75.282 9.41 11.9746 177588.7812
51000 0.8242 148.6375 22533.3301 25.4402 66.2612 75.459 9.432 12.1471 186275.5156
54000 0.8727 147.6220 22394.1035 25.4404 66.4085 75.292 9.411 12.1140 185581.1406
57000 0.9212 148.8161 22679.8047 25.4400 66.3328 75.377 9.422 12.1230 187872.7812
60000 0.9697 146.8180 22345.2695 25.4392 66.5261 75.158 9.395 12.0317 181371.5625
61875 1.0 149.0526 22099.5410 25.4400 66.498 75.19 9.399 12.3048 181371.5625

Framework versions

  • Distily 0.2.0
  • Transformers 4.44.0
  • Pytorch 2.3.0
  • Datasets 2.20.0