End of training

README.md

@@ -0,0 +1,96 @@
+---
+base_model: roneneldan/TinyStories-33M
+library_name: Distily
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_obj_cross_v2.6
+  results: []
+---
+
+# distily_bench_obj_cross_v2.6
+
+This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 1915.6801
+- eval_frwikippl: 17380.0898
+- eval_zhwikippl: 80448.5391
+- eval_tinystoriesppl: 689.9122
+- eval_loss: 6.2077
+- eval_runtime: 11.4682
+- eval_samples_per_second: 87.198
+- eval_steps_per_second: 10.9
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=0, loss_fn=None, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10, loss_fn=cos, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- train_embeddings: True
+- learning_rate: 0.01
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 6.6276 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
+| 0 | 0 | 23978.2344 | 74374.2891 | 7.8203 | 11.5608 | 86.499 | 10.812 | 11110.3125 | 69209.9141 |
+| 500 | 0.0404 | 3768063.25 | 1320083162923008.0 | 6.3913 | 11.5562 | 86.534 | 10.817 | 123.9350 | 35325253458591744.0000 |
+| 1000 | 0.0808 | 40494.2930 | 34653921280.0 | 6.3220 | 11.5104 | 86.878 | 10.86 | 33.9454 | 1850179059712.0 |
+| 1500 | 0.1212 | 1197.6969 | 168759.6875 | 6.2397 | 11.5411 | 86.647 | 10.831 | 117.6943 | 1388815.625 |
+| 2000 | 0.1616 | 1228.2329 | 20482.3652 | 6.2140 | 11.4883 | 87.045 | 10.881 | 326.2055 | 114379.5781 |
+| 2500 | 0.2020 | 1239.8959 | 19846.2070 | 6.2130 | 11.5386 | 86.665 | 10.833 | 336.5318 | 103600.4453 |
+| 3000 | 0.2424 | 1914.1963 | 17243.5391 | 6.2077 | 11.5623 | 86.488 | 10.811 | 690.1402 | 78960.0391 |
+| 3500 | 0.2828 | 1921.0294 | 17666.4023 | 6.2095 | 11.5099 | 86.882 | 10.86 | 689.5698 | 83088.0156 |
+| 4000 | 0.3232 | 1873.1237 | 17426.6621 | 6.2077 | 11.4977 | 86.974 | 10.872 | 664.2795 | 81834.0312 |
+| 4500 | 0.3636 | 1893.8420 | 17517.7246 | 6.2077 | 11.5637 | 86.477 | 10.81 | 677.2543 | 82140.2188 |
+| 5000 | 0.4040 | 1924.9026 | 17336.0898 | 6.2075 | 11.5831 | 86.333 | 10.792 | 695.7541 | 80234.2344 |
+| 5500 | 0.4444 | 1891.4965 | 17372.75 | 6.2075 | 11.496 | 86.987 | 10.873 | 675.6886 | 80878.9453 |
+| 6000 | 0.4848 | 1897.3663 | 17453.6895 | 6.2073 | 11.5439 | 86.626 | 10.828 | 679.7222 | 81746.75 |
+| 6500 | 0.5253 | 1904.1395 | 17316.5586 | 6.2075 | 11.5881 | 86.295 | 10.787 | 685.9882 | 80362.7344 |
+| 7000 | 0.5657 | 1897.9545 | 17316.5586 | 6.2075 | 11.4888 | 87.042 | 10.88 | 679.2728 | 81485.4609 |
+| 7500 | 0.6061 | 1886.2292 | 17407.0449 | 6.2077 | 11.5303 | 86.728 | 10.841 | 673.7922 | 81528.9062 |
+| 8000 | 0.6465 | 1883.8922 | 17250.8242 | 6.2075 | 11.5743 | 86.399 | 10.8 | 673.7922 | 79892.4688 |
+| 8500 | 0.6869 | 1894.4291 | 17448.7793 | 6.2070 | 11.4754 | 87.143 | 10.893 | 678.1508 | 82008.8828 |
+| 9000 | 0.7273 | 1894.4291 | 17326.3223 | 6.2067 | 11.4686 | 87.195 | 10.899 | 677.5906 | 81746.75 |
+| 9500 | 0.7677 | 1915.6801 | 17380.0898 | 6.2077 | 11.4682 | 87.198 | 10.9 | 689.9122 | 80448.5391 |
+| 10000 | 0.8081 | 1907.6830 | 17326.3223 | 6.2075 | 11.5672 | 86.451 | 10.806 | 687.4073 | 81008.5469 |
+| 10500 | 0.8485 | 1899.1313 | 17596.8730 | 6.2075 | 11.5368 | 86.679 | 10.835 | 677.2543 | 82955.0859 |
+| 11000 | 0.8889 | 1887.6906 | 17461.0645 | 6.2073 | 11.5979 | 86.223 | 10.778 | 672.4567 | 82008.8828 |
+| 11500 | 0.9293 | 1895.6039 | 17399.6934 | 6.2077 | 11.5089 | 86.889 | 10.861 | 678.9360 | 81790.3438 |
+| 12000 | 0.9697 | 1898.5428 | 17429.1211 | 6.2077 | 11.5488 | 86.589 | 10.824 | 680.5093 | 81790.3438 |
+| 12375 | 1.0 | 1898.5428 | 17429.1211 | 6.2077 | 11.5716 | 86.418 | 10.802 | 680.7906 | 81790.3438 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.20.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/hs_loss_fn=cos, hs_weight=10, learning_rate=0.01/events.out.tfevents.1723850337.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1779b913cb1d95bebfe2f9e2936883225318df17b4f94650a9a39e2d5ddeb76
+size 307