End of training

README.md

@@ -0,0 +1,87 @@
+---
+base_model: Qwen/Qwen2-0.5B-Instruct
+library_name: distily
+license: apache-2.0
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_experiments_loss_cakld
+  results: []
+---
+
+# distily_experiments_loss_cakld
+
+This student model is distilled from the teacher model [Qwen/Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 12004.6143
+- eval_frwikippl: 41571.4961
+- eval_zhwikippl: 304727.5625
+- eval_loss: 5.0626
+- eval_runtime: 154.2547
+- eval_samples_per_second: 6.483
+- eval_steps_per_second: 3.241
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_strategy: logits_activations
+- loss_fn: cakld
+- train_embeddings: True
+- learning_rate: 4e-05
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 42
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 16.3131 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 13.0728 | 11.6511 |  |  |  |  | 21.6338 |
+| 0 | 0 | 180421.8594 | 182559.25 | 426.3911 | 153.7612 | 6.504 | 3.252 | 181425.2031 |
+| 500 | 0.0808 | 25914.9355 | 58400.8789 | 9.3101 | 153.5119 | 6.514 | 3.257 | 246441.4688 |
+| 1000 | 0.1616 | 18034.6934 | 51549.3594 | 7.7199 | 153.5062 | 6.514 | 3.257 | 268579.2188 |
+| 1500 | 0.2424 | 16219.1631 | 49107.0469 | 6.9828 | 153.332 | 6.522 | 3.261 | 295905.5312 |
+| 2000 | 0.3232 | 14691.3838 | 42907.7266 | 6.4286 | 153.9179 | 6.497 | 3.248 | 327841.9375 |
+| 2500 | 0.4040 | 16166.2109 | 47578.2539 | 6.0867 | 153.7651 | 6.503 | 3.252 | 314603.125 |
+| 3000 | 0.4848 | 15222.2021 | 45014.1914 | 5.7882 | 153.7316 | 6.505 | 3.252 | 333285.875 |
+| 3500 | 0.5657 | 14355.2061 | 44631.9570 | 5.5960 | 154.0532 | 6.491 | 3.246 | 350936.6875 |
+| 4000 | 0.6465 | 13183.8857 | 43155.7617 | 5.4694 | 154.1821 | 6.486 | 3.243 | 353883.125 |
+| 4500 | 0.7273 | 12622.3330 | 41256.7539 | 5.5192 | 154.1834 | 6.486 | 3.243 | 351506.7812 |
+| 5000 | 0.8081 | 12319.9580 | 40217.3828 | 5.4403 | 154.6126 | 6.468 | 3.234 | 345530.5 |
+| 5500 | 0.8889 | 12254.1387 | 42295.5117 | 5.1549 | 154.6614 | 6.466 | 3.233 | 350950.0625 |
+| 6000 | 0.9697 | 11398.9785 | 39517.2930 | 5.0769 | 154.199 | 6.485 | 3.243 | 359435.0625 |
+| 6187 | 0.9999 | 12004.6143 | 41571.4961 | 5.0626 | 154.2547 | 6.483 | 3.241 | 304727.5625 |
+
+### Framework versions
+- Distily 0.1.0
+- Transformers 4.43.3
+- Pytorch 2.3.0
+- Datasets 2.20.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "transformers_version": "4.43.3"
+}

runs/Aug07_13-26-55_232a0f8c3879/events.out.tfevents.1723065227.232a0f8c3879

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19781e43eebccede5824d78f079551ec614225cd82fbcb6335553dea05627b6b
+size 520