|
--- |
|
base_model: roneneldan/TinyStories-33M |
|
library_name: Distily |
|
tags: |
|
- generated_from_trainer |
|
model-index: |
|
- name: distily_bench_obj_cross_v2.6 |
|
results: [] |
|
--- |
|
|
|
# distily_bench_obj_cross_v2.6 |
|
|
|
This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified). |
|
|
|
The [Distily](https://github.com/lapp0/distily) library was used for this distillation. |
|
|
|
It achieves the following results on the evaluation set: |
|
- eval_enwikippl: 87049.7578 |
|
- eval_frwikippl: 148519.8594 |
|
- eval_zhwikippl: 112743.5078 |
|
- eval_tinystoriesppl: 68038.7344 |
|
- eval_loss: 32.1160 |
|
- eval_runtime: 11.5146 |
|
- eval_samples_per_second: 86.847 |
|
- eval_steps_per_second: 10.856 |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
--> |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=0, loss_fn=None, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10, loss_fn=kl, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None)) |
|
- train_embeddings: True |
|
- learning_rate: 0.0001 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 1.0 |
|
|
|
### Resource Usage |
|
Peak GPU Memory: 6.6287 GB |
|
|
|
### Eval-Phase Metrics |
|
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl | |
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | |
|
| **teacher eval** | | 169.9865 | 47377.9414 | | | | | 3.9789 | 4998.1294 | |
|
| 0 | 0 | 88697.0156 | 150478.2188 | 32.2330 | 11.5103 | 86.878 | 10.86 | 69390.6016 | 113346.8047 | |
|
| 500 | 0.0404 | 87049.7578 | 148519.8594 | 32.1160 | 11.5316 | 86.718 | 10.84 | 67960.0703 | 112623.2578 | |
|
| 1000 | 0.0808 | 87049.7578 | 148519.8594 | 32.1180 | 11.498 | 86.971 | 10.871 | 68016.2188 | 112743.5078 | |
|
| 1500 | 0.1212 | 87049.7578 | 148519.8594 | 32.1180 | 11.5171 | 86.828 | 10.853 | 67993.7812 | 112743.5078 | |
|
| 2000 | 0.1616 | 87049.7578 | 148519.8594 | 32.1160 | 11.5112 | 86.872 | 10.859 | 68038.7344 | 112743.5078 | |
|
| 2500 | 0.2020 | 87049.7578 | 148519.8594 | 32.1160 | 11.5174 | 86.825 | 10.853 | 68038.7344 | 112743.5078 | |
|
| 3000 | 0.2424 | 87049.7578 | 148519.8594 | 32.1160 | 11.5446 | 86.621 | 10.828 | 68016.2188 | 112743.5078 | |
|
| 3500 | 0.2828 | 87049.7578 | 148519.8594 | 32.1160 | 11.5015 | 86.945 | 10.868 | 68038.7344 | 112743.5078 | |
|
| 4000 | 0.3232 | 87049.7578 | 148519.8594 | 32.1160 | 11.5349 | 86.693 | 10.837 | 68038.7344 | 112743.5078 | |
|
| 4500 | 0.3636 | 87049.7578 | 148519.8594 | 32.1160 | 11.5299 | 86.731 | 10.841 | 68038.7344 | 112743.5078 | |
|
| 5000 | 0.4040 | 87049.7578 | 148519.8594 | 32.1160 | 11.5259 | 86.761 | 10.845 | 68038.7344 | 112743.5078 | |
|
| 5500 | 0.4444 | 87049.7578 | 148519.8594 | 32.1160 | 11.5002 | 86.955 | 10.869 | 68038.7344 | 112743.5078 | |
|
| 6000 | 0.4848 | 87049.7578 | 148603.5938 | 32.1160 | 11.5135 | 86.855 | 10.857 | 68061.25 | 112743.5078 | |
|
| 6500 | 0.5253 | 87049.7578 | 148603.5938 | 32.1160 | 11.5069 | 86.904 | 10.863 | 68061.25 | 112743.5078 | |
|
| 7000 | 0.5657 | 87049.7578 | 148603.5938 | 32.1160 | 11.509 | 86.889 | 10.861 | 68061.25 | 112743.5078 | |
|
| 7500 | 0.6061 | 87049.7578 | 148603.5938 | 32.1160 | 11.508 | 86.896 | 10.862 | 68061.25 | 112743.5078 | |
|
| 8000 | 0.6465 | 87049.7578 | 148603.5938 | 32.1160 | 11.5151 | 86.843 | 10.855 | 68038.7344 | 112743.5078 | |
|
| 8500 | 0.6869 | 87049.7578 | 148519.8594 | 32.1160 | 11.4916 | 87.02 | 10.878 | 68038.7344 | 112743.5078 | |
|
| 9000 | 0.7273 | 87049.7578 | 148519.8594 | 32.1160 | 11.5189 | 86.814 | 10.852 | 68038.7344 | 112743.5078 | |
|
| 9500 | 0.7677 | 87049.7578 | 148519.8594 | 32.1160 | 11.5146 | 86.847 | 10.856 | 68038.7344 | 112743.5078 | |
|
| 10000 | 0.8081 | 87049.7578 | 148519.8594 | 32.1160 | 11.5098 | 86.883 | 10.86 | 68038.7344 | 112743.5078 | |
|
| 10500 | 0.8485 | 87049.7578 | 148519.8594 | 32.1160 | 11.5054 | 86.916 | 10.865 | 68038.7344 | 112743.5078 | |
|
| 11000 | 0.8889 | 87049.7578 | 148519.8594 | 32.1160 | 11.5094 | 86.885 | 10.861 | 68038.7344 | 112743.5078 | |
|
| 11500 | 0.9293 | 87049.7578 | 148519.8594 | 32.1160 | 11.5376 | 86.673 | 10.834 | 68038.7344 | 112743.5078 | |
|
| 12000 | 0.9697 | 87049.7578 | 148519.8594 | 32.1160 | 11.494 | 87.002 | 10.875 | 68038.7344 | 112743.5078 | |
|
| 12375 | 1.0 | 87049.7578 | 148519.8594 | 32.1160 | 11.4926 | 87.013 | 10.877 | 68038.7344 | 112743.5078 | |
|
|
|
### Framework versions |
|
- Distily 0.2.0 |
|
- Transformers 4.44.0 |
|
- Pytorch 2.3.0 |
|
- Datasets 2.20.0 |
|
|
