---
base_model: roneneldan/TinyStories-33M
library_name: Distily
tags:
- generated_from_trainer
model-index:
- name: distily_bench_obj_cross_v2.6
  results: []
---

# distily_bench_obj_cross_v2.6

This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).

The [Distily](https://github.com/lapp0/distily) library was used for this distillation.

It achieves the following results on the evaluation set:
- eval_enwikippl: 87049.7578
- eval_frwikippl: 148519.8594
- eval_zhwikippl: 112743.5078
- eval_tinystoriesppl: 68038.7344
- eval_loss: 32.1160
- eval_runtime: 11.5146
- eval_samples_per_second: 86.847
- eval_steps_per_second: 10.856

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed
-->

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=0, loss_fn=None, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=10, loss_fn=kl, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
- train_embeddings: True
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1.0

### Resource Usage
Peak GPU Memory: 6.6287 GB

### Eval-Phase Metrics
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
| 0 | 0 | 88697.0156 | 150478.2188 | 32.2330 | 11.5103 | 86.878 | 10.86 | 69390.6016 | 113346.8047 |
| 500 | 0.0404 | 87049.7578 | 148519.8594 | 32.1160 | 11.5316 | 86.718 | 10.84 | 67960.0703 | 112623.2578 |
| 1000 | 0.0808 | 87049.7578 | 148519.8594 | 32.1180 | 11.498 | 86.971 | 10.871 | 68016.2188 | 112743.5078 |
| 1500 | 0.1212 | 87049.7578 | 148519.8594 | 32.1180 | 11.5171 | 86.828 | 10.853 | 67993.7812 | 112743.5078 |
| 2000 | 0.1616 | 87049.7578 | 148519.8594 | 32.1160 | 11.5112 | 86.872 | 10.859 | 68038.7344 | 112743.5078 |
| 2500 | 0.2020 | 87049.7578 | 148519.8594 | 32.1160 | 11.5174 | 86.825 | 10.853 | 68038.7344 | 112743.5078 |
| 3000 | 0.2424 | 87049.7578 | 148519.8594 | 32.1160 | 11.5446 | 86.621 | 10.828 | 68016.2188 | 112743.5078 |
| 3500 | 0.2828 | 87049.7578 | 148519.8594 | 32.1160 | 11.5015 | 86.945 | 10.868 | 68038.7344 | 112743.5078 |
| 4000 | 0.3232 | 87049.7578 | 148519.8594 | 32.1160 | 11.5349 | 86.693 | 10.837 | 68038.7344 | 112743.5078 |
| 4500 | 0.3636 | 87049.7578 | 148519.8594 | 32.1160 | 11.5299 | 86.731 | 10.841 | 68038.7344 | 112743.5078 |
| 5000 | 0.4040 | 87049.7578 | 148519.8594 | 32.1160 | 11.5259 | 86.761 | 10.845 | 68038.7344 | 112743.5078 |
| 5500 | 0.4444 | 87049.7578 | 148519.8594 | 32.1160 | 11.5002 | 86.955 | 10.869 | 68038.7344 | 112743.5078 |
| 6000 | 0.4848 | 87049.7578 | 148603.5938 | 32.1160 | 11.5135 | 86.855 | 10.857 | 68061.25 | 112743.5078 |
| 6500 | 0.5253 | 87049.7578 | 148603.5938 | 32.1160 | 11.5069 | 86.904 | 10.863 | 68061.25 | 112743.5078 |
| 7000 | 0.5657 | 87049.7578 | 148603.5938 | 32.1160 | 11.509 | 86.889 | 10.861 | 68061.25 | 112743.5078 |
| 7500 | 0.6061 | 87049.7578 | 148603.5938 | 32.1160 | 11.508 | 86.896 | 10.862 | 68061.25 | 112743.5078 |
| 8000 | 0.6465 | 87049.7578 | 148603.5938 | 32.1160 | 11.5151 | 86.843 | 10.855 | 68038.7344 | 112743.5078 |
| 8500 | 0.6869 | 87049.7578 | 148519.8594 | 32.1160 | 11.4916 | 87.02 | 10.878 | 68038.7344 | 112743.5078 |
| 9000 | 0.7273 | 87049.7578 | 148519.8594 | 32.1160 | 11.5189 | 86.814 | 10.852 | 68038.7344 | 112743.5078 |
| 9500 | 0.7677 | 87049.7578 | 148519.8594 | 32.1160 | 11.5146 | 86.847 | 10.856 | 68038.7344 | 112743.5078 |
| 10000 | 0.8081 | 87049.7578 | 148519.8594 | 32.1160 | 11.5098 | 86.883 | 10.86 | 68038.7344 | 112743.5078 |
| 10500 | 0.8485 | 87049.7578 | 148519.8594 | 32.1160 | 11.5054 | 86.916 | 10.865 | 68038.7344 | 112743.5078 |
| 11000 | 0.8889 | 87049.7578 | 148519.8594 | 32.1160 | 11.5094 | 86.885 | 10.861 | 68038.7344 | 112743.5078 |
| 11500 | 0.9293 | 87049.7578 | 148519.8594 | 32.1160 | 11.5376 | 86.673 | 10.834 | 68038.7344 | 112743.5078 |
| 12000 | 0.9697 | 87049.7578 | 148519.8594 | 32.1160 | 11.494 | 87.002 | 10.875 | 68038.7344 | 112743.5078 |
| 12375 | 1.0 | 87049.7578 | 148519.8594 | 32.1160 | 11.4926 | 87.013 | 10.877 | 68038.7344 | 112743.5078 |

### Framework versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.20.0