---
base_model: gpt2
library_name: Distily
license: mit
tags:
- generated_from_trainer
model-index:
- name: istily_bench_gpt2_simple_objectives
  results: []
---

# distily_bench_gpt2_simple_objectives

This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).

The [Distily](https://github.com/lapp0/distily) library was used for this distillation.

It achieves the following results on the evaluation set:
- eval_enwikippl: 495.9718
- eval_frwikippl: 3345.0957
- eval_zhwikippl: 2696.0598
- eval_loss: 40.5622
- eval_runtime: 34.3051
- eval_samples_per_second: 58.3
- eval_steps_per_second: 7.288

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed
-->

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:kl_divergence_loss()), activations_weight=0.2, activations_loss_fn=(fn:mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:mse_loss()))
- train_embeddings: True
- learning_rate: 4e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- num_epochs: 1.0

### Resource Usage
Peak GPU Memory: 8.0893 GB

### Eval-Phase Metrics
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
| 0 | 0 | 54069.2930 | 57285.3438 | 133.3560 | 34.3024 | 58.305 | 7.288 | 54227.1016 |
| 1000 | 0.0404 | 1437.1931 | 8480.3613 | 43.3980 | 34.5438 | 57.898 | 7.237 | 74395.6406 |
| 2000 | 0.0808 | 997.8790 | 5260.8130 | 42.3937 | 34.3703 | 58.19 | 7.274 | 33120.7461 |
| 3000 | 0.1212 | 830.5260 | 5152.1616 | 41.7965 | 34.348 | 58.228 | 7.278 | 11334.5342 |
| 4000 | 0.1616 | 745.0864 | 4422.4756 | 41.2595 | 34.4519 | 58.052 | 7.256 | 5651.1323 |
| 5000 | 0.2020 | 644.0798 | 4158.1821 | 41.1632 | 34.4631 | 58.033 | 7.254 | 4903.9395 |
| 6000 | 0.2424 | 592.7726 | 3791.3215 | 40.8778 | 34.3097 | 58.293 | 7.287 | 4353.2559 |
| 7000 | 0.2828 | 545.3409 | 3490.1353 | 40.8020 | 34.4207 | 58.105 | 7.263 | 3123.4839 |
| 8000 | 0.3232 | 519.2236 | 3238.8032 | 40.6310 | 34.2625 | 58.373 | 7.297 | 1952.6049 |
| 9000 | 0.3636 | 495.9718 | 3345.0957 | 40.5622 | 34.3051 | 58.3 | 7.288 | 2696.0598 |
| 10000 | 0.4040 | 482.7110 | 3048.2520 | 40.4688 | 34.3828 | 58.169 | 7.271 | 2027.5375 |
| 11000 | 0.4444 | 453.9180 | 2860.8340 | 40.3758 | 34.2822 | 58.339 | 7.292 | 2861.5081 |
| 12000 | 0.4848 | 441.7129 | 2985.2966 | 40.2887 | 34.2175 | 58.45 | 7.306 | 2510.5007 |
| 13000 | 0.5253 | 429.0357 | 2882.7014 | 40.1765 | 34.4175 | 58.11 | 7.264 | 6012.3589 |
| 14000 | 0.5657 | 416.6578 | 2756.2913 | 40.1022 | 34.4762 | 58.011 | 7.251 | 12478.4199 |
| 15000 | 0.6061 | 406.1163 | 2797.8003 | 40.0135 | 34.5042 | 57.964 | 7.246 | 6068.8252 |
| 16000 | 0.6465 | 405.6435 | 2525.5491 | 39.9328 | 34.3124 | 58.288 | 7.286 | 4309.2979 |
| 17000 | 0.6869 | 394.7977 | 2709.6606 | 39.9735 | 34.3165 | 58.281 | 7.285 | 2797.2800 |
| 18000 | 0.7273 | 397.4739 | 2544.8535 | 39.7368 | 34.4016 | 58.137 | 7.267 | 9888.5605 |
| 19000 | 0.7677 | 387.6284 | 2540.5505 | 39.7513 | 34.3493 | 58.225 | 7.278 | 5071.7769 |
| 20000 | 0.8081 | 378.9675 | 2503.9182 | 39.6105 | 34.4198 | 58.106 | 7.263 | 3492.3926 |
| 21000 | 0.8485 | 376.9130 | 2442.8845 | 39.5590 | 34.343 | 58.236 | 7.28 | 10077.8555 |
| 22000 | 0.8889 | 374.1136 | 2348.3101 | 39.5182 | 34.2953 | 58.317 | 7.29 | 3595.5537 |
| 23000 | 0.9293 | 368.7203 | 2389.3955 | 39.4282 | 34.6197 | 57.771 | 7.221 | 11663.1113 |
| 24000 | 0.9697 | 365.7831 | 2363.9253 | 39.4065 | 34.6468 | 57.725 | 7.216 | 5269.2183 |
| 24750 | 1.0 | 363.6872 | 2441.5068 | 39.3040 | 34.7181 | 57.607 | 7.201 | 2566.7729 |

### Framework versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.20.0