|
--- |
|
base_model: gpt2 |
|
datasets: |
|
- wikimedia/wikipedia |
|
library_name: Distily |
|
license: mit |
|
tags: |
|
- bitnet |
|
- 1.58b |
|
- generated_from_trainer |
|
model-index: |
|
- name: distily_test_attn_ortho_whiteners |
|
results: [] |
|
--- |
|
|
|
|
|
# Summary |
|
|
|
Distilled with [Distily](https://github.com/lapp0/distily) library |
|
using teacher model [gpt2](https://huggingface.co/gpt2) |
|
on dataset [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia). |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. |
|
|
|
# Model description |
|
|
|
More information needed |
|
|
|
# Intended uses & limitations |
|
|
|
More information needed |
|
--> |
|
|
|
# Model Architecture: |
|
- **Architecture**: `GPT2LMHeadModel` |
|
- **Total Parameters**: 124,439,808 |
|
- **Data Type (dtype)**: torch.bfloat16 |
|
- **Model Size**: 0.24 GB |
|
|
|
|
|
# Benchmark Metrics Comparison |
|
|
|
| Metric | | |
|
| :--- | |
|
|
|
# Resource Usage Comparison |
|
|
|
- VRAM Use: 7.7868 GB |
|
|
|
# Distillation (Teacher -> Student) Architecture Difference: |
|
|
|
- **Architecture**: `GPT2LMHeadModel` -> `GPT2LMHeadModel` |
|
- **Total Parameters**: 124,439,808 -> 124,439,808 |
|
- **Data Type (dtype)**: torch.bfloat16 -> torch.bfloat16 |
|
- **Model Size**: 0.24 GB -> 0.24 GB |
|
|
|
<details> |
|
<summary>Module Diff Details</summary> |
|
|
|
```diff |
|
|
|
``` |
|
|
|
</details> |
|
<br/> |
|
|
|
# Train Dataset |
|
Trained on 145,714,513 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset. |
|
|
|
- Num Samples: `247,500` |
|
- Subset: `20231101.en` |
|
- Split: `train` |
|
|
|
|
|
# Training Objective |
|
|
|
``` |
|
DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2, norm=batchnorm, projector=orthogonal)) |
|
``` |
|
|
|
# Hyperparameters |
|
The following hyperparameters were used during training: |
|
|
|
<details> |
|
<summary>Expand</summary> |
|
|
|
- learning_rate: `0.0001` |
|
- train_batch_size: `4` |
|
- eval_batch_size: `8` |
|
- seed: `42` |
|
- optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08` |
|
- lr_scheduler_type: `cosine_with_min_lr` |
|
- lr_scheduler_warmup_ratio: `0.5` |
|
- num_epochs: `1.0` |
|
- distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2, norm=batchnorm, projector=orthogonal))` |
|
- train_embeddings: `True` |
|
- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fc4addc8820>` |
|
- student_model_name_or_path: `None` |
|
- student_config_name_or_path: `None` |
|
- student_model_config: `None` |
|
- reinitialize_weights: `None` |
|
- copy_teacher_modules: `[('lm_head', False)]` |
|
- student_model_as_bitnet: `True` |
|
- dropout: `None` |
|
- teacher_model_name_or_path: `gpt2` |
|
- teacher_load_in_8bit: `False` |
|
- teacher_load_in_4bit: `False` |
|
- dataset_uri: `wikimedia/wikipedia` |
|
- dataset_subset: `20231101.en` |
|
- dataset_split: `train` |
|
- dataset_column_name: `text` |
|
- dataset_sample_size: `250000` |
|
- dataset_test_size: `0.01` |
|
- gradient_accumulation_steps: `1` |
|
- weight_decay: `0.0` |
|
- max_grad_norm: `100` |
|
- warmup_ratio: `0.5` |
|
- warmup_steps: `0` |
|
- gradient_checkpointing: `True` |
|
|
|
</details> |
|
<br/> |
|
|
|
|
|
# Framework Versions |
|
- Distily 0.4.1 |
|
- Transformers 4.44.1 |
|
- Pytorch 2.4.0+cu121 |
|
- Datasets 2.21.0 |
|
|
