lapp0 commited on
Commit
a50fb1c
1 Parent(s): 9664f42

End of training

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -76,7 +76,7 @@ Trained on 145,744,973 tokens from the [wikimedia/wikipedia](https://huggingface
76
  # Training Objective
77
 
78
  ```
79
- DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=last, projector=linear))
80
  ```
81
 
82
  # Hyperparameters
@@ -93,9 +93,9 @@ The following hyperparameters were used during training:
93
  - lr_scheduler_type: `cosine_with_min_lr`
94
  - lr_scheduler_warmup_ratio: `0.5`
95
  - num_epochs: `1.0`
96
- - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=last, projector=linear))`
97
  - train_embeddings: `True`
98
- - lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fa4b4615600>`
99
  - student_model_name_or_path: `None`
100
  - student_config_name_or_path: `None`
101
  - student_model_config: `None`
 
76
  # Training Objective
77
 
78
  ```
79
+ DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2, projector=linear))
80
  ```
81
 
82
  # Hyperparameters
 
93
  - lr_scheduler_type: `cosine_with_min_lr`
94
  - lr_scheduler_warmup_ratio: `0.5`
95
  - num_epochs: `1.0`
96
+ - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=25.0, loss_fn=cos, layer_mapper=layer-2, projector=linear))`
97
  - train_embeddings: `True`
98
+ - lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fa4f426c3d0>`
99
  - student_model_name_or_path: `None`
100
  - student_config_name_or_path: `None`
101
  - student_model_config: `None`