End of training

README.md

@@ -78,12 +78,12 @@ LlamaForCausalLM(
 
 # Resource Usage
 
-- Max Train VRAM Use: 12.7946 GB
+- Max Train VRAM Use: 20.9417 GB
 - Available VRAM: 23.4329 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
 - CPUs: 64
-- CPU Memory: 251.7299 GB
+- CPU Memory: 251.7190 GB
 - CPU Memory Bandwidth: 1600 GB/s
 
 # Distillation (Teacher -> Student) Architecture Difference:
@@ -115,7 +115,7 @@ LlamaForCausalLM(
 <br/>
 
 # Train Dataset
-Trained on 84,871,894 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 84,851,671 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `99,800`
 - Subset: `20231101.en`
@@ -134,7 +134,11 @@ DistillationObjective(
         weight=0
     ),
     attn_loss_component=LossComponent(
-        weight=0
+        weight=5.0,
+        loss_fn='raw_mse',
+        layer_mapper='layer-2',
+        norm='layernorm_teacher_only_affine',
+        projector='mlp'
     )
 )
 ```
@@ -146,7 +150,7 @@ The following hyperparameters were used during training:
 <summary>Expand</summary>
 
 - learning_rate: `0.0002`
-- train_batch_size: `4`
+- train_batch_size: `2`
 - eval_batch_size: `2`
 - seed: `42`
 - optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08`
@@ -161,10 +165,14 @@ The following hyperparameters were used during training:
         weight=0
     ),
     attn_loss_component=LossComponent(
-        weight=0
+        weight=5.0,
+        loss_fn='raw_mse',
+        layer_mapper='layer-2',
+        norm='layernorm_teacher_only_affine',
+        projector='mlp'
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7eb253ff9660>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7f6ad1b142e0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `{'num_hidden_layers': 15}`
@@ -197,6 +205,6 @@ The following hyperparameters were used during training:
 
 # Framework Versions
 - Distily 0.5.0
-- Transformers 4.44.2
+- Transformers 4.45.0.dev0
 - Pytorch 2.5.0.dev20240911+cu121
-- Datasets 2.21.0
+- Datasets 3.0.0

config.json

@@ -7,6 +7,7 @@
   "attention_dropout": 0.0,
   "bos_token_id": 0,
   "eos_token_id": 0,
+  "head_dim": 64,
   "hidden_act": "silu",
   "hidden_size": 576,
   "initializer_range": 0.02,
@@ -23,7 +24,7 @@
   "rope_theta": 10000.0,
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.44.2",
+  "transformers_version": "4.45.0.dev0",
   "use_cache": false,
   "vocab_size": 49152
 }

generation_config.json

@@ -2,6 +2,6 @@
   "_from_model_config": true,
   "bos_token_id": 0,
   "eos_token_id": 0,
-  "transformers_version": "4.44.2",
+  "transformers_version": "4.45.0.dev0",
   "use_cache": false
 }

logs/attn_projector=mlp, per_device_train_batch_size=2, run_name=bs2/events.out.tfevents.1726128095.46d00238c241

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:05049f5d87e4a7ddb650af801d9766520fbc61a7d71713dd3954ec73157c8e85
-size 1242037
+oid sha256:f78bc4a2160f151a0d8f9bdf66792688f7bc2240e5b7bd32ad62225dc932b183
+size 1378148

logs/attn_projector=mlp, per_device_train_batch_size=2, run_name=bs2/events.out.tfevents.1726148603.46d00238c241

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7ca17bc388a40ebfeee488d1e80d7a08e2f9ca04365e57be9d8a1de4a8cf834
+size 253

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4138576611832b1bbe77dc349189271d76106c052e215808da84cf40e98bd034
+oid sha256:f4d35e5221c1a9340eef663c0674f7b544ed09e4f95457f0c98eb1c255a2acce
 size 162842416

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:555ceb45b35190b72c7d1a0f76611e172bacea5f42f7a7ad17b8bf6a95802281
-size 5432
+oid sha256:738e9326fbb30717baf30f64578731beac353bb8abd9108ce8783db4565fa56e
+size 5368