Model save

README.md

@@ -0,0 +1,123 @@
+---
+license: apache-2.0
+base_model: ondevicellm/tinyllama_moe
+tags:
+- trl
+- sft
+- generated_from_trainer
+datasets:
+- generator
+model-index:
+- name: tinyllama_moe_sft_ultrachat200k_v2_epochs5
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# tinyllama_moe_sft_ultrachat200k_v2_epochs5
+
+This model is a fine-tuned version of [ondevicellm/tinyllama_moe](https://huggingface.co/ondevicellm/tinyllama_moe) on the generator dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.1090
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 16
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 128
+- total_eval_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 115
+- num_epochs: 5
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.3359        | 0.09  | 100  | 1.3129          |
+| 1.2425        | 0.18  | 200  | 1.2363          |
+| 1.2079        | 0.26  | 300  | 1.2083          |
+| 1.1849        | 0.35  | 400  | 1.1910          |
+| 1.1545        | 0.44  | 500  | 1.1786          |
+| 1.174         | 0.53  | 600  | 1.1690          |
+| 1.1609        | 0.61  | 700  | 1.1610          |
+| 1.1449        | 0.7   | 800  | 1.1543          |
+| 1.1406        | 0.79  | 900  | 1.1485          |
+| 1.1241        | 0.88  | 1000 | 1.1432          |
+| 1.1477        | 0.96  | 1100 | 1.1385          |
+| 1.0644        | 1.05  | 1200 | 1.1382          |
+| 1.067         | 1.14  | 1300 | 1.1359          |
+| 1.0791        | 1.23  | 1400 | 1.1332          |
+| 1.0702        | 1.31  | 1500 | 1.1304          |
+| 1.0741        | 1.4   | 1600 | 1.1277          |
+| 1.0701        | 1.49  | 1700 | 1.1251          |
+| 1.0529        | 1.58  | 1800 | 1.1225          |
+| 1.072         | 1.66  | 1900 | 1.1199          |
+| 1.0759        | 1.75  | 2000 | 1.1178          |
+| 1.0618        | 1.84  | 2100 | 1.1152          |
+| 1.0359        | 1.93  | 2200 | 1.1134          |
+| 0.9918        | 2.01  | 2300 | 1.1195          |
+| 1.002         | 2.1   | 2400 | 1.1205          |
+| 0.993         | 2.19  | 2500 | 1.1194          |
+| 0.9872        | 2.28  | 2600 | 1.1184          |
+| 0.9849        | 2.37  | 2700 | 1.1172          |
+| 0.9924        | 2.45  | 2800 | 1.1156          |
+| 0.9971        | 2.54  | 2900 | 1.1145          |
+| 0.9786        | 2.63  | 3000 | 1.1130          |
+| 0.9923        | 2.72  | 3100 | 1.1122          |
+| 0.9888        | 2.8   | 3200 | 1.1106          |
+| 0.9826        | 2.89  | 3300 | 1.1091          |
+| 0.9997        | 2.98  | 3400 | 1.1090          |
+| 0.9267        | 3.07  | 3500 | 1.1219          |
+| 0.9465        | 3.15  | 3600 | 1.1225          |
+| 0.9255        | 3.24  | 3700 | 1.1221          |
+| 0.9532        | 3.33  | 3800 | 1.1214          |
+| 0.9372        | 3.42  | 3900 | 1.1215          |
+| 0.9206        | 3.5   | 4000 | 1.1213          |
+| 0.9394        | 3.59  | 4100 | 1.1207          |
+| 0.9367        | 3.68  | 4200 | 1.1195          |
+| 0.9245        | 3.77  | 4300 | 1.1191          |
+| 0.9386        | 3.85  | 4400 | 1.1187          |
+| 0.9209        | 3.94  | 4500 | 1.1187          |
+| 0.9028        | 4.03  | 4600 | 1.1261          |
+| 0.9087        | 4.12  | 4700 | 1.1278          |
+| 0.9114        | 4.2   | 4800 | 1.1277          |
+| 0.8854        | 4.29  | 4900 | 1.1280          |
+| 0.902         | 4.38  | 5000 | 1.1278          |
+| 0.9038        | 4.47  | 5100 | 1.1280          |
+| 0.8935        | 4.56  | 5200 | 1.1280          |
+| 0.9053        | 4.64  | 5300 | 1.1280          |
+| 0.9091        | 4.73  | 5400 | 1.1278          |
+| 0.8968        | 4.82  | 5500 | 1.1279          |
+| 0.9196        | 4.91  | 5600 | 1.1279          |
+| 0.9129        | 4.99  | 5700 | 1.1279          |
+
+
+### Framework versions
+
+- Transformers 4.36.2
+- Pytorch 2.1.2+cu118
+- Datasets 2.14.6
+- Tokenizers 0.15.0

all_results.json

@@ -0,0 +1,13 @@
+{
+    "epoch": 5.0,
+    "eval_loss": 1.1089974641799927,
+    "eval_runtime": 435.9692,
+    "eval_samples": 23110,
+    "eval_samples_per_second": 37.081,
+    "eval_steps_per_second": 1.161,
+    "train_loss": 1.0224046702974203,
+    "train_runtime": 86934.362,
+    "train_samples": 207865,
+    "train_samples_per_second": 8.4,
+    "train_steps_per_second": 0.066
+}

eval_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 5.0,
+    "eval_loss": 1.1089974641799927,
+    "eval_runtime": 435.9692,
+    "eval_samples": 23110,
+    "eval_samples_per_second": 37.081,
+    "eval_steps_per_second": 1.161
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.36.2",
+  "use_cache": false
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20bfe947271ea72b1b6611060a49fc8533c9fb844042b6cb78f23bca6709f708
+size 4984275728

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c83260b2b23c67e2d5667b6a08d688c4b4f898d40cefdd121d819e6362730df6
+size 4991625072

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:663f038f31e648f07c13d679891d516de7964db0ef2b4603de6a49a1045433cc
+size 2882730416

model.safetensors.index.json

@@ -0,0 +1,692 @@
+{
+  "metadata": {
+    "total_size": 12858544128
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00003-of-00003.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.block_sparse_moe.gate.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.block_sparse_moe.gate.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.block_sparse_moe.gate.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "model.norm.weight": "model-00003-of-00003.safetensors"
+  }
+}

runs/Jan18_13-46-53_main1/events.out.tfevents.1705585701.main1.12437.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6151dbf6d1c7ab5e0b454d5bb5bf2d02de492fe0527ab1414cb805bd071b52fb
-size 199238
+oid sha256:2c7e07539500a43918e4e36336350d88ea9afa60b2102dbb44fef65e2d9995d4
+size 199749

runs/Jan18_13-46-53_main1/events.out.tfevents.1705673068.main1.12437.1

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37d5f698e65e3b900a125d181db05f4e1c5a2b5ac4a6815b66b3106fb7536646
+size 359

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 5.0,
+    "train_loss": 1.0224046702974203,
+    "train_runtime": 86934.362,
+    "train_samples": 207865,
+    "train_samples_per_second": 8.4,
+    "train_steps_per_second": 0.066
+}