Crystalcareai
/

GemMoE-Beta-1

Text Generation

Model card Files Files and versions Community

Crystalcareai commited on Mar 13

Commit

d90f4c5

•

1 Parent(s): c99aadf

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -28,7 +28,7 @@
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,
   "rope_theta": 10000.0,
-  "router_aux_loss_coef": 0.001,
   "sliding_window": null,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.38.2",

   "rms_norm_eps": 1e-06,
   "rope_scaling": null,
   "rope_theta": 10000.0,
+  "router_aux_loss_coef": 0.02,
   "sliding_window": null,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.38.2",