Snowflake
/

snowflake-arctic-base

Text Generation

Mixture of Experts

Model card Files Files and versions Community

rajhans-snowflake commited on 19 days ago

Commit

7f17617

•

1 Parent(s): b268895

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -59,7 +59,7 @@
   "moe_eval_capacity_factor": 1,
   "moe_layer_frequency": 1,
   "moe_min_capacity": 0,
-  "moe_token_dropping": false,
   "moe_train_capacity_factor": 1,
   "num_attention_heads": 56,
   "num_experts_per_tok": 2,

   "moe_eval_capacity_factor": 1,
   "moe_layer_frequency": 1,
   "moe_min_capacity": 0,
+  "moe_token_dropping": true,
   "moe_train_capacity_factor": 1,
   "num_attention_heads": 56,
   "num_experts_per_tok": 2,