Qwen

/

Qwen1.5-MoE-A2.7B

like 181

Great job！
I've read your blog and the code for Hugging Face Transformers. In the blog, I came across this description: In the case of the Qwen1.5-MoE-A2.7B model, we have incorporated 4 shared experts to be always activated alongside 60 routing experts with 4 to be activated. These four consistently activated shared experts provide a more adaptive approach. However, in the code, I only saw one shared expert. Why is that?

This is the code from modeling_qwen2_moe.py.

self.gate = nn.Linear(config.hidden_size, config.num_experts, bias=False)
self.experts = nn.ModuleList(
    [Qwen2MoeMLP(config, intermediate_size=config.moe_intermediate_size) for _ in range(self.num_experts)]
)
# Only one shared expert??
self.shared_expert = Qwen2MoeMLP(config, intermediate_size=config.shared_expert_intermediate_size)
self.shared_expert_gate = torch.nn.Linear(config.hidden_size, 1, bias=False)

Again, Thanks for your great job!