Qwen

/

Qwen-7B-Chat

like 744

You can refer to the blog of Rope's author, Su Jianlin, at https://spaces.ac.cn/archives/9577. The experimental results indicate that incorporating qkv bias enhances the ability in extrapolating to longer sequences.

可以参阅Rope的作者苏剑林的博客https://spaces.ac.cn/archives/9577 ， 实验结果显示qkv bias有助于提升模型的长度外推能力

Updated model weights and MMLU/CEval scores. Now the benchmark scores are almost on par with the original Qwen-7B-chat.

更新了模型权重和 MMLU/CEval 分数。 现在，它的基准分数几乎与原来的 Qwen-7B-chat 持平。

@JosephusCheung Hi, could you please share how to merge the bias item of c_attn = nn.Linear(config.hidden_size, 3 * self.projection_size) into LLama?

@JosephusCheung Hi, could you please share how to merge the bias item of c_attn = nn.Linear(config.hidden_size, 3 * self.projection_size) into LLama?

One possible solution:

$$W_c=W+\frac{b}{\mathrm{\parallel }x{\mathrm{\parallel }}_2^2}x{x}^T=W+\frac{b_v\times av{g}^T}{n^2}W\_c\; =\; W\; +\; \backslash frac\{b\}\{\backslash |x\backslash |\_2^2\}\; xx^T\; =\; W\; +\; \backslash frac\{b\_\{v\}\; \backslash times\; avg^T\}\{n^2\}$$

@JosephusCheung Could you please show a detailed explanation of the variables in the formula? Or show your code, please?

You can refer to the blog of Rope's author, Su Jianlin, at https://spaces.ac.cn/archives/9577. The experimental results indicate that incorporating qkv bias enhances the ability in extrapolating to longer sequences.

可以参阅Rope的作者苏剑林的博客https://spaces.ac.cn/archives/9577 ， 实验结果显示qkv bias有助于提升模型的长度外推能力

Su have updated his blog, it reads:"【注：后来经过反复测试发现，发现此篇文章的长度外推结果可复现性比较不稳定（可能跟模型结构、超参数等紧密相关），请自行斟酌使用。】"(Note: After repeated testing, it was found that the reproducibility of the length extrapolation results of this article is relatively unstable (may be closely related to the model structure, hyperparameters, etc.), please use it at your own discretion.)