arxiv:2409.03752

Attention Heads of Large Language Models: A Survey

Published on Sep 5

· Submitted by

fan2goa1 on Sep 6

#2 Paper of the day

Upvote

Authors:

Zifan Zheng ,

Yezhaohui Wang ,

Yuxin Huang ,

Shichao Song ,

Zhiyu Li

Abstract

Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in various tasks but remain largely as black-box systems. Consequently, their development relies heavily on data-driven approaches, limiting performance enhancement through changes in internal architecture and reasoning pathways. As a result, many researchers have begun exploring the potential internal mechanisms of LLMs, aiming to identify the essence of their reasoning bottlenecks, with most studies focusing on attention heads. Our survey aims to shed light on the internal reasoning processes of LLMs by concentrating on the interpretability and underlying mechanisms of attention heads. We first distill the human thought process into a four-stage framework: Knowledge Recalling, In-Context Identification, Latent Reasoning, and Expression Preparation. Using this framework, we systematically review existing research to identify and categorize the functions of specific attention heads. Furthermore, we summarize the experimental methodologies used to discover these special heads, dividing them into two categories: Modeling-Free methods and Modeling-Required methods. Also, we outline relevant evaluation methods and benchmarks. Finally, we discuss the limitations of current research and propose several potential future directions. Our reference list is open-sourced at https://github.com/IAAR-Shanghai/Awesome-Attention-Heads.

View arXiv page View PDF Add to collection

Community

fan2goa1

Paper author Paper submitter Sep 6

•

edited Sep 6

🚀 We are excited to share our latest survey, “Attention Heads of Large Language Models: A Survey”. In this paper, we delve into the potential mechanisms of how attention heads in Large Language Models (LLMs) contribute to the reasoning process.

🔍 Highlights

We propose an innovative four-stage framework, inspired by human cognitive neuroscience, to analyze the reasoning process of LLMs (Knowledge Recalling, In-Context Identification, Latent Reasoning, Expression Preparation).
We classify current research on the interpretability of LLM attention heads according to the four-stage framework and explore the collaborative mechanisms among them.
We provide a comprehensive summary and classification of the experimental methodologies; also summarize the limitations of current research in this field and propose directions for future research.
We summarize a large number of related papers and have created the Awesome-Attention-Heads repository, aiming to provide valuable reference information for researchers interested in the LLM interpretability.

🔗 Related Links:

GitHub Repository: https://github.com/IAAR-Shanghai/Awesome-Attention-Heads
arXiv Paper: https://arxiv.org/abs/2409.03752

framework

🌟 If this paper proves helpful to you, please consider starring and sharing our work!

Chinese Version

🚀 我们很高兴能与大家分享我们最新的综述《Attention Heads of Large Language Models: A Survey》。在这篇文章中，我们深入探讨了LLMs的注意力头在推理过程中的潜在作用机制。

🔍 本文亮点

结合人类认知神经学提出了一种创新的四阶段框架来分析 LLMs 的推理过程（Knowledge Recalling, In-Context Identification, Latent Reasoning, Expression Preparation）
对当前有关LLMs attention heads可解释性的研究按照四阶段框架进行分类，并阐述了它们之间的协同作用机制
对于实验探究方法进行了全面的汇总和分类；总结当前该领域研究的不足之处，并提出了未来研究方向
分别总结了大量相关论文并创建了Awesome-Attention-Heads仓库；旨在为对LLM可解释性感兴趣的研究人员提供有价值的参考信息

🔗 相关链接

Github仓库: https://github.com/IAAR-Shanghai/Awesome-Attention-Heads
arXiv论文: https://arxiv.org/abs/2409.03752

🌟 如果这篇文章能够帮助到您，欢迎star和分享我们的工作！

meng-shao

Sep 6

Thx for your paper.
Your comprehensive survey on attention heads in LLMs provides invaluable insights into the internal workings of these complex systems.
The four-stage framework you've proposed for analyzing LLM reasoning processes is particularly innovative and offers a fresh perspective for future research in AI interpretability.

shared in X:
https://x.com/shao__meng/status/1831963645057888438