Papers
arxiv:2408.14332
One-layer transformers fail to solve the induction heads task
Published on Aug 26, 2024
Authors:
Abstract
A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2408.14332 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2408.14332 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2408.14332 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a
collection
to link it from this page.