Uncovering mesa-optimization algorithms in Transformers
Paper
•
2309.05858
•
Published
•
11
Transformers
-
-
ProPainter: Improving Propagation and Transformer for Video Inpainting
Paper • 2309.03897 • Published • 24 -
Approximating Two-Layer Feedforward Networks for Efficient Transformers
Paper • 2310.10837 • Published • 10 -
CLEX: Continuous Length Extrapolation for Large Language Models
Paper • 2310.16450 • Published • 9 -
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Paper • 2312.00752 • Published • 132 -
Efficient Monotonic Multihead Attention
Paper • 2312.04515 • Published • 6 -
Subobject-level Image Tokenization
Paper • 2402.14327 • Published • 16 -
Scalable Diffusion Models with Transformers
Paper • 2212.09748 • Published • 10
An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels
Paper • 2406.09415 • Published • 47Note 本文挑戰了傳統的局部性是視覺任務中必要的歸納偏置的觀點。通過將每個像素作為Transformer的單位，作者證明了即使在去除局部性偏置的情況下，模型仍然可以達到甚至超越基線模型的性能。這一發現對未來的神經網絡架構設計具有重要的指導意義。
Transformers meet Neural Algorithmic Reasoners
Paper • 2406.09308 • Published • 43Note TransNAR模型成功地將Transformer和NAR的優勢結合，顯著提升了算法推理任務的性能，尤其是在分布外數據上。這表明，通過將不同AI模型的特長結合，可以有效地應對現有模型的不足，開拓新的應用領域。未來的研究可以進一步探索如何在純語言模型中部署類似TransNAR的思想，以提升其廣泛應用的可行性。
-
Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task
Paper • 2406.14213 • Published • 20 -
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
Paper • 2406.16747 • Published • 15