All Things ViTs: Understanding and Interpreting Attention in Vision (CVPR'23 tutorial)

By: Hila Chefer and Sayak Paul

Website: all-things-vits.github.io/atv/

Code samples: all-things-vits/code-samples

Abstract: The attention mechanism has revolutionized deep learning research across many disciplines starting from NLP and expanding to vision, speech, and more. Different from other mechanisms, the elegant and general attention mechanism is easily adaptable and eliminates modality-specific inductive biases. As attention becomes increasingly popular, it is crucial to develop tools to allow researchers to understand and explain the inner workings of the mechanism to facilitate better and more responsible use of it. This tutorial focuses on understanding and interpreting attention in the vision and the multi-modal setting. We present state-of-the-art research on representation probing, interpretability, and attention-based semantic guidance, alongside hands-on demos to facilitate interactivity. Additionally, we discuss open questions arising from recent works and future research directions.

This organization hosts all the interactive demos to be presented at the tutorial. Below, you can find some of them.