Attention Is Not What You Need
Attention Is Not What You Need (arxiv.org)
要約
本論文では、シーケンスモデリングにおける基本的な問いとして、明示的な自己注意機構が強力な性能や推論に本当に必要かを再検討しています。標準的なマルチヘッドアテンションは、隠れベクトルをペアワイズ相互作用の高次元空間にマッピングし、勾配降下法で学習を進めるテンソルリフティングの一形態と見なせると論じています。これに対し、グラスマンフローに基づくアテンションフリーなアーキテクチャを提案しており、トークン状態を線形削減し、グラスマン多様体上の2次元部分空間としてエンコードし、ゲート付き混合で隠れ状態に融合させます。このアプローチは、Wikitext-2やSNLIなどのベンチマークでTransformerに匹敵する性能を示し、より構造化された幾何学的・不変量ベースの解釈への道を開く可能性を示唆しています。
全文翻訳
Computer Science > Machine Learning
arXiv:2512.19428 (cs)
[Submitted on 22 Dec 2025]
Title:Attention Is Not What You Need
Authors:[Zhang Chong](https://arxiv.org/search/cs?searchtype=author&query=Chong,+Z)
View a PDF of the paper titled Attention Is Not What You Need, by Zhang Chong
Abstract:We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants.
To explore an alternative, we propose an attention-free architecture based on Grassmann flows. Instead of forming an L by L attention matrix, our Causal Grassmann layer (i) linearly reduces token states, (ii) encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and (iii) fuses these geometric features back into the hidden states through gated mixing. Information therefore propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space.
On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head, with best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. We analyze the complexity of Grassmann mixing, show linear scaling in sequence length for fixed rank, and argue that such manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Algebraic Geometry (math.AG) |
| Cite as: | arXiv:2512.19428 [cs.LG] |
| (or arXiv:2512.19428v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.19428 arXiv-issued DOI via DataCite |
Submission history
From: Chong Zhang [view email]
[v1] Mon, 22 Dec 2025 14:29:18 UTC (16 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Attention Is Not What You Need, by Zhang Chong
［画像: license icon］ view license
Additional Features
Current browse context:
cs.LG
Change to browse by:
References & Citations
Loading...
BibTeX formatted citation
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)
- Author
- Venue
- Institution
- Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)