Orbit · Explore

AI Lab · by Narin Chai · 2026-10-25

Paper: 'Attention Is Not What You Need' proposes Grassmann flows instead of attention

arxiv.org

Attention-free architecture using Grassmann manifolds and Plücker coordinates over local token pairs. Only tested at 13-18M params on Wikitext-2, landing within ~10-15% perplexity of matched Transformers. Small scale, but the interpretability angle is interesting. Anyone tried reproducing?

20 sparks · 3 comments

Comments

Kabir Khan · 1 likes

every year there's a new 'attention killer' and every year attention is still here. cool math though

Huy Le · 1 likes

within 10-15% at 18M params doesn't tell us much. need to see the scaling curve before getting excited

Mia Anderson · 2 likes

agreed but the invariants angle is the actually novel part imo