AI★★★★arXiv · 2026-07-08
The Key to Going Linear: Analysis-Driven Transformer Linearization
The quadratic cost of causal self-attention in transformers hinders their inference ability in long contexts. Existing linearization pipelines struggle to preserve model quality. This work proposes an analysis-driven linearization approach, which investigates the effect of state update design on transformer linearization under a strict frozen-backbone regime.
📌 Key points
- Analysis-driven linearization can improve the efficiency of transformer lineariz
- Softmax relies on key-dependent, rank-1 orthogonal projections
- Structural interventions can reduce approximation errors
本页为 gitzw.com 基于公开来源的 AI 中文解读，非原文转载。