Description
Linformer is a linear Transformer that utilises a linear self-attention mechanism to tackle the self-attention bottleneck with Transformer models. The original scaled dot-product attention is decomposed into multiple smaller attentions through linear projections, such that the combination of these operations forms a low-rank factorization of the original attention.
Papers Using This Method
CacheFormer: High Attention-Based Segment Caching2025-04-18LinFormer: A Linear-based Lightweight Transformer Architecture For Time-Aware MIMO Channel Prediction2024-10-28Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity2024-10-09GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction2024-10-01Sumformer: Universal Approximation for Efficient Transformers2023-07-05MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention2022-11-25Treeformer: Dense Gradient Trees for Efficient Attention Computation2022-08-18Linearizing Transformer with Key-Value Memory2022-03-23Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences2021-12-10Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation2021-08-24Vision Xformers: Efficient Attention for Image Classification2021-07-05Styleformer: Transformer based Generative Adversarial Networks with Style Vector2021-06-13Self-supervised Depth Estimation Leveraging Global Perception and Geometric Smoothness Using On-board Videos2021-06-07A Practical Survey on Faster and Lighter Transformers2021-03-26Revisiting Linformer with a modified self-attention with linear complexity2020-12-16Efficient Transformers: A Survey2020-09-14Linformer: Self-Attention with Linear Complexity2020-06-08