Tasks SotA Datasets Papers Methods Submit About

Papers With Code 2

A community resource for machine learning research: papers, code, benchmarks, and state-of-the-art results.

Explore

Notable Benchmarks All SotA Datasets Papers Methods

Community

Submit Results About

Data sourced from the PWC Archive (CC-BY-SA 4.0). Built by the community, for the community.

Methods/Neighborhood Attention

Neighborhood Attention

GeneralIntroduced 200021 papers

Description

Neighborhood Attention is a restricted self attention pattern in which each token's receptive field is limited to its nearest neighboring pixels. It was proposed in Neighborhood Attention Transformer as an alternative to other local attention mechanisms used in Hierarchical Vision Transformers.

NA is in concept similar to stand alone self attention (SASA), in that both can be implemented with a raster scan sliding window operation over the key value pair. However, NA would require a modification to handle corner pixels, which helps maintain a fixed receptive field size and an increased number of relative positions.

The primary challenge in experimenting with both NA and SASA has been computation. Simply extracting key values for each query is slow, takes up a large amount of memory, and is eventually intractable at scale. NA was therefore implemented through a new CUDA extension to PyTorch, NATTEN.

Papers Using This Method

Attention on the Sphere2025-05-16 Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light2025-04-23 Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention2025-02-19 D3RM: A Discrete Denoising Diffusion Refinement Model for Piano Transcription2025-01-09 PIGUIQA: A Physical Imaging Guided Perceptual Framework for Underwater Image Quality Assessment2024-12-20 EdgeNAT: Transformer for Efficient Edge Detection2024-08-20 Compression-Realized Deep Structural Network for Video Quality Enhancement2024-05-10 A Point-Based Approach to Efficient LiDAR Multi-Task Perception2024-04-19 DeblurDiNAT: A Compact Model with Exceptional Generalization and Visual Fidelity on Unseen Domains2024-03-19 Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level2024-03-07 NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding2023-12-12 SDLFormer: A Sparse and Dense Locality-enhanced Transformer for Accelerated MR Image Reconstruction2023-08-08 ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer2023-06-09 Dilated-UNet: A Fast and Accurate Medical Image Segmentation Approach using a Dilated Transformer and U-Net Architecture2023-04-22 Incorporating Transformer Designs into Convolutions for Lightweight Image Super-Resolution2023-03-25 StyleNAT: Giving Each Head a New Perspective2022-11-10 OneFormer: One Transformer to Rule Universal Image Segmentation2022-11-10 Dilated Neighborhood Attention Transformer2022-09-29 V$^2$L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval2022-07-26 GAF-NAU: Gramian Angular Field encoded Neighborhood Attention U-Net for Pixel-Wise Hyperspectral Image Classification2022-04-21