Description
Single-Headed Attention is a single-headed attention module used in the SHA-RNN language model. The principle design reasons for single-headedness were simplicity (avoiding running out of memory) and scepticism about the benefits of using multiple heads.
Papers Using This Method
Meta-Learning-Based Delayless Subband Adaptive Filter using Complex Self-Attention for Active Noise Control2024-12-27SHAQ: Single Headed Attention with Quasi-Recurrence2021-08-18Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization2020-11-01Attention-based Joint Detection of Object and Semantic Part2020-07-05Single Headed Attention RNN: Stop Thinking With Your Head2019-11-26Attention and Lexicon Regularized LSTM for Aspect-based Sentiment Analysis2019-07-01