Description
Movement Pruning is a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. Magnitude pruning can be seen as utilizing zeroth-order information (absolute value) of the running model. In contrast, movement pruning methods are where importance is derived from first-order information. Intuitively, instead of selecting weights that are far from zero, we retain connections that are moving away from zero during the training process.
Papers Using This Method
On Importance of Pruning and Distillation for Efficient Low Resource NLP2024-09-21LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning2023-05-28What Matters In The Structured Pruning of Generative Language Models?2023-02-07Block Pruning For Faster Transformers2021-09-10Movement Pruning: Adaptive Sparsity by Fine-Tuning2020-05-15