Description
PipeMare is an asynchronous (bubble-free) pipeline parallel method for training large neural networks. It involves two main techniques: learning rate rescheduling and discrepancy correction.
PipeMare is an asynchronous (bubble-free) pipeline parallel method for training large neural networks. It involves two main techniques: learning rate rescheduling and discrepancy correction.