ReGLU

GeneralIntroduced 20002 papers
Source Paper

Description

ReGLU is an activation function which is a variant of GLU. The definition is as follows:

ReGLU(x,W,V,b,c)=max⁡(0,xW+b)⊗(xV+c)\text{ReGLU}\left(x, W, V, b, c\right) = \max\left(0, xW + b\right) \otimes \left(xV + c\right)

Papers Using This Method

ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs2024-02-06GLU Variants Improve Transformer2020-02-12