GloVe

GloVe Embeddings are a type of word embedding that encode the co-occurrence probability ratio between two words as vector differences. GloVe uses a weighted least squares objective $J$ that minimizes the difference between the dot product of the vectors of two words and the logarithm of their number of co-occurrences:

$J=\sum\_{i, j=1}^{V}f\left(𝑋\_{i j}\right)(w^{T}\_{i}\tilde{w}_{j} + b\_{i} + \tilde{b}\_{j} - \log{𝑋}\_{ij})^{2}$

where $w\_{i}$ and $b\_{i}$ are the word vector and bias respectively of word $i$, $\tilde{w}_{j}$ and $b\_{j}$ are the context word vector and bias respectively of word $j$, $X\_{ij}$ is the number of times word $i$ occurs in the context of word $j$, and $f$ is a weighting function that assigns lower weights to rare and frequent co-occurrences.