PMLM
Probabilistically Masked Language Model
Description
Probabilistically Masked Language Model, or PMLM, is a type of language model that utilizes a probabilistic masking scheme, aiming to bridge the gap between masked and autoregressive language models. The basic idea behind the connection of two categories of models is similar to MADE by Germain et al (2015). PMLM is a masked language model with a probabilistic masking scheme, which defines the way sequences are masked by following a probabilistic distribution. The authors employ a simple uniform distribution of the masking ratio and name the model as u-PMLM.
Papers Using This Method
From General to Specific: Tailoring Large Language Models for Personalized Healthcare2024-12-20Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model2021-10-29Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order2020-04-24