CrowS-Pairs
CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups.
Source: CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Benchmarks
Stereotypical Bias Analysis/GenderStereotypical Bias Analysis/ReligionStereotypical Bias Analysis/Race/ColorStereotypical Bias Analysis/Sexual OrientationStereotypical Bias Analysis/AgeStereotypical Bias Analysis/NationalityStereotypical Bias Analysis/DisabilityStereotypical Bias Analysis/Physical AppearanceStereotypical Bias Analysis/Socioeconomic statusStereotypical Bias Analysis/Overall