Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, Yejin Choi
We introduce Social IQa, the first largescale benchmark for commonsense reasoning about social situations. Social IQa contains 38,000 multiple choice questions for probing emotional and social intelligence in a variety of everyday situations (e.g., Q: "Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy. Why did Jordan do this?" A: "Make sure no one else could hear"). Through crowdsourcing, we collect commonsense questions along with correct and incorrect answers about social interactions, using a new framework that mitigates stylistic artifacts in incorrect answers by asking workers to provide the right answer to a different but related question. Empirical results show that our benchmark is challenging for existing question-answering models based on pretrained language models, compared to human performance (>20% gap). Notably, we further establish Social IQa as a resource for transfer learning of commonsense knowledge, achieving state-of-the-art performance on multiple commonsense reasoning tasks (Winograd Schemas, COPA).
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Question Answering | SIQA | Accuracy | 64.5 | BERT-large 340M (fine-tuned) |
| Question Answering | SIQA | Accuracy | 63.1 | BERT-base 110M (fine-tuned) |
| Question Answering | SIQA | Accuracy | 63 | GPT-1 117M (fine-tuned) |
| Question Answering | SIQA | Accuracy | 33.3 | Random chance baseline |
| Question Answering | COPA | Accuracy | 83.4 | BERT-SocialIQA 340M |
| Question Answering | COPA | Accuracy | 80.8 | BERT-large 340M |
| Coreference Resolution | Winograd Schema Challenge | Accuracy | 72.5 | BERT-SocialIQA 340M |
| Coreference Resolution | Winograd Schema Challenge | Accuracy | 67 | BERT-large 340M |