Modeling Naive Psychology of Characters in Simple Commonsense Stories
Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, Yejin Choi
Abstract
Understanding a narrative requires reading between the lines and reasoning about the unspoken but obvious implications about events and people's mental states - a capability that is trivial for humans but remarkably hard for machines. To facilitate research addressing this challenge, we introduce a new annotation framework to explain naive psychology of story characters as fully-specified chains of mental states with respect to motivations and emotional reactions. Our work presents a new large-scale dataset with rich low-level annotations and establishes baseline performance on several new tasks, suggesting avenues for future research.
Results
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Text Classification | ROCStories | F1 | 30.29 | NPN + Explanation Training |
| Emotion Classification | ROCStories | F1 | 30.29 | NPN + Explanation Training |
| Classification | ROCStories | F1 | 30.29 | NPN + Explanation Training |
Related Papers
NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech2025-07-17MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs2025-05-30What About Emotions? Guiding Fine-Grained Emotion Extraction from Mobile App Reviews2025-05-29EmoBench-UA: A Benchmark Dataset for Emotion Detection in Ukrainian2025-05-29Emotion Classification In-Context in Spanish2025-05-26University of Indonesia at SemEval-2025 Task 11: Evaluating State-of-the-Art Encoders for Multi-Label Emotion Detection2025-05-22The Super Emotion Dataset2025-05-21EmoGist: Efficient In-Context Learning for Visual Emotion Understanding2025-05-20