SVAMP
Simple Variations on Arithmetic Math word Problems
TextsMITIntroduced 2021-03-12
A challenge set for elementary-level Math Word Problems (MWP). An MWP consists of a short Natural Language narrative that describes a state of the world and poses a question about some unknown quantities.
The examples in SVAMP test a model across different aspects of solving MWPs: 1) Is the model question sensitive? 2) Does the model have robust reasoning ability? 3) Is it invariant to structural alterations?
Benchmarks
Math Word Problem Solving/Execution AccuracyMath Word Problem Solving/AccuracyMathematical Question Answering/Execution AccuracyMathematical Question Answering/AccuracyMathematical Reasoning/Execution AccuracyMathematical Reasoning/AccuracyQuestion Answering/Execution AccuracyQuestion Answering/Accuracy