HANS

The HANS (Heuristic Analysis for NLI Systems) dataset which contains many examples where the heuristics fail.

Source: Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference