Mukund Sundararajan, Ankur Taly, Qiqi Yan
We study the problem of attributing the prediction of a deep network to its input features, a problem previously studied by several other works. We identify two fundamental axioms---Sensitivity and Implementation Invariance that attribution methods ought to satisfy. We show that they are not satisfied by most known attribution methods, which we consider to be a fundamental weakness of those methods. We use the axioms to guide the design of a new attribution method called Integrated Gradients. Our method requires no modification to the original network and is extremely simple to implement; it just needs a few calls to the standard gradient operator. We apply this method to a couple of image models, a couple of text models and a chemistry model, demonstrating its ability to debug networks, to extract rules from a network, and to enable users to engage with models better.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Interpretability Techniques for Deep Learning | CelebA | Insertion AUC score | 0.3578 | Integrated Gradients |
| Image Attribution | VGGFace2 | Deletion AUC score (ArcFace ResNet-101) | 0.0749 | Integrated Gradients |
| Image Attribution | VGGFace2 | Insertion AUC score (ArcFace ResNet-101) | 0.5399 | Integrated Gradients |
| Image Attribution | CUB-200-2011 | Deletion AUC score (ResNet-101) | 0.0728 | Integrated Gradients |
| Image Attribution | CUB-200-2011 | Insertion AUC score (ResNet-101) | 0.0422 | Integrated Gradients |
| Image Attribution | CelebA | Deletion AUC score (ArcFace ResNet-101) | 0.068 | Integrated Gradients |
| Image Attribution | CelebA | Insertion AUC score (ArcFace ResNet-101) | 0.3578 | Integrated Gradients |