Junbum Cha, Kyungjae Lee, Sungrae Park, Sanghyuk Chun
Domain generalization (DG) aims to learn a generalized model to an unseen target domain using only limited source domains. Previous attempts to DG fail to learn domain-invariant representations only from the source domains due to the significant domain shifts between training and test domains. Instead, we re-formulate the DG objective using mutual information with the oracle model, a model generalized to any possible domain. We derive a tractable variational lower bound via approximating the oracle model by a pre-trained model, called Mutual Information Regularization with Oracle (MIRO). Our extensive experiments show that MIRO significantly improves the out-of-distribution performance. Furthermore, our scaling experiments show that the larger the scale of the pre-trained model, the greater the performance improvement of MIRO. Source code is available at https://github.com/kakaobrain/miro.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Domain Adaptation | PACS | Average Accuracy | 96.8 | MIRO (RegNetY-16GF, SWAD) |
| Domain Adaptation | PACS | Average Accuracy | 88.4 | MIRO (ResNet-50, SWAD) |
| Domain Adaptation | Office-Home | Average Accuracy | 83.3 | MIRO (RegNetY-16GF, SWAD) |
| Domain Adaptation | Office-Home | Average Accuracy | 72.4 | MIRO (ResNet-50, SWAD) |
| Domain Adaptation | DomainNet | Average Accuracy | 60.7 | MIRO (RegNetY-16GF, SWAD) |
| Domain Adaptation | DomainNet | Average Accuracy | 47 | MIRO (ResNet-50, SWAD) |
| Domain Adaptation | VLCS | Average Accuracy | 81.7 | MIRO (RegNetY-16GF, SWAD) |
| Domain Adaptation | VLCS | Average Accuracy | 79.6 | MIRO (ResNet-50, SWAD) |
| Domain Adaptation | TerraIncognita | Average Accuracy | 64.3 | MIRO (RegNetY-16GF, SWAD) |
| Domain Adaptation | TerraIncognita | Average Accuracy | 52.9 | MIRO (ResNet-50, SWAD) |
| Domain Generalization | PACS | Average Accuracy | 96.8 | MIRO (RegNetY-16GF, SWAD) |
| Domain Generalization | PACS | Average Accuracy | 88.4 | MIRO (ResNet-50, SWAD) |
| Domain Generalization | Office-Home | Average Accuracy | 83.3 | MIRO (RegNetY-16GF, SWAD) |
| Domain Generalization | Office-Home | Average Accuracy | 72.4 | MIRO (ResNet-50, SWAD) |
| Domain Generalization | DomainNet | Average Accuracy | 60.7 | MIRO (RegNetY-16GF, SWAD) |
| Domain Generalization | DomainNet | Average Accuracy | 47 | MIRO (ResNet-50, SWAD) |
| Domain Generalization | VLCS | Average Accuracy | 81.7 | MIRO (RegNetY-16GF, SWAD) |
| Domain Generalization | VLCS | Average Accuracy | 79.6 | MIRO (ResNet-50, SWAD) |
| Domain Generalization | TerraIncognita | Average Accuracy | 64.3 | MIRO (RegNetY-16GF, SWAD) |
| Domain Generalization | TerraIncognita | Average Accuracy | 52.9 | MIRO (ResNet-50, SWAD) |