Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, Noah Goodman
Self-supervised learning algorithms, including BERT and SimCLR, have enabled significant strides in fields like natural language processing, computer vision, and speech processing. However, these algorithms are domain-specific, meaning that new self-supervised learning algorithms must be developed for each new setting, including myriad healthcare, scientific, and multimodal domains. To catalyze progress toward domain-agnostic methods, we introduce DABS: a Domain-Agnostic Benchmark for Self-supervised learning. To perform well on DABS, an algorithm is evaluated on seven diverse domains: natural images, multichannel sensor data, English text, speech recordings, multilingual text, chest x-rays, and images with text descriptions. Each domain contains an unlabeled dataset for pretraining; the model is then is scored based on its downstream performance on a set of labeled tasks in the domain. We also present e-Mix and ShED: two baseline domain-agnostic algorithms; their relatively modest performance demonstrates that significant progress is needed before self-supervised learning is an out-of-the-box solution for arbitrary domains. Code for benchmark datasets and baseline algorithms is available at https://github.com/alextamkin/dabs.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Self-Supervised Learning | DABS | Images & Text | 57.5 | Pretraining: None |
| Self-Supervised Learning | DABS | Med. Imaging | 68.1 | Pretraining: None |
| Self-Supervised Learning | DABS | Natural Images | 10.1 | Pretraining: None |
| Self-Supervised Learning | DABS | Sensors | 69.8 | Pretraining: None |
| Self-Supervised Learning | DABS | Speech | 24.9 | Pretraining: None |
| Self-Supervised Learning | DABS | Text | 42.3 | Pretraining: None |
| Self-Supervised Learning | DABS | Images & Text | 54.3 | Pretraining: ShED |
| Self-Supervised Learning | DABS | Med. Imaging | 74.5 | Pretraining: ShED |
| Self-Supervised Learning | DABS | Natural Images | 20.9 | Pretraining: ShED |
| Self-Supervised Learning | DABS | Sensors | 88.7 | Pretraining: ShED |
| Self-Supervised Learning | DABS | Speech | 36.5 | Pretraining: ShED |
| Self-Supervised Learning | DABS | Text | 48.4 | Pretraining: ShED |
| Self-Supervised Learning | DABS | Images & Text | 48.9 | Pretraining: e-Mix |
| Self-Supervised Learning | DABS | Med. Imaging | 72.4 | Pretraining: e-Mix |
| Self-Supervised Learning | DABS | Natural Images | 27.9 | Pretraining: e-Mix |
| Self-Supervised Learning | DABS | Sensors | 79.5 | Pretraining: e-Mix |
| Self-Supervised Learning | DABS | Speech | 41.8 | Pretraining: e-Mix |
| Self-Supervised Learning | DABS | Text | 44.1 | Pretraining: e-Mix |