Marius Dragoi, Elena Burceanu, Emanuela Haller, Andrei Manolache, Florin Brad
Analyzing the distribution shift of data is a growing research direction in nowadays Machine Learning (ML), leading to emerging new benchmarks that focus on providing a suitable scenario for studying the generalization properties of ML models. The existing benchmarks are focused on supervised learning, and to the best of our knowledge, there is none for unsupervised learning. Therefore, we introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. This type of data meets the premise of shifting the input distribution: it covers a large time span ($10$ years), with naturally occurring changes over time (eg users modifying their behavior patterns, and software updates). We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. Next, we propose AnoShift, a protocol splitting the data in IID, NEAR, and FAR testing splits. We validate the performance degradation over time with diverse models, ranging from classical approaches to deep learning. Finally, we show that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical training which assumes independent and identically distributed data (on average, by up to $3\%$ for our approach). Dataset and code are available at https://github.com/bit-ml/AnoShift/.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Anomaly Detection | AnoShift | ROC-AUC FAR | 50.42 | COPOD |
| Anomaly Detection | AnoShift | ROC-AUC IID | 85.62 | COPOD |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 54.24 | COPOD |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 80.89 | COPOD |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 49.57 | OC-SVM |
| Anomaly Detection | AnoShift | ROC-AUC IID | 76.86 | OC-SVM |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 71.43 | OC-SVM |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 68.73 | OC-SVM |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 49.35 | SO-GAAL |
| Anomaly Detection | AnoShift | ROC-AUC IID | 50.48 | SO-GAAL |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 54.55 | SO-GAAL |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 49.9 | SO-GAAL |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 49.19 | ECOD Li et al. (2022) |
| Anomaly Detection | AnoShift | ROC-AUC IID | 84.76 | ECOD Li et al. (2022) |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 44.87 | ECOD Li et al. (2022) |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 79.41 | ECOD Li et al. (2022) |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 34.96 | LOF |
| Anomaly Detection | AnoShift | ROC-AUC IID | 91.5 | LOF |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 79.29 | LOF |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 87.61 | LOF |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 34.53 | deepSVDD |
| Anomaly Detection | AnoShift | ROC-AUC IID | 92.67 | deepSVDD |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 87 | deepSVDD |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 88.24 | deepSVDD |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 28.19 | LUNAR |
| Anomaly Detection | AnoShift | ROC-AUC IID | 85.75 | LUNAR |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 49.03 | LUNAR |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 78.53 | LUNAR |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 28.15 | BERT |
| Anomaly Detection | AnoShift | ROC-AUC IID | 84.54 | BERT |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 86.05 | BERT |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 79.62 | BERT |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 27.16 | IsoForest |
| Anomaly Detection | AnoShift | ROC-AUC IID | 86.09 | IsoForest |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 75.26 | IsoForest |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 81.27 | IsoForest |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 22.45 | Internal Contrastive Learning |
| Anomaly Detection | AnoShift | ROC-AUC IID | 84.86 | Internal Contrastive Learning |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 52.26 | Internal Contrastive Learning |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 66.99 | Internal Contrastive Learning |
| Anomaly Detection | AnoShift | ROC-AUC FAR | 19.96 | AE for anomalies |
| Anomaly Detection | AnoShift | ROC-AUC IID | 81 | AE for anomalies |
| Anomaly Detection | AnoShift | ROC-AUC NEAR | 44.06 | AE for anomalies |
| Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 64.08 | AE for anomalies |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 50.42 | COPOD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 85.62 | COPOD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 54.24 | COPOD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 80.89 | COPOD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 49.57 | OC-SVM |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 76.86 | OC-SVM |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 71.43 | OC-SVM |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 68.73 | OC-SVM |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 49.35 | SO-GAAL |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 50.48 | SO-GAAL |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 54.55 | SO-GAAL |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 49.9 | SO-GAAL |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 49.19 | ECOD Li et al. (2022) |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 84.76 | ECOD Li et al. (2022) |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 44.87 | ECOD Li et al. (2022) |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 79.41 | ECOD Li et al. (2022) |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 34.96 | LOF |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 91.5 | LOF |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 79.29 | LOF |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 87.61 | LOF |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 34.53 | deepSVDD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 92.67 | deepSVDD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 87 | deepSVDD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 88.24 | deepSVDD |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 28.19 | LUNAR |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 85.75 | LUNAR |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 49.03 | LUNAR |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 78.53 | LUNAR |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 28.15 | BERT |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 84.54 | BERT |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 86.05 | BERT |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 79.62 | BERT |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 27.16 | IsoForest |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 86.09 | IsoForest |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 75.26 | IsoForest |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 81.27 | IsoForest |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 22.45 | Internal Contrastive Learning |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 84.86 | Internal Contrastive Learning |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 52.26 | Internal Contrastive Learning |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 66.99 | Internal Contrastive Learning |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC FAR | 19.96 | AE for anomalies |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC IID | 81 | AE for anomalies |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC NEAR | 44.06 | AE for anomalies |
| Unsupervised Anomaly Detection | AnoShift | ROC-AUC-ID (In-Distribution setup) | 64.08 | AE for anomalies |