Jacek Klimek, Jakub Klimek, Witold Kraskiewicz, Mateusz Topolewski
Various modifications of TRANSFORMER were recently used to solve time-series forecasting problem. We propose Query Selector - an efficient, deterministic algorithm for sparse attention matrix. Experiments show it achieves state-of-the art results on ETT, Helpdesk and BPI'12 datasets.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Time Series Forecasting | ETTh2 (336) Univariate | MAE | 0.371 | QuerySelector |
| Time Series Forecasting | ETTh2 (336) Univariate | MSE | 0.2088 | QuerySelector |
| Time Series Forecasting | ETTh2 (336) Univariate | MAE | 0.3805 | Transformer |
| Time Series Forecasting | ETTh2 (336) Univariate | MSE | 0.2191 | Transformer |
| Time Series Forecasting | ETTh2 (168) Univariate | MAE | 0.3322 | QuerySelector |
| Time Series Forecasting | ETTh2 (168) Univariate | MSE | 0.1753 | QuerySelector |
| Time Series Forecasting | ETTh2 (168) Univariate | MAE | 0.3547 | Transformer |
| Time Series Forecasting | ETTh2 (168) Univariate | MSE | 0.1974 | Transformer |
| Time Series Forecasting | ETTh1 (24) Multivariate | MAE | 0.4627 | QuerySelector |
| Time Series Forecasting | ETTh1 (24) Multivariate | MSE | 0.4226 | QuerySelector |
| Time Series Forecasting | ETTh1 (24) Multivariate | MAE | 0.4788 | Transformer |
| Time Series Forecasting | ETTh1 (24) Multivariate | MSE | 0.4496 | Transformer |
| Time Series Forecasting | ETTh2 (168) Multivariate | MAE | 0.9726 | Transformer |
| Time Series Forecasting | ETTh2 (168) Multivariate | MSE | 1.6225 | Transformer |
| Time Series Forecasting | ETTh2 (168) Multivariate | MAE | 1.0125 | QuerySelector |
| Time Series Forecasting | ETTh2 (168) Multivariate | MSE | 1.7385 | QuerySelector |
| Time Series Forecasting | ETTh1 (24) Univariate | MAE | 0.1616 | QuerySelector |
| Time Series Forecasting | ETTh1 (24) Univariate | MSE | 0.0436 | QuerySelector |
| Time Series Forecasting | ETTh1 (24) Univariate | MAE | 0.183 | Transformer |
| Time Series Forecasting | ETTh1 (24) Univariate | MSE | 0.0548 | Transformer |
| Time Series Forecasting | ETTh2 (720) Multivariate | MAE | 1.3084 | QuerySelector |
| Time Series Forecasting | ETTh2 (720) Multivariate | MSE | 3.0664 | QuerySelector |
| Time Series Forecasting | ETTh2 (720) Multivariate | MAE | 1.3668 | Transformer |
| Time Series Forecasting | ETTh2 (720) Multivariate | MSE | 3.1805 | Transformer |
| Time Series Forecasting | ETTh1 (720) Multivariate | MAE | 0.8399 | Transformer |
| Time Series Forecasting | ETTh1 (720) Multivariate | MSE | 1.108 | Transformer |
| Time Series Forecasting | ETTh1 (720) Multivariate | MAE | 0.8428 | QuerySelector |
| Time Series Forecasting | ETTh1 (720) Multivariate | MSE | 1.115 | QuerySelector |
| Time Series Forecasting | ETTh2 (336) Multivariate | MAE | 1.1859 | QuerySelector |
| Time Series Forecasting | ETTh2 (336) Multivariate | MSE | 2.3168 | QuerySelector |
| Time Series Forecasting | ETTh2 (336) Multivariate | MAE | 1.2189 | Transformer |
| Time Series Forecasting | ETTh2 (336) Multivariate | MSE | 2.6617 | Transformer |
| Time Series Forecasting | ETTh1 (720) Univariate | MAE | 0.373 | QuerySelector |
| Time Series Forecasting | ETTh1 (720) Univariate | MSE | 0.2136 | QuerySelector |
| Time Series Forecasting | ETTh1 (720) Univariate | MAE | 0.4213 | Transformer |
| Time Series Forecasting | ETTh1 (720) Univariate | MSE | 0.2501 | Transformer |
| Time Series Forecasting | ETTh2 (24) Multivariate | MAE | 0.4864 | QuerySelector |
| Time Series Forecasting | ETTh2 (24) Multivariate | MSE | 0.4124 | QuerySelector |
| Time Series Forecasting | ETTh2 (24) Multivariate | MAE | 0.5013 | Transformer |
| Time Series Forecasting | ETTh2 (24) Multivariate | MSE | 0.4237 | Transformer |
| Time Series Forecasting | ETTh2 (24) Univariate | MAE | 0.2239 | QuerySelector |
| Time Series Forecasting | ETTh2 (24) Univariate | MSE | 0.0843 | QuerySelector |
| Time Series Forecasting | ETTh2 (24) Univariate | MAE | 0.2479 | Transformer |
| Time Series Forecasting | ETTh2 (24) Univariate | MSE | 0.0999 | Transformer |
| Time Series Forecasting | ETTh1 (336) Multivariate | MAE | 0.7041 | Transformer |
| Time Series Forecasting | ETTh1 (336) Multivariate | MSE | 0.8321 | Transformer |
| Time Series Forecasting | ETTh1 (336) Multivariate | MAE | 0.7039 | QuerySelector |
| Time Series Forecasting | ETTh1 (336) Multivariate | MSE | 0.8503 | QuerySelector |
| Time Series Forecasting | ETTh1 (168) Multivariate | MAE | 0.6088 | QuerySelector |
| Time Series Forecasting | ETTh1 (168) Multivariate | MSE | 0.6835 | QuerySelector |
| Time Series Forecasting | ETTh1 (168) Multivariate | MAE | 0.6325 | Transformer |
| Time Series Forecasting | ETTh1 (168) Multivariate | MSE | 0.7146 | Transformer |
| Time Series Forecasting | ETTh1 (168) Univariate | MAE | 0.2371 | QuerySelector |
| Time Series Forecasting | ETTh1 (168) Univariate | MSE | 0.0935 | QuerySelector |
| Time Series Forecasting | ETTh1 (168) Univariate | MAE | 0.2539 | Transformer |
| Time Series Forecasting | ETTh1 (168) Univariate | MSE | 0.1049 | Transformer |
| Time Series Forecasting | ETTh1 (48) Multivariate | MAE | 0.4878 | QuerySelector |
| Time Series Forecasting | ETTh1 (48) Multivariate | MSE | 0.4581 | QuerySelector |
| Time Series Forecasting | ETTh1 (48) Multivariate | MAE | 0.4968 | Transformer |
| Time Series Forecasting | ETTh1 (48) Multivariate | MSE | 0.4668 | Transformer |
| Time Series Forecasting | ETTh2 (720) Univariate | MAE | 0.413 | QuerySelector |
| Time Series Forecasting | ETTh2 (720) Univariate | MSE | 0.2585 | QuerySelector |
| Time Series Forecasting | ETTh2 (720) Univariate | MAE | 0.434 | Transformer |
| Time Series Forecasting | ETTh2 (720) Univariate | MSE | 0.2853 | Transformer |
| Time Series Forecasting | ETTh1 (48) Univariate | MAE | 0.2118 | QuerySelector |
| Time Series Forecasting | ETTh1 (48) Univariate | MSE | 0.0721 | QuerySelector |
| Time Series Forecasting | ETTh1 (48) Univariate | MAE | 0.2144 | Transformer |
| Time Series Forecasting | ETTh1 (48) Univariate | MSE | 0.074 | Transformer |
| Time Series Forecasting | ETTh1 (336) Univariate | MAE | 0.2844 | QuerySelector |
| Time Series Forecasting | ETTh1 (336) Univariate | MSE | 0.1267 | QuerySelector |
| Time Series Forecasting | ETTh1 (336) Univariate | MAE | 0.3201 | Transformer |
| Time Series Forecasting | ETTh1 (336) Univariate | MSE | 0.1541 | Transformer |
| Time Series Forecasting | ETTh2 (48) Multivariate | MAE | 0.9317 | QuerySelector |
| Time Series Forecasting | ETTh2 (48) Multivariate | MSE | 1.4074 | QuerySelector |
| Time Series Forecasting | ETTh2 (48) Multivariate | MAE | 0.9488 | Transformer |
| Time Series Forecasting | ETTh2 (48) Multivariate | MSE | 1.522 | Transformer |
| Time Series Forecasting | ETTh2 (48) Univariate | MAE | 0.2622 | QuerySelector |
| Time Series Forecasting | ETTh2 (48) Univariate | MSE | 0.1117 | QuerySelector |
| Time Series Forecasting | ETTh2 (48) Univariate | MAE | 0.2763 | Transformer |
| Time Series Forecasting | ETTh2 (48) Univariate | MSE | 0.1218 | Transformer |
| Time Series Forecasting | BPI challenge '12 | Accuracy | 0.79 | QuerySelector |
| Time Series Forecasting | Helpdesk | Accuracy | 0.743 | QuerySelector |
| Time Series Analysis | ETTh2 (336) Univariate | MAE | 0.371 | QuerySelector |
| Time Series Analysis | ETTh2 (336) Univariate | MSE | 0.2088 | QuerySelector |
| Time Series Analysis | ETTh2 (336) Univariate | MAE | 0.3805 | Transformer |
| Time Series Analysis | ETTh2 (336) Univariate | MSE | 0.2191 | Transformer |
| Time Series Analysis | ETTh2 (168) Univariate | MAE | 0.3322 | QuerySelector |
| Time Series Analysis | ETTh2 (168) Univariate | MSE | 0.1753 | QuerySelector |
| Time Series Analysis | ETTh2 (168) Univariate | MAE | 0.3547 | Transformer |
| Time Series Analysis | ETTh2 (168) Univariate | MSE | 0.1974 | Transformer |
| Time Series Analysis | ETTh1 (24) Multivariate | MAE | 0.4627 | QuerySelector |
| Time Series Analysis | ETTh1 (24) Multivariate | MSE | 0.4226 | QuerySelector |
| Time Series Analysis | ETTh1 (24) Multivariate | MAE | 0.4788 | Transformer |
| Time Series Analysis | ETTh1 (24) Multivariate | MSE | 0.4496 | Transformer |
| Time Series Analysis | ETTh2 (168) Multivariate | MAE | 0.9726 | Transformer |
| Time Series Analysis | ETTh2 (168) Multivariate | MSE | 1.6225 | Transformer |
| Time Series Analysis | ETTh2 (168) Multivariate | MAE | 1.0125 | QuerySelector |
| Time Series Analysis | ETTh2 (168) Multivariate | MSE | 1.7385 | QuerySelector |
| Time Series Analysis | ETTh1 (24) Univariate | MAE | 0.1616 | QuerySelector |
| Time Series Analysis | ETTh1 (24) Univariate | MSE | 0.0436 | QuerySelector |
| Time Series Analysis | ETTh1 (24) Univariate | MAE | 0.183 | Transformer |
| Time Series Analysis | ETTh1 (24) Univariate | MSE | 0.0548 | Transformer |
| Time Series Analysis | ETTh2 (720) Multivariate | MAE | 1.3084 | QuerySelector |
| Time Series Analysis | ETTh2 (720) Multivariate | MSE | 3.0664 | QuerySelector |
| Time Series Analysis | ETTh2 (720) Multivariate | MAE | 1.3668 | Transformer |
| Time Series Analysis | ETTh2 (720) Multivariate | MSE | 3.1805 | Transformer |
| Time Series Analysis | ETTh1 (720) Multivariate | MAE | 0.8399 | Transformer |
| Time Series Analysis | ETTh1 (720) Multivariate | MSE | 1.108 | Transformer |
| Time Series Analysis | ETTh1 (720) Multivariate | MAE | 0.8428 | QuerySelector |
| Time Series Analysis | ETTh1 (720) Multivariate | MSE | 1.115 | QuerySelector |
| Time Series Analysis | ETTh2 (336) Multivariate | MAE | 1.1859 | QuerySelector |
| Time Series Analysis | ETTh2 (336) Multivariate | MSE | 2.3168 | QuerySelector |
| Time Series Analysis | ETTh2 (336) Multivariate | MAE | 1.2189 | Transformer |
| Time Series Analysis | ETTh2 (336) Multivariate | MSE | 2.6617 | Transformer |
| Time Series Analysis | ETTh1 (720) Univariate | MAE | 0.373 | QuerySelector |
| Time Series Analysis | ETTh1 (720) Univariate | MSE | 0.2136 | QuerySelector |
| Time Series Analysis | ETTh1 (720) Univariate | MAE | 0.4213 | Transformer |
| Time Series Analysis | ETTh1 (720) Univariate | MSE | 0.2501 | Transformer |
| Time Series Analysis | ETTh2 (24) Multivariate | MAE | 0.4864 | QuerySelector |
| Time Series Analysis | ETTh2 (24) Multivariate | MSE | 0.4124 | QuerySelector |
| Time Series Analysis | ETTh2 (24) Multivariate | MAE | 0.5013 | Transformer |
| Time Series Analysis | ETTh2 (24) Multivariate | MSE | 0.4237 | Transformer |
| Time Series Analysis | ETTh2 (24) Univariate | MAE | 0.2239 | QuerySelector |
| Time Series Analysis | ETTh2 (24) Univariate | MSE | 0.0843 | QuerySelector |
| Time Series Analysis | ETTh2 (24) Univariate | MAE | 0.2479 | Transformer |
| Time Series Analysis | ETTh2 (24) Univariate | MSE | 0.0999 | Transformer |
| Time Series Analysis | ETTh1 (336) Multivariate | MAE | 0.7041 | Transformer |
| Time Series Analysis | ETTh1 (336) Multivariate | MSE | 0.8321 | Transformer |
| Time Series Analysis | ETTh1 (336) Multivariate | MAE | 0.7039 | QuerySelector |
| Time Series Analysis | ETTh1 (336) Multivariate | MSE | 0.8503 | QuerySelector |
| Time Series Analysis | ETTh1 (168) Multivariate | MAE | 0.6088 | QuerySelector |
| Time Series Analysis | ETTh1 (168) Multivariate | MSE | 0.6835 | QuerySelector |
| Time Series Analysis | ETTh1 (168) Multivariate | MAE | 0.6325 | Transformer |
| Time Series Analysis | ETTh1 (168) Multivariate | MSE | 0.7146 | Transformer |
| Time Series Analysis | ETTh1 (168) Univariate | MAE | 0.2371 | QuerySelector |
| Time Series Analysis | ETTh1 (168) Univariate | MSE | 0.0935 | QuerySelector |
| Time Series Analysis | ETTh1 (168) Univariate | MAE | 0.2539 | Transformer |
| Time Series Analysis | ETTh1 (168) Univariate | MSE | 0.1049 | Transformer |
| Time Series Analysis | ETTh1 (48) Multivariate | MAE | 0.4878 | QuerySelector |
| Time Series Analysis | ETTh1 (48) Multivariate | MSE | 0.4581 | QuerySelector |
| Time Series Analysis | ETTh1 (48) Multivariate | MAE | 0.4968 | Transformer |
| Time Series Analysis | ETTh1 (48) Multivariate | MSE | 0.4668 | Transformer |
| Time Series Analysis | ETTh2 (720) Univariate | MAE | 0.413 | QuerySelector |
| Time Series Analysis | ETTh2 (720) Univariate | MSE | 0.2585 | QuerySelector |
| Time Series Analysis | ETTh2 (720) Univariate | MAE | 0.434 | Transformer |
| Time Series Analysis | ETTh2 (720) Univariate | MSE | 0.2853 | Transformer |
| Time Series Analysis | ETTh1 (48) Univariate | MAE | 0.2118 | QuerySelector |
| Time Series Analysis | ETTh1 (48) Univariate | MSE | 0.0721 | QuerySelector |
| Time Series Analysis | ETTh1 (48) Univariate | MAE | 0.2144 | Transformer |
| Time Series Analysis | ETTh1 (48) Univariate | MSE | 0.074 | Transformer |
| Time Series Analysis | ETTh1 (336) Univariate | MAE | 0.2844 | QuerySelector |
| Time Series Analysis | ETTh1 (336) Univariate | MSE | 0.1267 | QuerySelector |
| Time Series Analysis | ETTh1 (336) Univariate | MAE | 0.3201 | Transformer |
| Time Series Analysis | ETTh1 (336) Univariate | MSE | 0.1541 | Transformer |
| Time Series Analysis | ETTh2 (48) Multivariate | MAE | 0.9317 | QuerySelector |
| Time Series Analysis | ETTh2 (48) Multivariate | MSE | 1.4074 | QuerySelector |
| Time Series Analysis | ETTh2 (48) Multivariate | MAE | 0.9488 | Transformer |
| Time Series Analysis | ETTh2 (48) Multivariate | MSE | 1.522 | Transformer |
| Time Series Analysis | ETTh2 (48) Univariate | MAE | 0.2622 | QuerySelector |
| Time Series Analysis | ETTh2 (48) Univariate | MSE | 0.1117 | QuerySelector |
| Time Series Analysis | ETTh2 (48) Univariate | MAE | 0.2763 | Transformer |
| Time Series Analysis | ETTh2 (48) Univariate | MSE | 0.1218 | Transformer |
| Time Series Analysis | BPI challenge '12 | Accuracy | 0.79 | QuerySelector |
| Time Series Analysis | Helpdesk | Accuracy | 0.743 | QuerySelector |
| Multivariate Time Series Forecasting | BPI challenge '12 | Accuracy | 0.79 | QuerySelector |
| Multivariate Time Series Forecasting | Helpdesk | Accuracy | 0.743 | QuerySelector |