Yulia Rubanova, Ricky T. Q. Chen, David Duvenaud
Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these ODE-based models outperform their RNN-based counterparts on irregularly-sampled data.
| Task | Dataset | Metric | Value | Model |
|---|---|---|---|---|
| Imputation | PhysioNet Challenge 2012 | mse (10^-3) | 2.118 | Latent ODE (ODE enc) |
| Imputation | PhysioNet Challenge 2012 | mse (10^-3) | 2.789 | Latent ODE + Poisson |
| Imputation | MuJoCo | MSE (10^2, 50% missing) | 0.285 | Latent ODE (ODE enc) |
| Imputation | MuJoCo | MSE (10^2, 50% missing) | 0.665 | ODE-RNN |
| Time Series Forecasting | MuJoCo | MSE (10^-2, 50% missing) | 1.258 | Latent ODE (ODE enc) |
| Time Series Forecasting | MuJoCo | MSE (10^-2, 50% missing) | 26.463 | ODE-RNN |
| Time Series Forecasting | PhysioNet Challenge 2012 | MSE stdev | 0.05 | Latent ODE + Poisson |
| Time Series Forecasting | PhysioNet Challenge 2012 | mse (10^-3) | 2.208 | Latent ODE + Poisson |
| Time Series Forecasting | PhysioNet Challenge 2012 | MSE stdev | 0.029 | Latent ODE (ODE enc) |
| Time Series Forecasting | PhysioNet Challenge 2012 | mse (10^-3) | 2.231 | Latent ODE (ODE enc) |
| Feature Engineering | PhysioNet Challenge 2012 | mse (10^-3) | 2.118 | Latent ODE (ODE enc) |
| Feature Engineering | PhysioNet Challenge 2012 | mse (10^-3) | 2.789 | Latent ODE + Poisson |
| Feature Engineering | MuJoCo | MSE (10^2, 50% missing) | 0.285 | Latent ODE (ODE enc) |
| Feature Engineering | MuJoCo | MSE (10^2, 50% missing) | 0.665 | ODE-RNN |
| Time Series Analysis | MuJoCo | MSE (10^-2, 50% missing) | 1.258 | Latent ODE (ODE enc) |
| Time Series Analysis | MuJoCo | MSE (10^-2, 50% missing) | 26.463 | ODE-RNN |
| Time Series Analysis | PhysioNet Challenge 2012 | MSE stdev | 0.05 | Latent ODE + Poisson |
| Time Series Analysis | PhysioNet Challenge 2012 | mse (10^-3) | 2.208 | Latent ODE + Poisson |
| Time Series Analysis | PhysioNet Challenge 2012 | MSE stdev | 0.029 | Latent ODE (ODE enc) |
| Time Series Analysis | PhysioNet Challenge 2012 | mse (10^-3) | 2.231 | Latent ODE (ODE enc) |
| Multivariate Time Series Forecasting | MuJoCo | MSE (10^-2, 50% missing) | 1.258 | Latent ODE (ODE enc) |
| Multivariate Time Series Forecasting | MuJoCo | MSE (10^-2, 50% missing) | 26.463 | ODE-RNN |
| Multivariate Time Series Forecasting | PhysioNet Challenge 2012 | MSE stdev | 0.05 | Latent ODE + Poisson |
| Multivariate Time Series Forecasting | PhysioNet Challenge 2012 | mse (10^-3) | 2.208 | Latent ODE + Poisson |
| Multivariate Time Series Forecasting | PhysioNet Challenge 2012 | MSE stdev | 0.029 | Latent ODE (ODE enc) |
| Multivariate Time Series Forecasting | PhysioNet Challenge 2012 | mse (10^-3) | 2.231 | Latent ODE (ODE enc) |