Finance and Economics Discussion Series (FEDS)
January 2025
Missing Data Substitution for Enhanced Robust Filtering and Forecasting in Linear State-Space Models
Dobrislav Dobrev and Paweł J. Szerszeń
Abstract:
Replacing faulty measurements with missing values can suppress outlier-induced distortions in state-space inference. We therefore put forward two complementary methods for enhanced outlier-robust filtering and forecasting: supervised missing data substitution (MD) upon exceeding a Huber threshold, and unsupervised missing data substitution via exogenous randomization (RMDX).
Our supervised method, MD, is designed to improve performance of existing Huber-based linear filters known to lose optimality when outliers of the same sign are clustered in time rather than arriving independently. The unsupervised method, RMDX, further aims to suppress smaller outliers whose size may fall below the Huber detection threshold. To this end, RMDX averages filtered or forecasted targets based on measurement series with randomly induced subsets of missing data at an exogenously set randomization rate. This gives rise to regularization and bias-variance trade-off as a function of the missing data randomization rate, which can be set optimally using standard cross-validation techniques.
We validate through Monte Carlo simulations that both methods for missing data substitution can significantly improve robust filtering, especially when combined together. As further empirical validation, we document consistently attractive performance in linear models for forecasting inflation trends prone to clustering of measurement outliers.
Keywords: Kalman filter, outliers, Huberization, missing data, randomization
DOI: https://doi.org/10.17016/FEDS.2025.001
PDF: Full Paper
Disclaimer: The economic research that is linked from this page represents the views of the authors and does not indicate concurrence either by other members of the Board's staff or by the Board of Governors. The economic research and their conclusions are often preliminary and are circulated to stimulate discussion and critical comment. The Board values having a staff that conducts research on a wide range of economic topics and that explores a diverse array of perspectives on those topics. The resulting conversations in academia, the economic policy community, and the broader public are important to sharpening our collective thinking.