The identification of systems with high noise levels is very challenging, since models of such systems are typically plagued by high model variance. This lead to higher expected prediction errors. This project will investigate two promising new approaches for reducing model variance and thus the variance of prediction errors: (1) New classes of smoothing regularizes for both feedforward and recurrent networks for reducing model variance while imposing desirable model biases. The PI expect, that his new smoothing regularizes will outperform standard quadratic weight decay, and ad hoc methods, in many cases of interest. (2) New committee bootstrap methods for reducing the prediction errors due to model variance. These include independent bootstrapping of training and validation sets within the committee, mutual training and model selection methods, and robust adaptive committees. The PI expects that his new committee bootstrap methods will achieve better training, better model selection, and greater variance reduction than is attainable be individual networks or by conventional committee averaging methods. The research will involve new analytical work, algorithm development, and extensive empirical testing of the algorithms on noisy time series prediction problems n macroeconomics, physiology, and engineering.
