The Markov chain approximation numerical methods are the ones of current choice for continuous time stochastic control problems. This project will develop analogous algorithms for nonlinear stochastic systems subject to delays. Computations and simulations show that appropriate controls that take the delays into account can improve performance considerably. Yet little is currently available concerning algorithms or convergence proofs for nonlinear problems. Due to the delays in the dynamics, the original model problem is infinite-dimensional, and the memory requirements of the numerical algorithms are a serious concern and cause major technical challenges for both the structure of the algorithms and the proofs of convergence. One must be careful since the delay equations can be very sensitive to model perturbations. Approximations to the original models are being developed to simplify the memory problems for the numerical algorithms, and convergence proofs are being developed. The methods of proof are purely probabilistic, being based on weak convergence techniques, hence are very flexible and general. Furthermore the numerical algorithms can be represented as control problems for Markov chain models, which gives the procedure intuitive meaning. Communications systems are of interest for a long time period, hence the popularity of the ergodic (or average cost per unit time) cost criterion. There is little relevant ergodic theory for the delay case and the problem of effective numerical approximations is almost completely open, from both the analytical and algorithmic perspectives. Extensions to singular and impulsive controls and neutral equations will be carried out.
Nonlinear controlled stochastic dynamical models subject to delays are ubiquitous. They occur in models of internet control (and elsewhere in communications, control, biology, financial economics, etc.) where the control signals are subject to communications delays. Indeed, the effects of communications delays are major concerns in the design of internet protocols for efficient and stable usage. The Principal Investigator?s methods for numerically solving stochastic control problems are widely used (in control systems, economics, financial models) for continuous time problems without delays. This work will extend the range of the methods to general nonlinear stochastic systems with delays. Little is known about such problems at present. The algorithms and analytical methods to be developed will enable much more effective control of such systems and demonstrate the usefulness on concrete problems of current importance.
