This project addresses a problem which has kept virtual reality from widespread use. Some 15 years ago high-capability graphics cards in PCs reduced the cost of computing for virtual environments from hundreds-of-thousands of dollars to (today) hundreds of dollars. Low-cost head-mounted displays have just appeared. The similar advance in viewpoint tracking has not occurred; accurate, low-latency wide-area viewpoint tracking remains very costly. Virtual reality demands stereo 60 frames per second per eye and system latencies below 50 ms. This research is developing a novel system to provide accurate, low-latency viewpoint tracking to meet these requirements with consumer-cost components. The research is based upon a recently demonstrated proof-of-concept system. A standard RGB-Depth camera sits on the user's head. Pose is calculated by matching images against an environment model. A Kalman filter integrates rotational velocity and linear acceleration from a cheap high-speed inertial measurement unit (IMU) to update the pose estimate many times between frames. This not only gives low-latency pose readings, it also improves initial values for the next camera calculation. The depth images and reconstruction software are concurrently used to incrementally build/update the depth model of the environment for the camera matching.
The current research is demonstrating the system's potential. To work completely successfully, both conceptual and algorithmic advances are in process. IMU calibrations are being improved. Temperature and dynamic bias must be compensated in the calibration to improve stimation and reduce jitter. Using multiple cameras to reduce overall noise and handle difficult cases (such as blank walls) are being addressed with new algorithms and evaluated. The merging of new and modeled data is computationally expensive. The feasibility demo uses two GPUs, one for rendering; one for tracking. Ways are being invented to do it on one. Additional future research includes tracking dynamic objects and incorporating object recognition (e.g., such as a desk or chairs) to improve estimates. Widespread access to virtual reality may well open new, unexpected creative uses of the technology. The research is inventing the proof-of-concept system forward to one that can make this exciting leap.
Immersive virtual reality was proposed by Ivan Sutherland in 1965, and a prototype demonstrated by him in 1968. Over the years, there's been lots of hype, lots of progress, and substantial corporate and military use. Until recently VR systems have been too costly to get much public use. Three things are required for a head-mounted immersive VR system: a binocular head-mounted visual and audio display system, a graphic computer capable of generating complex images at ~30 frames a second with end-to-end system latency of ~50 msec, and a fast system for tracking the position and pose of the two eyes. New head-mounted display systems such as the Oculus Rift have brought high performance and consumer prices. Graphics add-on cards for personal computers have brought fast performance and consumer prices for computational image generation. There has not heretofore been a consumer-price-level solution to the eye and limb tracking problem. This research investigated a two-part solution which used consumer-priced accelerometers and gyroscopes to measure linear and angular accelerations and velocities. These enable dead-reckoning navigation, such as that used to navigate the U.S.S Nautilus under the polar ice to the North Pole. Such navigation is subject to cumulative drift. For a system using consumer-grade sensors, this is a serious problem. This research undertook to harness computer-vision techniques to generate, over hundreds of milliseconds, high accuracy key frames which would provide benchmarks for the correction of drift. A key concept was that the system would as it ran bootstrap up the model of the visible surround from which the key frames would be generated. A running prototype of such a system, without model bootstrapping, was built and a conference paper published. The student doing this work for his Ph.D. thesis decided to join a company interested in doing very similar work. He therefore took his M.S. degree and joined the company full time, thus terminating this grant work.
