This Small Business Innovation Research (SBIR) Phase I project aims to demonstrate the feasibility of a 3D gestural user input system for computer aided design (CAD) that enables engineering design firms to more rapidly explore and produce new designs. This technology uses commodity depth-sensing cameras and applies computer vision algorithms to deliver markerless finger-precise hand-tracking, allowing users to simply reach out with their hands to directly manipulate virtual 3D objects. In order to confirm feasibility, the objectives of the project include data gathering, investigation of computer vision algorithms, investigation of 3D input usability issues, and a thorough technical and usability evaluation. More specifically, a data set of 3D scans of hand shapes and hand poses will be collected. Next, a detailed investigation will be performed of multi-view registration and database-driven hand pose estimation algorithms for rapid calibration and tracking of a user's hands. The success of the proposed technology will crucially depend on its delivery of a superior user experience, and the usability issues regarding 3D input will be carefully analyzed. Finally, a detailed technical and usability evaluation will be performed, and the commercial viability will be based on the results.
The broader impact/commercial potential of this project is in the area of engineering design. In today's market place, the competitive capacity of an engineering firm is determined by its ability to rapidly design new products and evaluate new engineering solutions. Most engineering design firms use computer aided design software to accelerate this process. However, the productivity of CAD engineers is still bottlenecked by the user interface and the user input devices available to them. This is because many CAD tasks involve 3D manipulation, which is difficult and tedious to perform with a 2D mouse. A typical CAD user spends four to eight hours a day in CAD software. Over a million CAD users fit this profile. The proposed technology makes these users more productive by enabling them to directly manipulate virtual CAD objects with their bare hands in 3D, at their desks---a technical feat that has never before been achieved. The result is faster product and engineering design in a variety of industries including manufacturing, construction and architecture which spent $6B on CAD software in 2011. This research will advance scientific and technological understanding by creating a fundamentally new way to interact with computers for productivity applications.
We have developed technology that enables rich gestural interaction with a computer much like the futuristic interfaces shown in the movies "Minority Report" and "Iron man." Our technology begins with finger-precise tracking of the user's hands with a commodity depth sensing camera (such as the Microsoft Kinect). Finger and hand motions are then interpreted as gestures which can in turn drive actions in different computer applications. We developed the hand-tracking and gestural recognition algorithms with two specific applications in mind: 3D manipulation for computer aided design (CAD) and navigating medical images in the sterile field of a medical operating room. Computer aided design (CAD) plays a crucial role in mechanical engineering, industrial design, architecture and construction. Practically any new product that is mass-produced is first designed in 3D in CAD software. Our technology enables versatile 3D manipulation for CAD. Using our technology, a CAD engineer can literally grab virtual objects and place them in a 3D design space. This accelerates the design process for the one million users who spendan average of 6 hours each day in CAD design. Image-guided surgery is now an integral part of modern surgery. Before almost any major surgical operation, the patient is imaged through x-ray or magentic resonance. These CT / MRI scans are then used to plan and guide the surgery, serving as a detailed map for the operation. Our technology enables surgeons to directly manipulate this map in a sterile operating room without touching a mouse, keyboard or a touchscreen. The result is faster, more efficient access to critical medical information without the risk of contamination. During the NSF SBIR Phase I effort, we succeeded in developing a finger-precise hand-tracking technology. We developed a system that tracks the hands with spatial precision of under 1mm and angular precision of one degree. We have made the technology free to academics as well as to individuals for commercial use. We now have more than 350 licensees across the world and several paying customers. In addition, we are working with two of our users / clients in pilot studies to deploy the technology in a hospital. We are also continuing to develop a CAD plugin with the help of SolidWorks, one of the leading providers of CAD software in the United States. In the Phase IB effort, we reduced the hardware requirements of our technology to a single depth sensing camera (from two) while also improving the fidelity of index-finger tracking. In the next year, our goal is to work with several of our partners to embed our technology into a variety of products, including a medical device, a car, and a PC.
