Computational photography refers broadly to sensing strategies and algorithmic techniques that enhance or extend the capabilities of digital photography. Representative techniques include high dynamic range (HDR) imaging, flash-noflash imaging, panoramic stitching, and refocusable photography. Although interest in computational photography has steadily increased among graphics and vision researchers, progress has been hampered by the lack of a portable, programmable camera platform with enough image quality and computing power to be used outside the laboratory, i.e. for everyday photography. Similarly, courses in computational photography are offered in dozens of universities nationwide. However, none of these courses provide students with a camera on which they can implement the algorithms currently being published in the literature.
To address these two problems, we are building an open-source camera platform (called Frankencamera) that is portable, self-powered, connected to the Internet, and accommodates SLR-quality lenses and sensors. We also describe a software architecture based on Linux, and an API with bindings for C++, that permits control and synchronization of camera functions at the microsecond time scale. Our API includes pre-capture functions like metering and focusing, an image post-processing pipeline, a user interface toolkit for the viewfinder, and support for current and future I/O devices. Our plan is to distribute this platform at minimal cost to researchers and university instructors nationwide, using the computational photography courses they already teach as a natural distribution vehicle. Instructors will apply to us to be part of this outreach program, and a standing committee will evaluate these applications. Our long-term goal is to spur creation of a community of photographer-programmers who write plug-ins and apps for cameras.
We introduced a new programming language and compiler, Halide, that allow regular programmers to get an order of magnitude speedup for image processing pipelines, while keeping their code concise and easy to maintain. Furthermore, Halide allows the same code to be compiled for regular (x86) CPUS, for GPUS and for the ARM processors found on mobile devices. The key intellectual breakthrough is to decouple the notion of an algorithm (operations that lead to pixel values) from the organization of this computation (in what order and at what spatial granularity is the computation performed and stored). With our compiler, programmers can easily explore different organizations of computation and find a strategy that maximizes parallelism and locality (such as cache coherence) while minimizing redundant computation. We released the compiler free and open source. We developed automatic and user-assisted algorithms that enable the enhancement of tones and color in photographs. Fort his, we gathered a database of photographs together with enhancements performed by trained photographers. This allowed us to train a supervised machine learning algorithm to infer enhancement for arbitrary new input images. The algorithm seeks to predict what our trained users would have done for the new image, and takes into account characteristics of the input photo that are both low level (such as tone distribution) and high level (such as the presence of faces). We also designed a new user-driven approach that allows a photographer to address spatially-varying white balance for scenes with mixed lighting. The photographer can specify information about the scene such as which parts share the same color or which parts are grey, and we use a new propagation approach to extend this information to the whole image in order to remove color cast due to lighting. We developed a technique that can process time lapse videos to remove unwanted motion jitter due to temporal undersampling. We set up a Bayesian optimization method that synthesizes a new video that respects the long-range changes in the video but eliminates high-frequency artifacts. We studied the problem of blind deblurring where we week to remove blur in images where the exact blur is unknown. We introduced an algorithm that is much simpler than previous approaches while providing better results. It is based on a probabilistic formulation and a derivation of efficient approximation for the corresponding variational problem.
