Computational photography refers broadly to sensing strategies and algorithmic techniques that enhance or extend the capabilities of digital photography. Representative techniques include high dynamic range (HDR) imaging, flash-noflash imaging, panoramic stitching, and refocusable photography. Although interest in computational photography has steadily increased among graphics and vision researchers, progress has been hampered by the lack of a portable, programmable camera platform with enough image quality and computing power to be used outside the laboratory, i.e. for everyday photography. Similarly, courses in computational photography are offered in dozens of universities nationwide. However, none of these courses provide students with a camera on which they can implement the algorithms currently being published in the literature.
To address these two problems, we are building an open-source camera platform (called Frankencamera) that is portable, self-powered, connected to the Internet, and accommodates SLR-quality lenses and sensors. We also describe a software architecture based on Linux, and an API with bindings for C++, that permits control and synchronization of camera functions at the microsecond time scale. Our API includes pre-capture functions like metering and focusing, an image post-processing pipeline, a user interface toolkit for the viewfinder, and support for current and future I/O devices. Our plan is to distribute this platform at minimal cost to researchers and university instructors nationwide, using the computational photography courses they already teach as a natural distribution vehicle. Instructors will apply to us to be part of this outreach program, and a standing committee will evaluate these applications. Our long-term goal is to spur creation of a community of photographer-programmers who write plug-ins and apps for cameras.
Photography has under gone an enormous change over the past two decades, as we have shifted from using film to using silicon as our image sensing media. Original "digital" cameras were replica of "analog" film camera – the only change was in the material used in light sensing. Yet, the fact that the image was now essentially a collection of digital values opened up the ability to compute on these values before presenting them to the user, enabling one to "compute" the final image. This greatly increased the capabilities of the photographer, and led to the creation of a new research area called computation photography. Our grant was to further research in this area by creating new computational photography algorithms, and programmable camera platforms that would allow researchers to try these ideas out in the "field." This research has led to a number of user programmable cameras which we called Frankencameras, and software systems that work with them. The need to let the user have tighter control of the image led to the development of Fcam, a new camera application interface. This interface has been adopted by Google (Gcam) and is in their current release of Android. We also created Halide, a new language which made it much easier for users to create high-performance imaging applications. The key insight behind Halide is to separate the description of the algorithm from the scheduling optimizations (loop unrolling, exchange, fusions, vectorization, etc.) that are needed to achieve good performance. Halide has also been picked up and is being used in industry. With the ability for application designers to create interesting new applications, in the fourth version of the Frankencamera, the F4, we merged a high-performance image sensor, micro 4/3 lens, Zynq FPGA, and Android system to create an Android camera where an application programmer can program the image signal processor as well as the code running on the application processors. This allows a designer to create a new computational photography application, and try it out on his "camera" in the same day.
