Image sensors are becoming ubiquitous in daily life as they are incorporated in future intelligent systems including autonomous navigation, health monitoring, and robotics. A central challenge in these camera-driven applications is the inflexibility of current sensor designs and their consequent energy cost. This project designs a new category of image sensors which exploit hardware -- software co-design to attain better sensing at lower cost. The project advances a vertically-integrated design from the mixed-signal sensor circuitry to the computational architecture and the operating system software support. The project's impacts are the creation of new, flexible image sensor systems that can be used for a variety of visual computing applications. The project further seeks to include broadening access to education and research through curriculum material which emphasize smart cameras of the future, outreach to middle and high school students in a summer program to discover imaging applications, and industry engagement through workshops on software-defined imaging.
The project focuses on designing software-defined image sensors, which offer new dimensions of configurability along with system support and programming abstractions to support application-specific needs. To achieve this, the project focuses on three main objectives: (1) Design and implementation of configurable sensor primitives, including programmable regions of interest with custom exposure, readout, and quantization, (2) architectures to control these sensor primitives, and to accelerate image signal processing and vision workloads, and (3) operating system services for scheduling the memory needs for our new sensor primitives, and OS interfaces that enable low-latency reactive sensor control for key applications. These innovations are evaluated in an integrative evaluation testbed that includes a fabricated software-defined image sensor prototype and a Field Programmable Gate Array (FPGA)-based system to measure energy and performance for a set of end-to-end visual applications.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
