The success of streaming video has generated interest in newer forms of multi-perspective video content, such as those generated by 360-degree cameras, multi-angle camera arrays, or light-field cameras. The immersive experience provided by these cameras can enhance user satisfaction in domains including sports, training (e.g., construction safety), and virtual exploration (e.g., college walkthroughs, historical sites). With content from these cameras, users do not just passively consume content, but may interactively traverse the content along many different paths from a perspective of their choice, with different users observing different perspectives of the same content. To support this at Internet-scale is challenging; client players must be able to switch perspectives with low latency, the perspectives may need to be generated on demand by video servers, and the infrastructure must support a variety of devices for capture and consumption of this content.
This project explores architectural enhancements, algorithms, and techniques to deliver multi-perspective video at Internet-scale. It couples delivery optimization with video coding and human computer interaction. The project will develop interactivity abstractions by which content publishers can specify the range of perspectives users are permitted to choose from at each point in the video. The interactivity specified in a video will (i) drive perspective coding and novel dynamic perspective generation algorithms; (ii) enable infrastructure provisioning to meet Content Delivery Network (CDN) storage and cost constraints; and (iii) will guide adaptive perspective retrieval from CDN servers or from nearby caches. Finally, the project will explore methods to predict and guide user behavior to improve delivery quality based on user studies.
By enabling more responsive and predictable user experience for multi-perspective video at Internet scale, and by designing networks that can cope with growth in such video, the project will impact the media, IT, education, and networking industries. It will convene an Industrial Advisory Board comprising relevant practitioners and organize workshops to foster collaborations. The research will not only extensively involve graduate students, but also undergraduate students from underrepresented communities, will create new curriculum material, as well as a new undergraduate design project that involves students ranging from the freshman to senior year with strong participation from women and underrepresented students.
The results from the project will be available at: https://engineering.purdue.edu/~isl/multi-perspective-video.html and https://nsl.usc.edu/projects/multi-perspective-video/ The results will be available throughout the duration of the project, and for three years after the project ends.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
