Many of today's applications provide data in different formats (e.g., text, videos) from different kinds of sensors (e.g., tweets, cameras, sensors mounted on robots). There is work on how to search each source of data separately, but this misses the hidden connections between the data across the sources. As an example, in disaster management, collaborative perception would allow searching for locations where the semantic concepts "fire" or "crowds" may be present, by jointly analyzing text and image content from social posts, video from stationary or mobile phone cameras, and the images or videos recorded by UAVs. This project will study how to jointly search across data sources by mapping the information coming from all data sources to a common information space. The project has the potential to increase the utility of social and video surveillance data for tasks that require situational awareness. In the area of disaster response, responders will have a more integrated and holistic view of the situation, so they can better allocate their resources. The project will capitalize on the student diversity at UC Riverside, which is a Hispanic Serving Institution, and thus broaden the participation of under-represented groups in the research process. This project will strengthen and extend the ongoing high school and college outreach activities of the PIs.
The goal of this project is to create the knowledge to facilitate effective and efficient collaborative perception on top of a set of independent and multi-modal data generating agents. The project will study how to jointly model social and sensor data and use this modeling to efficiently support spatio-temporal queries on the joint embedding space. In addition to mapping information from multi-modal disparate sources to a common information space, this project will study how to optimize the attention routing of controllable agents like UAVs to maximize the reliability and coverage of the collected information.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
