With recent advances in computer vision and widespread availability of capable hardware, augmented reality has become a consumer-grade technology. It has enabled a new paradigm of digital interaction using camera-based devices, presenting novel ways to interact with the real world. Simultaneously, another emerging trend is the prevalence of data exploration in the real world. One challenge in exploring real-world data is that it is often found in isolation, lacking any contextual information. Further, such data is commonly found in ad-hoc formats, making it hard to combine with digital data. Building on these trends, this project aims to enable augmented reality (AR)-driven discovery and decision making over ad-hoc real-world data. This approach towards data exploration and enrichment using AR-based interaction will open up possibilities in a wide variety of domains, including education, healthcare, architecture design, and emergency services. AR broadens the scope of participation for groups who would otherwise be unable to use traditional computing interfaces such as keyboards, addressing challenges of literacy and motor abilities.
This project will develop a principled framework for exploration and understanding of real-world data using AR. The primary goals of this project are as follows. As a first step, the project will design a new grammar and visual encoding that allows for both querying and layering of structured data results in AR space. Secondly, it will devise methods to enrich real-world images and video streams with schema-rich information using cloud-based data stores by considering not just optical character recognition, but also visual features and context. Third, the project will develop a data storage and query processing backend that is optimized specifically for highly interactive, AR-specific workloads. The outcomes of this research will not only advance the understanding of building data exploration systems for AR, but it will additionally help reduce the barrier to entry for software developers to build data-rich AR applications.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
