This PFI: AIR Technology Translation project aims to build and commercialize a reusable mixed-reality platform that engages children in scientific experimentation and uses computer vision and automated feedback to help them effectively learn scientific principles. The project focuses on translating prior science and technology research that produced the first Intelligent Science Station, called EarthShake. EarthShake is a mixed-reality educational game in which children discover physical properties of balance by running experiments with block towers on a simulated earthquake table. This project has the potential to have substantial societal, commercial and educational impacts. It aims to impact millions of children and families that spend time in Children's and Science Museums, libraries, schools, and indoor play spaces. It will transform traditional hands-on exhibits and play spaces into intelligent learning environments that foster children's curiosity and improve their learning in an engaging and collaborative way. It will also help to transform the traditional educational system in schools, introducing an interactive, engaging, mixed-reality system that can help students understand the underlying evidence for scientific principles, not just memorize them. With over 270 Children's and Science Museums in the US alone, over 800 malls with play spaces for children, 100s of indoor play/learning spaces, over 4,000 clubs and after-school programs, 119,487 libraries, 36,767 private and charter elementary schools and 103,460 elementary schools, this project can have a commercial impact in addition to societal and educational benefit.
The system uses computer vision to follow children's progress and provide intelligent guidance in a predict-observe-explain scientific inquiry process. Experiments demonstrate that grades K to 3 children learn much more from EarthShake than from an otherwise identical flatscreen tablet or laptop implementation showing as much as a 5 times greater pre to post increase on tests of prediction, explanation, and stable tower building. The initial prototype will be translated into a reusable platform for Intelligent Science Stations and demonstrated in scalable development of multiple instances of three games, EarthShake, RaceCars, and BalloonScale. The technical challenges are to generalize the vision algorithm to work across the variety of objects and configurations in these games and create modular hardware components that provide for scale in development, distribution, and new products. This project draws on technical expertise in computer vision/AI, mechatronics, tangible interfaces, physical computing, human/child-computer interaction, and educational technology.
Intelligent Science Stations bridge the advantages of physical and virtual worlds to improve children's science and inquiry learning in an enjoyable and collaborative way and can be commercialized to do so in various settings, indoor play spaces, museums, libraries, and schools. The project contributes to the general robotics challenge by creating an automated intelligent tutoring system operating in the physical 3D world, incorporating sensors and perceptual algorithms to track what learners are doing, actuators to manipulate the scientific apparatus, and an intelligent dialogue system to orchestrate interaction. It addresses technical challenges of creating a generalizable vision algorithm and a modular physical hardware platform that allows customers to easily switch between different applications. It will create new knowledge about how to develop a general and reusable mixed-reality platform that integrates scientific apparatus including mechatronic components (e.g,. sensors, actuators, servo motors, physical experimental set-up) with an intelligent tracking system (including hardware and software components with an AI vision algorithm).
