Non-text-based smart learning refers to technology-supported learning that uses (1) non-text features (i.e. visualized information) and (2) adapted learning materials based on the individual?s needs. Fully immersive multi-person virtual reality (MVR) refers to humans located at different places wearing VR devices to join a single virtual room, a classroom with unlimited size, to learn from an instructor. The learning environment is poised to undergo a major reformation, and Virtual Reality (VR) will augment, and possibly replace, the traditional classroom learning environment. The purpose of this research is to discover new smart learning methodologies within the MVR environment using nonintrusive multimodal analysis of physiological measures, including eye movement characteristics, haptic interactions, and brain activities. By doing this research, the project explores new questions on the range of human behaviors (e.g. what they observe and how they interact) within the non-text-based MVR learning environment. Non-verbal physiological measures can be used to assess the individuals? engagement in learning and support the individuals? learning needs in a timely manner. Furthermore, the project includes assisting the children, especially the underrepresented minorities and those in rural, under-developed areas, that will allow the investigation of the physical shifts/changes when a child is learning or not learning, and provide opportunities to explore what the child can learn in specific environments and with specific teaching methods.
The project bridges smart learning pedagogy in MVR with the assessed humans? behaviors and transforms them into a cohesive multimodal analysis model. Multimodal analyses in MVR are composed of two main thrusts. The first thrust is to evaluate the multi-person learning performance through discovering the relationships among the multiple physiological measures and to use the relationships to predict learning performance. The physiological measures that can effectively address the data complexity and variability include, but are not limited to, entropy (i.e. degree of disorder or randomness in the visual interrogation or haptic interaction patterns), visual groupings (i.e. objects or information blocks that were closely interrogated or interacted back and forth more often than others), and cerebral hemodynamics. The second thrust is to develop MVR learning materials and discover non-text-based smart learning scaffolding strategies to improve learning performance. The utilization of the predictive measures paired with behavioral traits will lead to the development of a more personalized training and creation of a more integrated smart learning system to meet the diverse needs of underperforming learners. The research results from the project will be used for professional development of teachers and state-of-the-art learning opportunities to the students, through the Oklahoma Virtual Academic Library (OVAL), a virtual meeting place designed for teaching and teaching teachers.
This project is co-funded by the Established Program to Stimulate Competitive Research (EPSCoR).
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
