Mobile Music Touch (MMT), developed in prior work by the PI and his colleagues, is a lightweight, wireless tactile music instruction system consisting of fingerless gloves and a Bluetooth-enabled mobile phone. Piano passages to be learned are loaded into the mobile phone and played repeatedly in the user's earpiece while s/he performs other tasks. As each note of the song plays, a vibrator on the appropriate finger in the gloves activates, indicating which finger is used to play the note. MMT users showed significant improvement in their ability to reproduce simple piano melodies, even though they were actively engaged in a reading comprehension task during practice. The PI terms this effect Passive Tactile Learning (PTL). He also has initial evidence for a related effect, Passive Tactile Rehabilitation (PTR), where tactile stimulation of the hand correlates to improvements in dexterity and sensation for people with tetraplegia resulting from incomplete spinal cord injury. In this project, the PI will explore PTL/PTR effects in a variety of applications, develop procedural and device design guidelines to best elicit these effects, and illustrate methods and metrics that are sensitive to detecting these effects.
Using MMT, he will compare users' learning of piano melodies while being distracted by a math-intensive test in four conditions: vibration stimulation with accompanying audio, vibration alone, audio alone, and no stimulation. This study will quantify the PTL effect and establish if vibration alone is sufficient to cause a learning effect. In an effort to broaden the applicability of the method, he will study the use of PTL to aid in learning stenographic typing and dance steps. To explore the possibility of PTR, he will run a 12-18 person controlled study with participants with tetraplegia due to partial spinal cord injury. This "in-the-wild" PTL/PTR study will compare active practice on a piano with active practice augmented with MMT. Participants will wear the glove during their everyday lives for at least 2 hours a day. The PI will compare learning rates as well as changes in participants' scores on standard sensation and dexterity tests. And he will also conduct a wearability study on MMT for persons with spinal cord injury to better accommodate their needs.
Broader Impacts: Establishing guidelines for the use and effectiveness of PTL and PTR will enable others to apply the concepts in different domains, e.g., learning sign language or manual procedures such as a pre-flight checklist or training users of prosthetic limbs in how to trigger different actions in their limb. Beyond partial spinal cord injury, PTR might also be exploited for recovery from damage from stroke or lesions due to Multiple Sclerosis.
The PI maintains an aggressive outreach program. Beyond professional publication of results in both the Human Computer Interaction and Rehabilitation literature, he will leverage his contacts to present the work in mainstream press and media (e.g., CNN), as well as more niche communities such as the Georgia Radio Relay Service. One of his focuses is exposing children in Georgia area schools for the deaf to his research, as there is a distinct need for encouraging deaf children in STEM areas. This effort will also be part of the undergraduate "Device Thread" in Georgia Tech's College of Computing, which exposes undergraduates to the rigors of designing computing for situations with physical device limitations, such as mobile phones, wearable devices, and robotics. Graduates and undergraduates in these classes will help investigate and fabricate new PTL/PTR devices. Finally, the PI intends to use this effort in his Computing4Good campaign, which inspires students to think of computing as a means for enabling social change.
