Carnegie Mellon University Sieworek This effort is a collaboration between the Quality of Life Technology (QoLT) Center, an NSF Engineering Research Center (ERC) since 2006, and Myomo, a small business partner of the ERC. Myomo has been developing and testing neuro-robotic devices in leading US hospitals. The Quality of Life Technology (QoLT) Center has been developing a technology called Virtual Coach to coach exercises, to encourage general physical activity, and to ensure proper use of assistive technologies.
The goal of the proposed collaboration is to enable therapists to provide support to their patients in a virtual context, i.e., without actually being present, through the use of computer-based surrogates. This Virtual Coach will complement another promising approach, neuro-robotic therapy, that helps patients re-learn how to move partially paralyzed (hemi-paretic) arms using their own muscle signals. This project has two objectives. The translational research objective is to specialize QoLT Virtual Coach technology to realize a neuro-robotic therapy monitoring and reporting capability. Myomo will commercialize the resulting Virtual Coach prototype, thereby broadening its product line to include valued companion software for its new line of mPower hardware devices. The exploratory research objective is to make the Virtual Coach more autonomous, i.e., to have it behave more like a human therapist through functions like self-adjusting the therapeutic regimen and responding to non-verbal cues from the patient during exercise sessions.
Intellectual Merit
The research addresses the significant challenge in computer-based coaching of devising algorithms that affect human behavior in an intended way. It is difficult to predict how people will react to commands, instructions, prompts and rewards. Some people respond well to nurturing, while others respond better to commands. In addition a particular individual may respond differently from day-to-day. To be a viable, pervasive solution, Virtual Coach devices will need to tune themselves to the psychology of the people they serve. The research also addresses the challenge of designing interactions between humans and sophisticated technologies for users who themselves may not be technologically inclined. This is particularly relevant for products that would most benefit older adults, who are generally among the least versed in, and sometimes intimidated by, computers, the Internet, smartphones, etc
Broader Impact
Stroke occurs in 2.9% of the US adult population and there are nearly 800,000 new or recurring cases annually. Neuromuscular impairment is a common consequence that can be overcome with appropriate therapy. The new product that will result form this project will alleviate some of the need for face-to-face therapy sessions, will require less time from qualified care providers, and potentially lower the total cost for therapy for stroke victims.
A Virtual Coach was developed that evaluates and offers corrections and feedback for rehabilitation of stroke survivors. The Virtual Coach is composed of a tablet for clinician programming, a Kinect for monitoring motion, and a machine learning model to evaluate the quality of the exercise. Based on a set of rehabilitation exercises established by clinicians employed by our small business partner, Myomo, Inc., (e.g., bring a cup up to the mouth, lift an object from the floor and chair, one/two arm stand up and sit back, walking, etc.) we established the correct and most typical erroneous exercise postures and movements for those exercises. The virtual coach can operate in conjunction with the Myomo mPower, a robotic power assist arm brace that senses contraction and relaxing of arm muscles to trigger a motor providing power assistance. A normalized Hidden Markov Model (HMM) was trained to recognize correct and erroneous postures and exercise movements. Parameters for the HMM were selected to be normalized to height and distance between joints as measured in an exercise calibration session. Feedback for typical problems with each exercise (e.g. go faster, lift higher, keep your posture straight, repetitions are too slow/fast) was solicited from therapists and built into the audio and graphical feedback to the patient. Summaries of performance and progress that can be easily interpreted by clinicians and patients are also provided. A game was developed based on one exercise motion (wiping motion) to control a virtual reality hang glider. By positioning targets different sequences of motion could be encouraged. The goal of the game was not only to make routine exercises more fun thereby increasing exercise duration but also to practice coordinated motion between the two hands. The following outcomes have resulted from the project: Design and implementation of the Virtual Coach for Stroke Rehabilitation Exercises that evaluates and offers corrections and feedback for rehabilitation of stroke survivors. The Virtual Coach prototype that is composed of a tablet for clinician programming, a Kinect for monitoring motion, and a machine learning model (Hidden Markov Model) to evaluate the quality of the exercise. This Virtual Coach was demonstrated to clinicians and patients at four rehabilitation hospitals in and around Pittsburgh, Pennsylvania. A game was developed based on one exercise motion (wiping motion) to control a virtual reality hang glider. A Game Suite concept was developed and seeded with two games that have a connected storyline, artwork and animations, providing feedback to users about their performance. The Game Suite can be combined with Virtual Coach for Stroke Rehabilitation to offer the choice of exercising by using either a coach or game approach. The Virtual Coach can be used for home-based stroke recovery therapy.
