The objective of this research is to develop new principles for creating and comparing models of skilled human activities, and to apply those models to systems for teaching, training and assistance of humans performing these activities. The models investigated will include both hybrid systems and language-based models. The research will focus on modeling surgical manipulations during robotic minimally invasive surgery. Models for expert performance of surgical tasks will be derived from recorded motion and video data. Student data will be compared with these expert models, and both physical guidance and information display methods will be developed to provide feedback to the student based on the expert model.
The intellectual merit of this work lies in the development of a new set of mathematical tools for modeling human skilled activity. These tools will provide new insights into the relationship between skill, style, and content in human motion. Additional intellectual merit lies in the connection of hybrid systems modeling to language models, the creation of techniques for automated training, and in the assessment of new training methods.
The broader impact of this research will be the creation of automated methods for modeling and teaching skilled human motion. These methods will have enormous implications for the training and re-training of the US workforce. This project will also impact many diversity and outreach activities, including REU programs and summer camps for K-12 outreach. The senior personnel of this project also participate in the Robotic Systems Challenge and the Women in Science and Engineering program.
This project investigated methods for modeling the performance of skilled, complex manipulation tasks using hand movement and video data. For example, modeling surgical tasks by decomposing them into sequences of surgical gestures (which we call surgemes) while assessing the skill level of a surgeon at performing those tasks. The key idea is to first build language-like models that describe the hand movements necessary to perform a skilled task, followed by analysis of the hand movements to determine if there are structures that are indicative of skilled movement. The same ideas can be applied to many different types of movement ranging from dexterous manipulation to whole body motion. We have focused specifically on data that we have acquired from a da Vinci Surgical System (Intuitive Surgical, Inc., Sunnyvale, CA) while surgeons with differing levels of training and experience performed a series of standardized training tasks. With this data, we have developed and evaluated a number of models and implemented a number of applications. With regards to modeling, we began with traditional hidden Markov models (HMMs), which are widely used in speech and language processing by machines. We adapted and improved these models for our domain of interest in several ways. Our most successful methods were based on adapting the notion of dictionary learning for these models. A dictionary is a small set of typical mini-movements that are highly descriptive of all of the observed hand movements. We represent the actual data in terms of the dictionary. We then learn the HMMs from these dictionary-based representations. We also extended these modeling methods to perform joint learning of the dictionary and the HMM simultaneously. We showed that these methods, which we call Sparse HMMs (S-HMMs), outperformed all prior methods in terms of recognition accuracy across a variety of surgical tasks including suturing, needle passing, and knot tying. We then applied the models that we developed to skill classification tasks. We did so by asking an expert to provide a "ground truth" label for the execution of a given task on a numeric scale, with 1 being poor, and 5 being excellent. We then disaggregated the data into three groups which we called novice, intermediate, and expert, and we learned a model, as described above, for each group. We then classified held-out data by determining which of the three models best described the held out data. We found that when the system had a chance to observe the surgeon in question during training, it was able to classify his or her skill with 94-97% accuracy, but when it did not, the accuracy dropped to 40-50%. This suggests that the classifier associates high skill with the surgical styles of the particular surgeons it has seen before, and mislabels a skilled movement performed in a different style as unskilled. This problem is most easily cured by acquiring larger data sets that allow the classifier to see a wider range of individual variability. We have also used time-series data models to develop new modes of training and assistance. for performing surgical tasks. In one study, we used the data to learn a HMM for the task, as described above. We then annotated the states of the HMM to indicate what portions of the task an operator would perform manually, versus portions of the task that the robot would perform automatically. For the automatic portions, the training data was averaged to produce a nominal trajectory for the robot. During execution, the robot observed the actions of the operator and when it recognized that it was transitioning to an automated action, took control and carried out the action. As a result, the operator was able to perform the task more efficiently, but with the same fidelity, through collaboration with the robot. In the course of this project, we have trained four graduate students and two postdoctoral scholars. We have also engaged several undergraduates in our research. We have developed a robust collaboration with faculty at the school of medicine, and we are continuing to work with those faculty to translate the methods described above into the surgical teaching and training environment. Our impact has been quite broad – we have published over a dozen papers in the technical literature as well as complementary publications in the medical literature. The Principal Investigators of this project have also given many invited seminars that describe this unique collaboration of engineering and doctors aimed at improving the practice of medicine. There is of course still a long way to go in translating these ``laboratory’’ advances into practice. The PI’s, in collaboration with educators in the School of Medicine, are pursuing support from agencies such as the National Institutes of Health to study how best to incorporate
