The design and use of information systems to support the collaborative activity of collocated teams in dynamic, high-risk scenarios remains a challenge. This project will develop novel methods to more efficiently capture and communicate this activity in environments that currently rely on human observation, verbal communication, and collective memory. More efficient teamwork capture processes will enable both larger-scale collection (which supports retrospective analysis that is critical for improved training and technology design) and contemporaneous collection (which provides real-time feedback to workers to assist in error detection).
To achieve these goals, domain-specific knowledge and probabilistic reasoning will be used to identify patterns of work and communication. The representative domain of trauma resuscitation is ideal for this work since the roles and tasks of players are well-defined and the flow of work follows a general schema regardless of the patient?s injuries. Because of the complexity of this environment, manual tracking of all activities using video recordings requires repeated review and is very time-consuming even for experienced observers. A computer system will be developed that uses video analysis to determine the location of each player, motion analysis to track their movements, and speech recognition targeted at a limited lexicon to identify their communication. Using these inputs, a probabilistic reasoning model will be constructed that correlates data from the environment with a domain-specific model of teamwork. The tagged recording of the resuscitation event will be available in real time during the event as well as post-event for analysis.
The scientific importance of this work is in the need to tag these video observations. Many forms of videos are of repetitive behaviors, whether in surveillance applications, work situations, or other uses. In all such cases, applying a grammar to the video, and matching actions and sounds to that grammar, has the possibility of greatly simplifying work analysis, which is the critical phase in the development computer support for complex, high-risk human activities.
The proposed approach will develop novel algorithms and methods for: (i) person and resource tracking in crowded collaborative environments; (ii) recognition of human activity based on fusion of unreliable data from multimodal sensors and a model of the process being recorded; and (iii) reasoning about human activities at different time scales based on heterogeneous technologies (Hidden Markov Models, Bayesian Nets, and Petri Nets) that mutually interact for activity and event detection. Moreover, the methods will be developed and evaluated in a clinical environment that currently uses limited information technology.
Broader Impacts. This work will also provide the foundation for implementing decision aids in environments such as trauma resuscitation and related medical domains that lack effective methods for instrumented tracking of teamwork. Trauma care is a significant health care crisis and any improvements in resuscitation processes will save lives.
