The brain is extremely complex as we know, and involves a complicated interplay between functional infor- mation interacting with a structural (but not static) substrate. Brain imaging technology provides a way to sample various aspects of the brain albeit incompletely, providing a rich set of multitask and multimodal information. The field has advanced significantly in its approach to multimodal data, as there are more studies correlating, e.g. functional and structural measures. However, the vast majority of studies still ignore the joint information among two or more modalities or tasks. Such information is critical to consider as each brain imaging modality reports on a different aspect of the brain (e.g. gray matter integrity, blood flow changes, white matter integrity). The field is still striving to understand how to diagnose and treat complex mental illness, such as schizophrenia, bipolar disorder, depression, and others, and ignoring the joint information among tasks and modalities misses a critical, but available, part of the puzzle. Combining multimodal imaging data is not easy since, among other reasons, the combination of multiple data sets consisting of thousands of voxels or timepoints yields a very high dimen- sional problem, requiring appropriate data reduction strategies. In the previous phase of the project we devel- oped advanced approaches to capture high-dimensional relationships among 2 or more modalities. Our work continues to strongly support the benefits of multimodal data fusion to both provide a more complete picture of brain function and structure, but also to improve our ability to study and predict the impact of complex mental illness. In this new phase of the project, we will focus on methods that can fill some existing gaps, such as the ability to bridge spatial/temporal as well as structural/functional connectivity scales. We also propose a novel framework to integrate unimodal and multimodal features called chromatic fusion, which searches for combina- tions of multimodal `notes' which occupy a unique position in a latent (or dictionary) space. The proposed meth- ods will be validated using simulations, hybrid-data, and large N normative imaging data. Our proposed approach will be thoroughly tested using this large data set which includes multiple illnesses that have overlapping symp- toms and which can sometimes be misdiagnosed and treated with the wrong medications for months or years (schizophrenia, bipolar disorder, and unipolar depression). We will provide open source tools and release data throughout the duration of the project via GitHub and the NITRIC repository, hence enabling other investigators to compare their own methods with our own as well as to apply them to a large variety of brain disorders. 37
The promise of multimodal imaging is clear, and our work has clearly shown the substantial benefits of ex- tracting multimodal features estimated jointly from the data. In this renewal, we focus on some key understudied areas including fusing across vast spatiotemporal and functional/structural connectivity scales as well as devel- oping powerful new frameworks for integrating the resulting information to enable decision making and enable identification of potential targets for further study or possible treatment. Completion of our aims will result in a powerful new toolkit for data fusion of multimodal brain imaging data. 36
Showing the most recent 10 out of 202 publications
