This project explores a novel approach to multimodal information retrieval from multimedia databases. Self-diagnosis and personal health monitoring for skin diseases serves as the motivation driving this research as well as a setting for the evaluation. The multimedia database consists of both images (visual symptoms) and textual information (such as patient history). An individual interacts with the system via a multimodal interface. He/she uploads digital photographs of his/her visual symptoms and enters additional textual information pertaining to their symptoms and general health information. The retrieval consists of (anonymized) cases similar to his/hers (images along with the associated disease identifications/description), their possible prognoses, the need for immediate physician consult, etc. The key to the success of the retrieval engine lies in the novel indexing of the cases on the basis of both image content and text, along with the definition of the similarity metric used for retrieval. This metric must optimize over both image and text content. It needs to be noted that the system that will be explored in this project does not aim to replace the physician. Its aim is to better inform the individuals and assist them in monitoring the symptoms exhibited by their medical condition. The broader impacts of this work include a novel approach to patient participation in the diagnosis process, as well as an individual's life expectancy when early diagnosis is critical for successful treatment. The project's Web site (http://rvl1.ecn.purdue.edu/~cbirdev/multimodal.html) will be used for disseminating fundamental results in multimodal retrieval of multimedia information as well as provide access to the experimental system.
