The search for relevant and actionable information is key to achieving clinical and research goals in biomedicine. Biomedical information exists in different forms: as text and illustrations in journal articles and other documents, in images stored in databases, and as patients cases in electronic health records. In the context of this work, image refers not only to biomedical images, but also to illustrations, charts, graphs, and other visual material appearing in biomedical journals, electronic health records, and other relevant databases. We are seeking better ways to retrieve information from these entities, by moving beyond conventional text-based searching to combining both text and visual features in search queries. To meet these objectives, we use a combination of techniques and tools from the fields of Information Retrieval (IR), Content-Based Image Retrieval (CBIR), and Natural Language Processing (NLP). The first objective is to improve the retrieval of biomedical literature by targeting the visual content in articles, a rich source of information not typically exploited by conventional bibliographic or full-text databases. We index these figures (including illustrations and images) using (i) text in captions and where they are mentioned in the body of the article (mentions), (ii) image features, such as color, shape, size, etc., and, if available, (iii) annotation markers within figures such as arrows, letters or symbols that are extracted from the image and correlated with concepts in the caption. These annotation markers can help isolate regions of interest (ROI) in images, the ROI being useful for improving the relevance of the figures retrieved. It is hypothesized that augmenting conventional search results with relevant images offers richer search results. A second objective is to find semantically similar images in image databases, an important step in differential diagnosis. We explore approaches that automatically combine image and text features in contrast to visual decision support systems (for example, VisualDx) that use only text driven menus. Such menu driven systems guide a physician to describe a patient and then present a set of images from which a clinician can select the ones most similar to the patients, and access relevant information manually linked to the images. Our methods use text and image features extracted from relevant components in a document, database, or case description to achieve our objectives. For the document retrieval task, we rely on the U.S. National Library of Medicine (NLM) developed search engine. This is a phrase-based search engine with NLMs Unified Medical Language System (UMLS) based term and concept query expansion and probabilistic relevancy ranking that exploits document structure. Optimizing these features, we create structured representations of every full-text document and all its figures. These structured documents presented to the user as search results include typical fields found in MEDLINE citations (e.g., titles, abstracts and MeSH terms), the figures in the original documents, and image-specific fields extracted from the original documents (such as captions segmented into parts pertaining to each pane in a multi-panel image, ROI described in each caption, and the modality of the image). In addition, patient-oriented outcomes extracted from the abstracts are provided to the user. Automatic image annotation and retrieval are achieved in the following ways: (i) using image analysis; (ii) by indexing the text assigned to images; and (iii) using a combination of image and text analysis. Additional steps include describing an image with visual features, automatically detecting its modality (for example, CT, MR, X-ray, ultrasound, etc.), and generating a visual ontology, i.e., concepts assigned to image patches. Elements from the visual ontology are called visual keywords and are used to find images with similar concepts. To evaluate and demonstrate our techniques, we have developed Open-i (pronounced open eye, available at http://openi.nlm.nih.gov), a hybrid system combining text-based searching with an image similarity engine. The Open-i system enables users to search for and retrieve citations that are enriched with relevant images and bottom line (or take away) statements extracted from a collection of approximately 1.2 million open access articles and nearly 3.7 million illustrations from the biomedical literature hosted at the National Library of Medicine's PubMed Central repository; over 8,000 radiology images and 4,000 radiology examination reports from the Indiana University collection of chest x-rays; 67,517 images from NLM History of Medicine collection; and about 2,064 orthopedic anatomy illustrations provided by Norris Medical Library, University of Southern California. Each enriched citation is linked to PubMed Central, PubMed, MedlinePlus as well as to the article itself at the publisher's Web site. A user may search by text words, as well as by query images. Using this framework we explore alternative approaches to search for information using a combination of visual and text features: (i) starting a text-based search of an image database, and refining the search using image features; (ii) starting a visual search using a clinical image of a given patient, and then linking the image to relevant information found by using visual and text features; (iii) starting a multimodal search that combines text and image features. Open-i indexes all the text and illustrations in medical articles by features, both textual and image-based. Open-i also indexes a collection of 8000 digital chest x-rays and accompanying radiology reports with an aim to provide easy access to publicly available and de-identified patient records, as well as the orthopedic and historical images. To compute text and image features efficiently, the system is built on a high performance distributed computing platform. As the first and perhaps only production-quality system of its kind in the biomedical domain, Open-i has enabled medical professionals and the public to access visual information from biomedical articles that are highly relevant to their query, as well as the take away messages of the articles. The quality of the information delivered by Open-i has been evaluated in international competitions, in which the system consistently ranks among the best. For example, the system placed first in the 2013 image retrieval evaluation that attracted participants from academia, industry and clinical settings. For the past year the site has attracted over 10,000 unique visitors daily (excluding bots) with 690,000 hits daily and is able to support searches of vast multimedia collections. During the FY2017, the Open-i suite of services was joined by MedPix https://medpix.nlm.nih.gov/home, a searchable online database of medical images, teaching cases and clinical topics, integrating images and textual metadata including over 19,000 patient case scenarios and nearly 54,000 images. MedPix is organized by disease location (organ system), pathology category, patient profiles, and by image classification and captions. The collection is searchable by keywords, image types, authors, and many other search options. In addition to searching and browsing images and cases, the MedPix website provides free AMA Category 1 CME credits online. Registered users can submit interesting cases to Open-i, MedPix, or both using a dedicated Case Upload Server https://cup.nlm.nih.gov/login. Additionally this year we explored or advanced algorithms for summarizing video, labeling images, panel segmentation, and visual question answering.
| Kalpathy-Cramer, Jayashree; de Herrera, Alba García Seco; Demner-Fushman, Dina et al. (2015) Evaluating performance of biomedical image retrieval systems--an overview of the medical image retrieval task at ImageCLEF 2004-2013. Comput Med Imaging Graph 39:55-61 |
| Rahman, Md Mahmudur; Antani, Sameer K; Demner-Fushman, Dina et al. (2015) Biomedical image representation approach using visualness and spatial information in a concept feature space for interactive region-of-interest-based retrieval. J Med Imaging (Bellingham) 2:046502 |
| Simpson, Matthew S; You, Daekeun; Rahman, Md Mahmudur et al. (2015) Literature-based biomedical image classification and retrieval. Comput Med Imaging Graph 39:3-13 |
| Demner-Fushman, Dina; Antani, Sameer; Kalpathy-Cramer, Jayashree et al. (2015) A decade of community-wide efforts in advancing medical image understanding and retrieval. Comput Med Imaging Graph 39:1-2 |
| Demner-Fushman, Dina; Kohli, Marc D; Rosenman, Marc B et al. (2015) Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc : |
| Rahman, Md Mahmudur; Antani, Sameer K; Thoma, George R (2011) A learning-based similarity fusion and filtering approach for biomedical image retrieval using SVM classification and relevance feedback. IEEE Trans Inf Technol Biomed 15:640-6 |
| Stanley, R Joe; De, Soumya; Demner-Fushman, Dina et al. (2011) An image feature-based approach to automatically find images for application to clinical decision support. Comput Med Imaging Graph 35:365-72 |
| Simpson, Matthew S; Demner-Fushman, Dina; Thoma, George R (2010) Evaluating the Importance of Image-related Text for Ad-hoc and Case-based Biomedical Article Retrieval. AMIA Annu Symp Proc 2010:752-6 |
| Demner-Fushman, Dina; Antani, Sameer; Simpson, Matthew et al. (2009) Annotation and retrieval of clinically relevant images. Int J Med Inform 78:e59-67 |
