The Directorate of Social, Behavioral and Economic Sciences offers postdoctoral research fellowships to provide opportunities for recent doctoral graduates to obtain additional training, to gain research experience under the sponsorship of established scientists, and to broaden their scientific horizons beyond their undergraduate and graduate training. Postdoctoral fellowships are further designed to assist new scientists to direct their research efforts across traditional disciplinary lines and to avail themselves of unique research resources, sites, and facilities, including at foreign locations. This postdoctoral fellowship supports a rising scientist in the interdisciplinary area overlapping computer vision and psychology, with a research project that investigates the web of relationships within visual data in both humans and machines. To a human observer, no photograph is an island: it is connected to the rest of the visual world by a web of similarities, associations, and other relationships. For example, two photos of Paris share a certain similarity; images of boats are associated with images of water; a photo of a tadpole and a photo of a frog show the same organism at two stages of life. In each of these cases, a human can readily reason about the link between two images. Not only can people identify that a relationship exists, but can also identify the nature of this relationship. These relationships shed light on how the human brain organizes visual information, and also give insight into how to build intelligent systems that automatically make visual connections. The latter will bring the field closer to producing an intelligent visual web, able to organize visual information in the same way as the current Internet is able to organize text.
Computer vision scientists and psychologists have both studied relationships between visual data, but from different directions. In computer vision, the focus has been on models of natural image similarity. These models handle complex stimuli but are usually limited to one simple kind of relationship, namely similarity in appearance. Psychologists have studied a richer set of relationships - association, causation, analogy, antonymy, transformation, etc. - but their models usually only apply to simple, artificial stimuli. This project unites the best of both fields by modeling subtle visual relationships between complex, natural images. The objective is to model both which images humans consider to be related and how are those images related. An additional objective is to study how certain relationships, such as visual associations, can arise in an unsupervised manner from natural visual experience. This will help explain how humans might learn about the relationships in the first place. Better models of inter-image relationships will have deep implications across cognitive psychology. In particular, similarity and association play fundamental roles in theories of human learning and memory. A sense of similarity underlies our ability to learn from one visual experience and then apply our knowledge in a future, similar setting. The associations made from the experience additionally impact human memory of it. The present project also has applications toward computer vision systems. Reverse image search has recently become a popular tool. However, current systems are only able to retrieve look-alike images. If the computer is instead able to retrieve images linked by more diverse kinds of relationships, many possibilities open up. For example, one could imagine a system that lets users navigate through artistic styles, or that recommends shoes that match a pair of pants. If successful, this project could pave the way toward a world-wide web of visual connections that parallels the current web of hypertext connections.
