The amount of available data and its dimensionality is soaring, and by now, images are one of the most widely used modalities for sharing impressions. This makes easy-to-use image editing capabilities increasingly important. However present day editing techniques are either very simple or designed for expert users which have a detailed understanding of image properties. This project contributes to an understanding of easy to use algorithms for complex image editing, which we refer to as visual speculation. Examples include attribute transformation (change a scene recorded in summer to what it could have looked like during winter), colorization (producing a color image from a monochrome input), and reshading (changing the illumination of the image). These tasks have numerous applications by producing controllable and photorealistic images for different purposes such as virtual/augmented reality content generation. The project integrated with education through curriculum development and supporting graduate/undergraduate student research.
This research studies methods for visual speculation, i.e., applying complex image editing tasks which modify existing pictures to fit the users? desire while looking real. Examples include: attribute transformation, colorization, and reshading. In each case, solutions are widely ambiguous and good methods generate a diverse range of plausible solutions while offering control to the user. These requirements pose challenges for existing methods which struggle to provide diversity while retaining control. For example, straightforward conditioning based on control variables causes artifacts because the amount of data per instance is no longer sufficient. To alleviate those issues, the proposed work develops high-diversity, high-quality generation models by augmenting state-of-the-art deep generative machinery with image representations that expose data sharing opportunities. The insights obtained from this work result in image editing capabilities that provide high-level control mechanisms.
