Peyman Milanfar Electrical Engineering Department University of California at Santa Cruz
In this research effort a central challenge in computer vision is addressed: Namely, to recognize and enhance objects in complex visual scenes given imperfect images, and more generally, video data. This effort strengthens the theoretical and practical foundations for generic visual object recognition systems that can deal with significant variations in visual appearance, a large number of categories, and stochastically and systematically degraded data. Data imperfections can include random noise, blur, and environmental degradations. The approach has transformative potential for a broad range of practical applications such as scalable image search and retrieval, automatic annotation, surveillance and security, video forensics, and medical image analysis for computer-aided diagnosis.
The research advances the state-of-the-art in two important ways: (a) a unified and robust framework is derived for both (2-D) object and (3-D) action recognition, even when the data is subject to significant distortions, and (b) recognition and restoration from degraded data are treated in a common, statistically optimal setting. Traditionally, recognition and restoration have been addressed with limited awareness of each other?s techniques and of potential commonalities in approach. By improving, generalizing, and refining previously separate approaches to recognition with degraded data in an adaptive, non-parametric setting, for both 2-D and 3-D, this project contributes to the technical foundations and toolkits that can connect computer vision and image processing intelligently.
During the last year of the project, despite the PI being on leave of absence, the research team has made excellent progress supported by this grant. In total, 4 full length journal and 4 conference papers were published in the highest quality venues. It is also important to note that software for all work performed during this grant were made freely available to the research community. Here is a summary of accomplishments: * Estimating the amount of blur in a given image is important for computer vision applications. More speci?cally,the spatially varying defocus point-spread-functions over an image reveal geometric information of the scene, and theirestimate can also be used to recover an all-in-focus image. APSF for a defocus blur can be speci?ed by a single parameterindicating its scale. Most existing algorithms can only selectan optimal blur from a ?nite set of candidate PSFs for eachpixel. Some of those methods require a coded aperture ?lterinserted in the camera. We derived an algorithm estimating a defocus scale map from a single image, which is applicable to conventional cameras. This method is capable of measuring the probability of local defocus scale in the continuousdomain. It also takes smoothness and color edge informationinto consideration to generate a coherent blur map indicatingthe amount of blur at each pixel. * We studied a general class of nonlinear and shift-varying smoothing ?lters that operate based onaveraging. This important class of ?lters includes many well-known examples such as the bilateral?lter, nonlocal means, general adaptive moving average ?lters, and more. (Many linear ?lters suchas linear minimum mean-squared error smoothing ?lters, Savitzky–Golay ?lters, smoothing splines,and wavelet smoothers can be considered special cases.) They are frequently used in both signal andimage processing as they are elegant, computationally simple, and high performing. The operatorsthat implement such ?lters, however, are not symmetric in general. The main contribution was to provide a provably stable method for symmetrizing the smoothing operators. Speci?cally,we propose a novel approximation of smoothing operators by symmetric doubly stochastic matricesand show that this approximation is stable and accurate, even more so in higher dimensions. Wedemonstrate that there are several important advantages to this symmetrization, particularly inimage processing/?ltering applications such as denoising. In particular, (1) doubly stochastic ?ltersgenerally lead to improved performance over the baseline smoothing procedure; (2) when the ?ltersare applied iteratively, the symmetric ones can be guaranteed to lead to stable algorithms; and (3) symmetric smoothers allow an orthonormal eigendecomposition which enables us to peer into the complex behavior of such nonlinear and shift-varying ?lters in a locally adapted orthogonal basis. * The human visual system possesses the remarkable ability to pick out salient objects in images. Even more impressiveis its ability to do the very same in the presence of disturbances. In particular, the ability persists despite the presenceof noise, poor weather, and other impediments to perfect vision. Meanwhile, noise can signi?cantly degrade the accuracyof automated computational saliency detection algorithms. We set out to remedy this shortcoming. We proposed a novel and statistically sound method forestimating saliency based on a non-parametric regression framework, and investigate the stability of saliency models fornoisy images and analyze how state-of-the-art computational models respond to noisy visual stimuli. The proposed modelof saliency at a pixel of interest is a data-dependent weighted average of dissimilarities between a center patch around thatpixel and other patches. Our method consistently outperforms six other state-of-the-art models Journal publications during the final year: X. Zhu, S. Cohen, S. Schiller, and P. Milanfar, " Estimating Spatially Varying Defocus Blur from A Single Image " IEEE Trans. on Image Processing, To appear Peyman Milanfar, " Symmetrizing Smoothing Filters " SIAM Journal on Imaging Sciences Vol. 6, No. 1, pp. 263–284 Also see related talk at SIAM Imaging Science Conference. H. Talebi and Peyman Milanfar, " How to SAIF-ly Improve Denoising Performance" To appear in IEEE Transactions on Image Processing Also see related project page Chelhwon Kim and Peyman Milanfar, " Visual Saliency in Noisy Images " Journal of Vision ,vol. 13, no. 4 article 5, March 2013. Also see project page . Conference publications during the final year: X. Zhu, and P. Milanfar, " Qpro: An improved no-reference image content metric using locally adapted SVD " Proceedings of Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics, January 2013, Scottsdale, Arizona J. Kotera, F. Sroubek, and P. Milanfar, " Blind deconvolution using alternating maximum a posteriori estimation with heavy-tailed priors " 15th International Conference on Computer Analysis of Images and Patterns , Aug. 2013, York, UK X. Zhu, F. Sroubek, and P. Milanfar, " Deconvolving PSFs for A Better Motion Deblurring using Multiple Images " European Conference on Computer Vision (ECCV), Oct. 2012, Florence, Italy H. Talebi and P. Milanfar, " Improving Denoising Filters by Optimal Diffusion "International Conference on Image Processing (ICIP), Sept. 2012, Orlando, FL
