The rich information generated by computer and human networks creates exciting opportunities for network analytics, namely, the process of gaining knowledge and insights by mining a large amount of network data collected by a diverse set of monitors. To enable effective network analytics, several significant challenges must be addressed: --- (i) Scalability. The enormous scale of computer and human networks makes it challenging to analyze massive network datasets in a scalable fashion. --- (ii) Complexity. Real-world network datasets are complex and often violate the operational conditions of existing analysis techniques. --- (iii) Robustness. Anomalies and imperfections are common in real-world network datasets. --- (iv) Diversity. Network analytics often requires mining information from diverse data sources with different characteristics and data quality.
This research addresses the above challenges by developing a series of novel compressive sensing techniques to effectively exploit the presence of structure and redundancy in real-world network datasets, including: --- (i) clustered spectral graph embedding, a novel technique for reducing a massive graph to a much smaller graph while preserving essential clustering and spectral information of the original graph, --- (ii) LENS decomposition, a novel method for accurately decomposing a network data matrix into a Low-rank matrix, an Error term, a Noise matrix, and a Sparse matrix, and --- (iii) multi-source spectral learning, a novel framework for effectively integrating information from diverse data sources. The research promises to significantly enhance the ability to analyze massive network datasets. The resulting tools and techniques have potential applications in business, information technology, networking and cyber security. Finally, the research includes a significant education and training component. The research results will be integrated into both undergraduate and graduate curricula as well as outreach activities.
The project develops scalable, accurate and robust compressive sensing techniques to enable effective analytics on massive, complex, noisy and diverse real-world network datasets. Intellectual merit: The project develops a series of novel techniques that significantly enhances the ability to analyze massive network datasets. The project also explores the application of compressive sensing techniques to a wide range of application scenarios, including social network analysis, network diagnosis, localization, physical analytics, and click spam detection. The analysis results not only deepen the understanding on the interaction among networked entities but also help broaden the foundation of network science. The tools and techniques resulted from the project have potential applications in business, information technology, networking and cyber security. Several of such technologies are making direct real-world impact through successful technology transfer. For example, the robust network anomaly detection technology is used by the operations team of a tier-1 ISP for network diagnosis, and the click spam detection technology is incorporated into the production system of one of the largest software companies in the world. Broader impact: The project has significant broader impact. The compressive sensing techniques developed in the project are valuable to multiple scientific fields. Through technology transfer and collaboration with the industry, some of the research has been successfully deployed in the real world to benefit the general public. The project produces multiple Ph.D. dissertations (including two female students and several students from under-represented groups). The research results have been incorporated into both undergraduate and graduate courses on computer networks.
