Internet fraud costs consumers and businesses billions of dollars each year. Through creative combinations of spam and social engineering, attackers regularly lure end users into visiting phishing sites, malware-hosting sites, and scam sites. One popular defense mechanism against Web-based attacks is blacklisting, but today's blacklists suffer from three fundamental deciencies. First, most of them employ a combination of Web crawling and human intervention to infer malicious sites. This adds an inherent delay in adding entries and causes many malicious sites to be missed. Second, blacklists are mostly based on exact URL strings, and hence unable to adapt to simple changes to the URLs that attackers are using today to evade detection. Third, as blacklist entries grow, matching them against URLs in real-time could create performance bottlenecks. To overcome these deficiencies, this project is developing novel mechanisms to aid in the construction, maintenance, and matching of blacklists in real time. Specifically, it is developing a scalable architecture that can discover new malicious websites by passively observing the onset of new techniques exploited by the miscreants, such as redirects and fast flux in network traffic. The architecture also leverages common attacker tendencies to find novel and automated ways of discovering new malicious URLs from existing blacklisted URLs. The final thrust of the project is on developing high-speed approximate matching algorithms for effective in-network blacklisting to match URLs embedded in packets against potentially millions of blacklist entries. If successful, this project will make the Web safer for millions of Internet users.
