Malware is code with malicious intent that can adversely affect the host on which it executes or the network over which they are transmitted. A malware detector classifies a program as malware or benign. Malware writers continuously test the limitations of malware detectors in an attempt to discover techniques to evade detection. This leads to an arms race, where malware writers find new ways to create malware that are undetected by commercial malware detectors, and where researchers working on malware detection respond by devising new detection techniques. Attackers create new malware using two main approaches: program obfuscation and evolution. There is strong evidence that malware writers are using obfuscation and evolution because the number of new malware families is growing at a much slower rate than the number of malware instances. For example, according to Symantec threat reports, in the first half of 2005 there were 10,866 new virus and worm variants but only 170 new families of malware. This data also indicates that signature-based techniques for malware detection will not be able to cope with the increase in the number of malware instances. Recent results by one of the PIs also suggests that current commercial malware detectors are not resilient to obfuscation and evolution techniques used by malware writers. All this evidence clearly suggests that we need a new approach to malware detection.
We propose to explore behavior-based malware detection: our algorithm focuses on detecting malicious behavior (such as mass-mailing behavior used by certain worms) rather than searching for syntactic patterns. We specify malicious behavior in a formal language and then perform static analysis on the code to determine whether it contains the specified behavior. Prior work by the investigators demonstrated that this behavior-based malware detector can detect families of malware using a single specification. However, there are challenges that need to be addressed in the context of behavior-based malware detection. We propose tasks to address these challenges. Solutions to the proposed tasks will lead to malware detection techniques that will resist evasion techniques used by malware writers better than existing malware detectors. Behavior-based malware detectors can also detect new malware that are variants of old malware..
