The integrity of commodity operating system kernels is threatened by rootkits that modify key kernel data structures to achieve a variety of malicious goals. While rootkits have historically been known to affect control data in the kernel, recent work demonstrates rootkits that affect system security by modifying non-control data, such as linked lists used to manage bookkeeping information and metadata used for memory management. Existing techniques fail to detect such rootkits effectively.
This project is developing techniques to provide real-time protection against rootkits by detecting anomalies in both control and non-control kernel data behavior using automatically-generated integrity specifications. This goal is being achieved in two steps. First, a technique to mine specifications of kernel data structure integrity is being developed. These specifications are be mined automatically as data structure invariants. Second, these techniques are being extended using operating system support to provide real-time detection.
Impacts and Results: The techniques developed in this project will defend against the next generation of rootkits, and will enable real-time detection of such rootkits. In addition, techniques to infer kernel invariants may also find applications in operating system reliability, fault tolerance and software engineering. The PIs will disseminate the results by releasing the tools developed. The results of this project will equip the workforce with an inter-disciplinary toolkit, that combines operating systems, computer security, and software engineering, to address the challenges posed by the next generation of stealth malware.
This project developed new techniques to detect malicious software that infects the operating system kernel. Such malicious software are also called kernel-level rootkits (or simply, rootkits), and are particularly dangerous because they can remain steathly for extended periods of time. By directly infecting the operating system kernel, rootkits are able to evade detection by user-space security monitors such as commercial anti-virus tools. As a consequence, attackers are increasingly beginning to use rootkits as a stepping stone to deploy and hide sophisticated forms of malicious software (e.g., the Torpig and Storm botnets used rootkits to hide the presence of malicious user-space agents). This project developed Gibraltar, a novel rootkit detector that uses data structure invariants as the core mechanism to detect infections. The main intuition underlying Gibraltar is that the data structures used by a benign (i.e., uninfected) operating system kernel often follow certain properties that hold for the lifetime of the data structure -- these properties are called data structure invariants. Rootkits that maliciously modify the kernel tend to achieve their malicious goals by modifying data structures, thereby violating invariants. For example, a rootkit that injects malicious code must modify a kernel function pointer to reference the injected code, thereby violating the invariant that the function pointer must reference only pre-loaded kernel code. Gibraltar developed techniques to both infer data structure invariants in uninfected operating system kernels, as well as to enforce (i.e., check) invariants in potentially rootkit-infected operating system kernels. Gibraltar was used to detect 25 previously-known rootkits. The work describing Gibraltar was awarded outstanding student paper at the 24th Annual Computer Security Applications Conference (ACSAC 2008). This project also investigated the threats of rootkits on personal computing devices such as smartphones. In a HotMobile 2010 article, the PIs together with their students demonstrated several rootkits that were targetted towards smartphones, and explored the possibility of using these rootits to steal personal information, such as conversations, location, etc. This work received wide-spread publiclity in the popular press, in addition to being the subject of an NSF press release and meet-the-press event in April 2010. In response to the threat of rootkits on mobile devices, this project explored the applicability of tools such as GIbraltar to mobile devices. This work explored the energy and security tradeoffs involved in executing a rootkit detection tool such as Gibraltar on a mobile device. The main findings were that directly applying a tool such as Gibraltar to a smartphone would cut the battery life of the smartphone by 50% (i.e., battery life reduces to half). Consequently, the project also developed new techniques to navigate the energy-security tradeoff curve by investigating the possibility of reducing energy consumption by scanning fewer data structures and scanning less often. The main finding was that with an appropriate choice of parameters, Gibraltar could be configured to detect as many as 96% of known rootkits while consuming just 14% additional energy. Another research direction explored by the project was on new architectures to isolate rootkit detectors such as Gibraltar from malicious software. Traditional isolation techniques rely on the use of virtual machine monitors or on a separate physical machine with DMA access to the target machine to be monitored. This research direction explored the use of heterogeneous multicore processors to isolate rootkit detectors. In particular, the project developed a new machine architecture called Limited Local Memory (LLM). LLM leverages recent advances in multi-core processor technology to isolate rootkit detectors such as Gibraltar without requiring virtualization technology, and without requiring physical machine isolation. Finally, this project has investigated the applicability of Gibraltar-like tools to cloud computing environments. In cloud environments, client virtual machines can potentially become infected with rootkits. While cloud providers can offer to scan their virtual machines using Gibraltar for a fee, doing so compromises privacy because theyexpose the memory pages of their virtual machines to a Gibraltar daemon that is controlled by the cloud provider. This project developed a new infrastructure called Self-service Cloud Computing that allows clients of cloud computing to execute Gibraltar-like rootkit detection tools on their VMs without allowing the cloud provider to compromise their privacy. Education-wise, the PIs were involved in several efforts to disseminate the research results and train students. The press attention received as part of the mobile rootkits work significantly enhanced public understanding of the threats of mobile rootkits, and more generally, of mobile malware that could infect smartphones. The PIs have made the source code of Gibraltar available to other researchers, who have used it in other projects. Finally, the PIs have actively been involved in the New Jersey Governor's school for Engineering and Technology. In this school, several high-school students worked on month-long projects exploring various issues in computer security. This increases awareness in computer security, and helps recruit and train the next generation of computer security researchers.
