This project is exploring how overlay networks can be designed and engineered to meet the demanding performance requirements of real-time distributed applications such as large-scale virtual worlds and distributed cyber-physical systems. These applications are characterized by sessions with large numbers of distributed endpoints that issue regular status reports that must be delivered to a dynamically changing set of recipients. They require non-stop data delivery, even as the communication pattern changes. We address this requirement using a novel communication primitive called a comtree that functions like a per-session private network that is provisioned to support continuous data delivery in the presence of rapidly changing traffic. Each comtree defines its own routing context, enabling simple and scalable routing for both unicast and multicast packets. Multicast groups can be highly dynamic with each endpoint subscribing to tens or hundreds of multicast groups and changing their subscriptions many times per second. Our provisioning methods ensure that each comtree has sufficient capacity to support its traffic, even in the presence of rapidly changing multicast groups. The methods developed in this project will enable large-scale, highly interactive virtual worlds that are better able to support social interaction and group collaboration applications in business and education.
This project has led to a better understanding of the challenges that must be met to enable internet-scale overlay networks capable of supporting applications with demanding performance requirements. We have developed prototype software for an overlay network designed to support large-scale virtual world applications. The network architecture and protocols provide provisioned capacity to virtual world applications and are designed to support worlds with millions of participants. The core provisioning mechanisms ensure consistent performance regardless of how participants move around and interact within the virtual world. The prototype software is available on an open-source basis and can be found at https://code.google.com/p/forest-net/. There are two specific areas where the research has led to a better understanding of the challenges that need to be addressed. The first area concerns the mechanisms used to allocate and manage resources for individual application sessions. Our original solution to this problem was highly centralized, limiting the rate at which new users could be added to an individual session. We have found a better solution that involves some centralized "guidance" to the resource allocation process, but allows individual routers to actually carry out the process. This dramatically increases the rate at which new users can be added, which allows for much larger application sessions. The second area concerns the design of scalable software routers based on modern multi-core compute servers. Most of our work on this project used a software router implementation that uses a single core. This router has been an effective experimental tool, but does not take advantage of the power of multi-core servers. Towards the end of the project, we did begin work on a multi-core router implementation. While this did yield to some improvement in performance, the overall results were disappointing and we were unable to demonstrate truly scalable performance. The main obstacle is the fine-grained synchronization requirements of the router data path. A variety of techniques were used to reduce the synchronization overhead, but we have concluded that a more drastic restructuring of the data path may be needed to produce a design capable of fully leveraging the power of multi-core servers. Addressing this problem should be the top priority for follow-on research, since any large-scale overlay network will require scalable software routers in order to move beyond the domain of research prototypes.
