Many research and commercial endeavors are experiencing dramatic transformations through the use of Big Data, wherein large data repositories are collected and analyzed to reveal trends, correlation, and information that may not be apparent in smaller samples. Current approaches assume centralizing the repository, which may be a poor fit in environments where the data generation rate exceeds the network capabilities. In this project, the PIs investigate system architectures for both real-time and historical analysis of geographically distributed data, combined with research in adaptively reducing data volumes to optimize bandwidth capabilities. This combination allows better use of the computation and storage associated with smarter end devices, including, but not limited to, distributed sensors, smart meters, and even full servers, without requiring network upgrades. Given the historical trends of the growth of computation and storage versus the capacity limits of wide-area networks, this research enables more data collection and analysis to be performed at a lower overall system cost. Further, the ability to dynamically adapt data precision and fidelity to available network bandwidth allows systems to gracefully and automatically improve performance in the presence of higher-capacity networks. The research enables the collection and analysis of data that is currently left unanalyzed because of network constraints. Such data can include finer-granularity usage data, which could indicate actionable steps to reducing household energy consumption, or it could include a greater olume of debugging and monitoring data, which could better predict system failures or provide greater insight than with current methods.
