Abstract
It is important to be able to place high confidence in a detection system of any kind, particularly one intended for detecting attacks against the nation's critical information infrastructure. One requirement for establishing such confidence is to have a complete understanding of a detector's "sweet spots" and operational limits, so as to calibrate the detector optimally for the conditions under which it performs best. Due to a lack of standard test data sets and measurement procedures, such calibrations have not previously been done.
The proposed research will address methods of achieving high confidence in intrusion and malicious-insider detectors by developing: (a) metrics for gauging the effectiveness of detection algorithms; (b) gold-standard reference data sets, with calibrated ground truth, to be shared among producers and consumers of detection technologies, particularly for replication of scientific experiments that determine detection efficacy for new algorithms across a range of data conditions; and (c) a data synthesizer for producing reference and calibrated data sets.
This work will put decision makers in a position to know the flaws, the strengths, and the weaknesses of detectors before deployment. Knowing the operational limitations of one detector provides the opportunity to design a companion detector whose strengths compensate for the weaknesses of the other, enabling accurate and efficient composition of detectors for the first time.
Experiments in CyberSpace
Roy Maxion, Carnegie-Mellon University
Award 0430474
Abstract
It is important to be able to place high confidence in a detection system of any kind, particularly one intended for detecting attacks against the nation's critical information infrastructure. One requirement for establishing such confidence is to have a complete understanding of a detector's "sweet spots" and operational limits, so as to calibrate the detector optimally for the conditions under which it performs best. Due to a lack of standard test data sets and measurement procedures, such calibrations have not previously been done.
The proposed research will address methods of achieving high confidence in intrusion and malicious-insider detectors by developing: (a) metrics for gauging the effectiveness of detection algorithms; (b) gold-standard reference data sets, with calibrated ground truth, to be shared among producers and consumers of detection technologies, particularly for replication of scientific experiments that determine detection efficacy for new algorithms across a range of data conditions; and (c) a data synthesizer for producing reference and calibrated data sets.
This work will put decision makers in a position to know the flaws, the strengths, and the weaknesses of detectors before deployment. Knowing the operational limitations of one detector provides the opportunity to design a companion detector whose strengths compensate for the weaknesses of the other, enabling accurate and efficient composition of detectors for the first time.
