Paper on Combining Heterogeneous Anomaly Detectors for Software Security through System Call Tracing submitted to IEEE TSE is under review

Abstract—Host-based Anomaly Detection Systems (ADSs) monitor for significant deviations from normal software behavior. Several techniques have been investigated over the last two decades for detecting anomalies in system call sequences. Among these two well-known techniques, Sequence Time-Delay
Embedding (STIDE) and Hidden Markov Model (HMM) have been shown to provide a high level of anomaly detection accuracy. Although ADSs can detect novel attacks, they generate large numbers of false alarms due to the difficulty in obtaining complete descriptions of normal software behavior, which also changes over time. This paper presents a multiple-detector ADS that efficiently combines the decisions from heterogeneous detectors (e.g., STIDE and HMM), using iterative Boolean combination in the Receiver Operating Characteristics (ROC) space, to reduce the false alarms. The proposed multiple-detector ADS is shown to consistently outperform an ADS based on a single “best” detector and on an ensemble of (homogeneous) HMMs. For instance, at 100% true positive rate it decreases the false alarm rate
from 20% (best result achieved to date) to 5%, whereas at an operating point of 3% false alarm rate our system achieves a true positive rate of about 95% compared to a maximum of 55% obtained on a modern real-world system call dataset.