Page308

Protocol Behavior

A Protocol Behavior IDS models the way protocols should work, often by analyzing RFCs (Request for Comments). RFC 793 (TCP, see https://datatracker.ietf.org/doc/html/rfc793) describes the TCP flags. A SYN means synchronize, and FIN means finish. One flag is used to create a connection, the other to end one.

Based on analysis of RFC 793, a resulting protocol behavior rule could be “if both SYN/FIN flags set in one packet: alert.” Based on the RFC, it makes no sense for a single segment to attempt to begin and end a connection.

Attackers craft such “broken” segments, so Protocol Behavior does detect malicious traffic. The issue is Hanlon’s Razor, a maxim that reads: “Never attribute to malice that which is adequately explained by stupidity” [8]. Protocol Behavior also detects “stupid” (broken) traffic: applications designed by developers who do not read or follow RFCs. This is fairly common: the application “works” (traffic flows), but violates the intent of the RFCs.

Note: All Information Security Professionals should understand Hanlon’s Razor. There is plenty of malice in our world: worms, phishing attacks, identity theft, etc. But there is more brokenness and stupidity: most disasters are caused by user error.

Anomaly Detection

An Anomaly Detection IDS works by establishing a baseline of normal traffic. The Anomaly Detection IDS then ignores that traffic, reporting on traffic that fails to meet the baseline.

Unlike Pattern Matching, Anomaly Detection can detect new attacks. The challenge is establishing a baseline of “normal”: this is often straightforward on small predictable networks, but can be quite difficult (if not impossible) on large complex networks.

Egress Monitoring

Historically, detection of compromises and adversary activity has focused rather exclusively on attack traffic coming into our more trusted internal networks; Intrusion Detection Systems (IDS) naturally fit this paradigm. Egress monitoring takes the opposite approach and focuses on detection via watching what leaves our networks. Much post-exploitation activity including both command and control (C2) and data exfiltration naturally will necessarily result in traffic patterns that might be discernable via monitoring of traffic egressing our networks. A classic example of egress monitoring for successful adversary detection is identification of a compromised host sending spam by observing traffic destined for TCP port 25 originating from systems other than the expected internal email servers.