Trapping to hunting: intelligent analysis of anomalies to detect compromises
Trapping to hunting: intelligent analysis of anomalies to detect compromises

Breach Detection Systems (BDSs) identify patterns of events to determine if a network has been compromised. Event streams include network activity,  host activity and the analysis of various artifacts that are observed in the network.

One of the goals of BDSs is to provide the most effective automated detection with minimal false positives, because excessive false positives cause “fatigue” in the incident responder. This means that the sensitivity threshold of a BDS system must be set so that an alert is generated only when a substantial amount of supporting evidence is gathered.

The less worrying information gathered by a BDS system can be used to detect attacks that cannot be automatically identified with a high degree of confidence. These attacks, instead of having clear patterns of malicious behaviour, are identifiable because their actions are anomalous when compared to normal network traffic.

Anomaly detection works under the assumption that malicious activity will result in anomalies in some event stream, and, at the same time, anomalies in an event stream are caused by malicious activity. Unfortunately, in the real world, both assumptions are sometimes incorrect, and anomaly detection has been riddled by both false negatives (because malicious activity does not generate anomalies) and false positives (because benign activity generates anomalies).

Pure anomaly detection can provide hints for where an analyst might look more deeply to make connections between seemingly unrelated events. This is a new approach that instead of providing machine-based, automated detection, supports human-centered analysis of interesting events. In a nutshell, the analyst moves from being a trapper to being a hunter.

A hunting system provides a series of observations that are either anomalous per se (according to a pre-established model), or anomalous when put in the context of the historical behaviour of a network. The following examples illustrate the observations produced by a hunting system:

●      Amount of traffic sent to a specific host, network, or country

●      Session timing and duration

●      Use of particular tools, application, and protocols

The important aspect of these examples is that none of the resulting observations is necessarily the result of a compromise. A user might have been tasked to perform a remote backup with a cloud-based service, resulting in a large amount of data being transferred to an outside host, could be under pressure for a deadline and working off-hours, or might start using new tools to improve productivity.

A human analyst can use a hunting system to recognise that the sudden spike in upload traffic, correlated with an unusual session time for a user in a department that is notorious for not working late hours is enough to warrant an in-depth investigation. Or, the appearance of chains of remote desktop connections (from host A to host B to host C), a pattern that has never been observed before, together with an anomalous number of failed accesses to a shared file system might be worth the attention of the hunter, as it could be evidence that an intruder has gained access to the system and is poking around, trying to get a larger foothold in the network.

Fundamentally, the hunting tool does five things:

1.    Collects various event streams (login records, DNS resolutions, netflow data, etc.)

2.    Models each type of event stream and each involved element (user, host, network).

3.    Reports unusual activity observed in the event streams, based on the established model.

4.    Presents the observations in different ways, to highlight connections and support sophisticated analysis.

5.    Expands the analysis by retrieving additional information about the network and its hosts (eg, by using systems like osquery) to augment the current observations.

It is clear that while the “Collects” and “Presents” functionality is less difficult to design, the “Models” and “Reports” components are the ones that require the development of novel approaches to produce relevant observations that contain sufficient explanatory power. Just observing that something is weird is often not enough: the system must explain why something is weird.

Contributed by Dr Giovanni Vigna, CTO, Lastline

*Note: The views expressed in this blog are those of the author and do not necessarily reflect the views of SC Media or Haymarket Media.