Big Data Security Analytics: Making the Impossible Possible

16 Jul

Big Data Security Analytics: Making the Impossible Possible

in Blog, Lessons Learned, Technology

by Nikfar Khaleeli


In my last blog, I concluded that for security analytics, it is not just about having more information, but also about having more of the right kind of information. This information will make it easy for security analysts to wade through the oceans of warnings being generated by numerous IT systems. They can more easily conduct effective threat hunting and surface risky user behavior, which is potentially indicative of a compromised user or a malicious insider. And because of this higher order information, there will be an increase in security operations center (SOC) and analyst efficiency. Why? Because this information elevates the capabilities of analysts thereby enabling accurate alert prioritization, identifying any relationship between seemingly unconnected alerts, allowing “what if” hypothesis exploration when searching unknown threats, providing needed supporting evidence and more. With this higher order information, analysts will no longer have to examine each and every security alert when trying to separate the real from the innocuous. They can immediately hone in on what’s important.

However, getting this higher order of information can be challenging. The building blocks for this information are already there, but buried in the log, flow, packet, file, alert and threat feed data that’s all over an organization. But remember we are talking about sophisticated threats, which routinely circumvent perimeter defenses. Uncovering them is no easy feat. For example, in a multi-stage attack, each action alone may not rise to the level of an alert, but the end result can be disastrous.

Marrying big data with machine learning can help address this challenge by providing security professionals with the big data security analytics (BDSA) they need to thwart the bad guys. Big data provides a platform that is economically appealing. Machine learning can automatically provide meaningful insights, and because of the distributed computing capabilities of big data architectures, can use advanced statistical modeling techniques. Because of this confluence, what was previously impossible is now possible:

  • Any data type and format – structured, semi-structured, or unstructured – can be used. Not being restricted to a single data source enables taking a holistic view of the threat landscape.
  • Unsupervised behavior profiling analytics, including entity and user behavior analytics (UBA), can operate on these vast swaths of data to learn what’s normal and subsequently unearth anomalous behavior.
  • Analytics can automatically be applied to each datum in the foundational data (Niara calls this ‘discrete analytics’), either immediately triggering an alert if deemed to be severe or being used to annotate the datum to facilitate further processing at a later stage.
  • Automatic correlation ensures that everything – data and analytics – is tied back to an entity, which could be a user, a device, an IP address, or an application. Operating at the entity level, rather than only the user level, adds to the holistic view of the threat landscape that is enabled by using varied data sources.
  • Data can be stored at high-fidelity for long periods of time, supporting complex queries and extended look-backs.
  • Comprehensive entity-based profiles with rich visualizations can be automatically created, providing a summary of all that’s important about the entity at any given point in time.

Big data security analytics can surface the higher order information that’s buried in foundational data, automatically teasing out relationships between disparate events, even if they are weakly related. Because of the highly automated functionality that’s possible, it provides the machine assist that security analysts need to make better decisions. And when rich, varied data sources – e.g., logs, flows, packets, files, alerts, threat feeds, etc. – are used to create the analytics and profiles, BDSA can provide unmatched visibility and context into sophisticated threats, something that’s just not possible with traditional systems. The resulting improvement in analyst efficacy and efficiency translates into lower organizational risk, especially when faced with advanced attacks. That’s a huge benefit given the all-too-real costs that arise from a breach – up to $1.25B in one case when considering lost business, various compensations costs, and new investments.

Tags: Blog, Lessons Learned, Technology