A Modular, Data-Agnostic, Scalable UEBA Architecture

14 Jun

A Modular, Data-Agnostic, Scalable UEBA Architecture

in Blog, Perspectives, Technology

by Dr. Jisheng Wang

In my last blog, I made the process for selecting a user and entity behavior analytics (UEBA) solution easy. Simply evaluate solutions along three axes - scalability, multi-dimensional analytics, and the integration between human and machine intelligence.

Guided by these considerations, Niara has designed a UEBA solution with a flexible architecture that delivers varying levels of machine assistance to complement analysts’ needs for behavioral analytics. As shown in Figure 1 below, Niara’s UEBA is built with 4 layers of abstraction – use case definition, feature selection, baseline profiling, and anomaly detection.

Figure 1

  • Use Case Definition. The 1st layer defines a behavioral use case (e.g., suspicious access to critical servers) that generally requires local context.
  • Feature Selection. The 2nd layer is about the selection of meaningful feature categories (e.g., time, data volume, counters, etc.,) for each use case. While feature selection is mostly a human driven effort, deep-learning algorithms (e.g., convolutional neural network) that can automatically extract discriminative features from a large volume of unlabeled training data can also be used.
  • Baseline Profiling. The 3rd layer learns the “normal baseline” for each entity (i.e., user or host) for each use case along two dimensions of behavior: historical and peer-group. The former uses the entity’s own historical behavior while the latter uses common behaviors across a peer group, which can be flexibly defined (e.g. from Active Directory, user-provided input) or derived through self-learning. Niara also uses adaptive learning to incorporate analyst feedback into its behavioral models.
  • Anomaly Detection. The 4th layer detects behavioral anomalies for each entity through the deviation from their baselines. This part is totally automated and machine driven. Given different dimensions and types of feature vectors selected in each behavior use case, Niara has built different unsupervised machine learning models with corresponding distance (i.e., deviation) calculations to automatically detect anomalies.

Building a flexible behavioral analytics solution requires deliberate design choices and significant investment during product implementation. It’s well worth it as Niara’s analytics identify threats that evade other simplistic approaches. This payoff is enabled by three critical design choices.


As described above, we have abstracted and decoupled the use case layers (the 1st and 2nd layers), which are security context-driven, from the detection layers (the 3rd and 4th layers), which are machine learning-driven. In addition, we have done all of the heavy-lifting to pre-tune and self-tune these machine learning models, so that security analysts can start benefiting from the solution without a deep understanding of machine learning.

All four layers are built in a totally modularized fashion, so that security analysts – no matter whether they come with security or data science backgrounds – can always interact and influence the results of UEBA with their own expertise in order to improve its overall accuracy.

Data Agnostic

As explained in my last blog, a multi-dimensional UEBA solution that combines anomalous signals from different data sources can highly improve its effectiveness. Niara’s UEBA solution is built in a data-agnostic way. This means an analyst can add UEBA support for a new use case from existing or new data sources with some simple schema and use case-specific configurations.


Figure 2

If you compare these two different behavior use cases in the above pictures – suspicious access to critical servers (Figure 1) and suspicious access to buildings (Figure 2), you’ll find that the main difference between them is the data source, i.e., the first is from either server logs or network packets and the second is from badge reader logs. Except that, both use cases are monitoring the same temporal features (plus some other behavior specific features) and detecting similar behavioral anomalies.


Niara’s behavior analytics platform is built using a big data architecture, leveraging Apache Hadoop and Spark based technologies. For data persistence, we use a mix of NoSQL key-value, columnar, and time-series databases, to store a high-volume of both raw and derived data in the most efficient format for different analytics uses.

A hierarchical data processing approach enables us to break and embed different analytics requirements, e.g., feature extraction, aggregation, etc., into all stream and batch processing layers, thus minimizing the data read-write cost to achieve the best scalability.


In “The Five Characteristics of an Intelligence-Driven Security Operations Center”, Gartner’s Neil McDonald and Oliver Rochford make a central point about how they see the enterprise security operations center (SOC) evolving.

“Rather than seek full automation of all SOC activities, enterprises should seek ‘automatability’ — the capability of being automated as higher levels of confidence are achieved. Even then, analytics-driven, human-augmented security decision support systems will be used to provide the SOC analyst with the context of the recommended action, along with the details behind the verdict and recommended action.”

This is the foundation of Niara’s product vision. By designing a behavioral analytics solution that’s modular, data-agnostic and scalable, we enable organizations to achieve that “automatibility”.

Niara ships with a broad range of behavioral use cases, developed by our own security experts, leveraging the modular architecture. This enables organizations to get value from Niara immediately upon deployment. Analysts can also influence and improve the quality of behavior detections in many different ways as outlined in the blog. Plus they can also define their own behavior use cases, allowing them to extend Niara to fit their specific requirements.

With Niara, it’s not about replacing security analysts with automated systems (which is what resonated with Drew Conry-Murray of Packet Pushers who wrote this article about Niara). Rather, it’s about enabling organizations to make optimal use of scarce SOC resources.

Tags: Blog, Perspectives, Technology