Network Security: What is NTA? And why IDS + DPI + traffic recording ≠ NTA

With the departure of foreign vendors, the domestic information security market in 2022 received a strong impetus for development – interest in the development and refinement of advanced products increased. As a result, many new developers and products appeared on the market, complicating the understanding of the unique features of solutions and what tasks each type of product solves. Thus, for the main part of the market, the line between IDS and NTA, NDR and NTA, SOAR and XDR, the difference between cloud-managed EDR and XDR is not fully visible. This material is the first article in a series in which I will help to understand what is behind the abbreviations, what technologies they work on, and what determines the cost of each system.

The article will be useful for information security specialists, network security engineers, analysts, and IT managers who want to delve deeper into various threat detection and prevention technologies. In addition, the article will be of interest to those who study modern approaches to cyber protection and plan to implement NDR.

Prerequisites for the emergence of NTA

When I was dealing with the class of NTA (Network Traffic Analysis) products, I did not find a single article that specifically described what requirements a product must meet and what tasks it must solve in order to belong to NTA, as well as the essence of the differences between this class of solutions and IDS/NGIDS.

Let's start with the basic definitions: IPS (Intrusion Prevention System) - intrusion prevention system. IDS (Intrusion Detection System) - intrusion detection system, the NG prefix denotes the next generation, Next Generation systems.

Previously, IPS/IDS were a separate class of systems, which included several sensors located in various network segments. This approach to detecting and preventing attacks has outlived itself. Now these two functions are an integral basic component of any NGFW (Next Generation Firewall). The difference lies only in the choice of operation type: prevent or detect.

Let's highlight the main disadvantages of IPS that led to the degeneration of these solutions as a separate class:

• a large number of false positives and inability to withstand zero-day threats;

• high cost due to the need to deploy a large number of sensors;

• additional costs for maintaining the entire IPS infrastructure;

• significant investments in configuration related to the specifics of signatures and heuristics.

It is worth highlighting the negative features of signatures that led to the cessation of the development of IPS/IDS as a separate class of solutions:

• ineffectiveness against zero-day attacks and APT;

• false positives. They are inevitable for the signature method, as new traffic patterns constantly appear, new network applications are developed, and infrastructure and software change. If at the time of writing the rule only real attack attempts were detected, after a while new legitimate applications and systems appear that use traffic capable of triggering the signature detection;

• it takes time to emulate a new attack in the lab and write IDS signatures, and then promptly update the sensor on the local installation. During this time, the IDS cannot detect new attacks, and the infrastructure is at risk.

NGIPS/IDS – an unsuccessful attempt to solve IPS/IDS problems

As an attempt to solve the problems of classic IPS/IDS systems, their functionality was improved in a new class – Next Generation IPS/IDS.

Next Generation IPS/IDS, depending on the vendor, could have the following set of features:

• DPI – deep packet inspection, detection of various application protocols;

• SSL Decryption – checking encrypted traffic with signatures;

• TI feeds – data streams with indicators of compromise (IP addresses, host names, URLs, and hash sums of malicious files, etc.);

• file interception and sending them for analysis to the sandbox;

• automatic application of signatures based on infrastructure inventory;

• integration with Endpoint Protection agents;

• statistical models with machine learning support.

Next Generation IPS/IDS were presented by Cisco (SourceFire), McAfee, Trend Micro, Fire Eye, and other vendors.

Gartner Magic Quadrant for Intrusion Detection and Prevention Systems

For example, the vendor Tipping Point, later acquired by Trend Micro, was one of the leaders in the Gartner Intrusion detection and prevention systems quadrant in 2018. For the TippingPoint Threat Protection System product, it declared the following functionality: "TPS applies statistical models developed using machine learning methods." The description shows that machine learning is not so much a tool here as a marketing ploy, and detection is carried out through statistical models.

In the Gartner Intrusion detection and prevention systems quadrant 2018, the vendor Vectra AI (formerly Vectra Networks) was present as a visionary with the Cognito product, which later became an active representative of the NTA and NDR product class.

Commenting on the release of the Gartner report, Vectra AI stated the following:

Cognito uses artificial intelligence (AI) for continuous automated threat hunting using constantly learning behavioral models to quickly and efficiently find hidden and unknown attackers before they cause damage.

That is, as early as 2018, certain products offered an approach based on the use of machine learning and behavioral anomaly detection, but at that time there was no generally accepted classification of such an approach.

In summary, it can be said that NGIPS supported various additional security technologies, but the fundamental ones were still the signatures of already known threats, and the use of a large number of sensors in different segments of the network.

This approach did not solve the main problem – detecting unknown attacks and vulnerabilities.

NTA – a new level of traffic analysis technologies

The shortcomings of NGIPS/IDS and the changed threat landscape led to the fact that vendors engaged in network security and network performance monitoring (NPMD – network performance monitoring and diagnostics) began to refine their products, not based on the signatures of known attacks and vulnerabilities.

In 2017, Gartner released a report Gartner Top Technologies for Security in 2017, which featured a new class of products called Network Traffic Analysis, NTA. The report provided the following definition:

NTA is an approach based on monitoring network traffic, flows, connections, and objects in search of malicious intentions.

NTA identifies, tracks, and sorts relevant events.

As a justification for the emergence of NTA, Deception, and other classes of products, it was stated:

Security and risk management leaders should evaluate and use the latest technologies to protect against advanced attacks, more effectively implement digital business transformation, and adopt new computing styles such as clouds, mobility, and DevOps.

In 2019, Gartner published the Market Guide for Network Traffic Analysis, which more precisely defined this class of products and specified the requirements for compliance with it.

Requirements and main misconceptions about NTA

According to Gartner, NTA has the following definition:

NTA uses a combination of machine learning, advanced analytics, and rule-based detection to detect suspicious activity in corporate networks. NTA continuously analyzes raw traffic and/or network telemetry data (e.g., NetFlow) to build models that reflect normal network behavior (baseline). When NTA systems detect abnormal traffic, they send alerts. It is also necessary to determine the direction of network traffic or network telemetry from the internet to the organization's network (north/south) and network interaction through internal subnets (east/west).

Gartner criteria for inclusion in the NTA class, according to Market Guide for Network Traffic Analysis 2019:

• analysis of network traffic or telemetry in real or near real-time;

• tracking and analyzing traffic directions – north/south traffic (when crossing the perimeter) and east/west traffic (horizontal movement within the network);

• focus on the threat detection phase more than just network traffic data for attack investigation (e.g., PCAP – packet capture analysis);

• building profiles of normal traffic and detecting anomalies;

• use of behavioral analysis technologies (non-signature technologies that do not use a database of known attacks for detection), such as machine learning or advanced analytics to detect anomalies.

Which systems cannot be considered NTA, according to Market Guide for Network Traffic Analysis 2019 Gartner:

• systems that require the mandatory use of additional security tools – for example, SIEM, NGFW, web proxy, endpoint agents, etc., are additionally required for anomaly detection;

• systems that work based on log analysis;

• systems that primarily use rules, signatures, or reputation lists to detect anomalies;

• systems that are based on user session activity analytics – for example, UEBA technologies;

• systems that focus on traffic analysis in the Internet of Things (IoT) or industrial technologies (OT) environment.

The described requirements allow us to conclude that NTA class systems, thanks to the support of network deviation detection technologies from normal behavior, allow closing the vulnerabilities of NGIPS.

Here you can also see a similar trend in terms of host protection. Endpoint Protection capabilities were not enough to protect against zero-day attacks, so both EDR (Endpoint Detection and Response) solutions, relying on non-signature methods of detecting attacks on hosts, and sandboxes for safe verification of suspicious file behavior scenarios appeared. It is the detection of anomalies based on network traffic that allows NTA to detect zero-day threats without using a known payload. The use of network deviation detection technologies from normal behavior in NTA does not exclude the possibility of using IDS signatures and TI feeds.

Several examples of anomaly detection in NTA class systems:

• DNS, SSH tunnels, and others;

• communication with command and control (C&C) centers of attackers;

• transmission of credentials in plain text or their theft;

• exfiltration of sensitive data in an encrypted channel;

• credential compromise;

• slow scans;

• signs of horizontal malware movement in the network.

NTA can only operate in passive mode, on a copy of traffic or network telemetry, excluding a single point of failure. Working in passive mode allows reducing the cost of implementation through centralized analysis of a copy of all traffic or telemetry and using significantly fewer sensors.

Limited functionality of NGFW based on IDS and DPI modules together with a traffic recording system is mistakenly taken for NTA. This scheme does not solve the main task – detecting those threats that are not in the network attack database or in reputation lists.

Thus, it is the profiling technologies (baseline – normal behavior and deviation detection from them) as opposed to detection based on the database of known network attacks IDS and IP/host reputation based on network traffic that determine the higher cost of the license per 1Gbps for NTA.

At the same time, from the point of view of implementation, NTA is cheaper because you can install one sensor, for example, for the entire network or the entire branch, if the data transmission channel allows.

Transition to NDR – the next step in the development of network protection systems

NTA was only an intermediate step and today has already moved into an outdated class of products for network traffic analysis worldwide, except for Russia, where this abbreviation usually means different functionality – IDS and DPI modules together with a traffic recording system. The NTA direction for international vendors is either closed or has evolved into the NDR class.

Unlike the NTA system, which is focused on threat detection based on network traffic behavior analysis, NDR adds automated response and incident management capabilities. The transition to NDR allows organizations to respond to threats faster, improve coordination between network and endpoint security, and provides tools for in-depth investigation and reduction of incident remediation time. In the next article, I will talk about the features of NDR and how this technology helps to more effectively protect the corporate network.