ML IPS in Ideco NGFW: signature-less protection against zero-day attacks

Doing anything else in an IT product at that time seemed untimely. It’s somewhat similar to now – not using AI tools in work and AI functionality in products for protection. At a time when attackers are actively using AI tools. And attacks are becoming increasingly sophisticated and faster.

In 2025, 90 zero-day exploits were recorded in the wild. 44% of zero-day attacks target corporate network devices - NGFWs and VPN gateways. The average time from CVE publication to its first exploitation in real attacks has reduced to 5 days and will shrink even further. No signature database can keep up with this pace. We discuss how we are working on the ML intrusion detection module in Ideco NGFW, what a real-world experiment with ISP RAN showed on 73 million sessions, and what limitations this approach has.

Why signatures are becoming ineffective

Signature-based IPS works fundamentally the same way as antivirus software in the 1990s: there is a database of known threats, there is incoming traffic, and there is comparison. IPS – for all its power, operates with pre-defined patterns.

The problem is not in the approach itself - the problem lies in the speed of emerging threats. According to Google Threat Intelligence Group, in 2025, 90 zero-day exploits were recorded in the wild. According to RAND Corporation, the average lifespan of a zero-day attack until its discovery is 312 days. During this time, a signature will not appear: it is impossible to write one for something that has not yet been detected.

A signature only appears after the attack has been discovered. There exists a vulnerability window between the emergence of a new exploit and the creation of a signature. A method is needed that does not rely on the existence of signatures. This is where machine learning comes into play.

What global leaders are doing

Palo Alto Networks, Fortinet, Cisco, Check Point, and Juniper have been integrating ML into their IPS for several years. Their approaches are fundamentally different - and this is instructive.

Among the variety of domestic NGFWs, ML-IPS has not yet been implemented by anyone, so Ideco NGFW will likely be the first.

Palo Alto: Deep Learning in the Cloud and on the Device

Advanced Threat Prevention is positioned as the first IPS in the industry capable of blocking zero-day attacks in inline mode. The architecture of the solution is hybrid: known traffic is processed by a classic signature engine (compatible with Snort/Suricata rules). Suspicious traffic that does not match signatures goes to the cloud for analysis by deep learning models—or is analyzed locally in Local Deep Learning mode directly on the device.

Specific results from Unit 42 tests on real traffic: ML model for detecting SQL injections - TPR 90% with FPR 0.02%. Model for command injection - TPR 92% with FPR 0.011%. According to the vendor's claims, the solution blocks 60% more zero-day injection attacks compared to traditional IPS.

Fortinet: ML as a Filter for the "Gray Zone"

In FortiOS 7.6, an interesting architectural principle has been implemented: ML is applied not to all traffic but only to the part that has passed through the signature filter without a definitive verdict. Signatures process "whitelist" and "blacklist," while ML works with the "gray zone." This reduces both false positives and computational load.

Models are delivered via a separate IPS-MLDB package from FortiGuard servers. Detection of Cobalt Strike has been reported with an accuracy of 99.6%.

Cisco: Fingerprinting Without Traffic Decryption

Cisco has focused on analyzing encrypted traffic without a MITM proxy. The Encrypted Visibility Engine (EVE) analyzes the TLS Client Hello during connection establishment and identifies the client process by the handshake "fingerprint." The database includes over 1 billion TLS fingerprints and 10,000 samples of malware daily. The system recognizes more than 5,000 client processes.

This is not a replacement for traffic inspection, but an additional layer: detecting malicious processes in encrypted connections without resource-intensive decryption.

Check Point: Collective AI

ThreatCloud AI aggregates telemetry from 150,000 connected networks and millions of endpoints. More than 50 engines with AI functions—including NLP for phishing pages and deep learning for macro classification in documents. The claimed accuracy of preventing zero-day malware is 99.7% according to VirusTotal testing.

Juniper: ML on Streaming Traffic

In Junos OS 24.2R1, ML detection of zero-day attacks operates on streaming traffic, without internet access, using only a fragment of the file to make a verdict. Models are delivered via CDN along with antivirus signatures. Additionally - Adaptive Threat Profiling: automatic creation of threat feeds based on specific customer IPS events.

Comparison of Approaches

Key takeaway: all vendors without exception use ML as a complement to signatures, not a complete replacement. And most deliver ready-made models centrally - from the cloud or through update packages.

This is where our approach fundamentally differs.

Why we did not pursue ready-made cloud models

All the vendors listed above train models centrally and deliver them to customers. This makes sense from a scaling perspective but creates a fundamental limitation.

There is no universal ML IDS. This is not a marketing statement - it's mathematics.

Research by the Institute for System Programming of the Russian Academy of Sciences on the portability of ML models demonstrates that a model trained on the traffic of one network performs significantly worse on the traffic of another. The reason is that the informative features of network traffic—inter-packet delays, transmission speeds, and size distributions of packets—depend on the physical topology of the network, hardware settings, service configurations, and even the number of concurrently active users. Transferring a model between networks is a transfer between different statistical distributions of data.

More details on this issue (and how IDS Suricata can serve as a teacher for an ML model) can be found in the very detailed article by the Institute for System Programming of the Russian Academy of Sciences titled “From Signatures to ML IDS: What Can IDS Suricata Teach the Model?” published on Habr. We recommend it to those who need a thorough technical analysis.

Our conclusion: the best ML model for a specific client is the one trained on their own traffic. That is why we design ML IPS to work directly on the client's NGFW. For Russian organizations, especially critical infrastructure facilities, this is one of the mandatory requirements.

Experiment: 73 million sessions, two weeks, real traffic

Before designing anything, it is necessary to understand whether the idea works at all on real traffic. We conducted a field experiment in collaboration with the Institute for System Programming of the Russian Academy of Sciences.

Testbed Architecture

Traffic from a gateway with an installed Ideco NGFW (including a functioning IPS) was mirrored simultaneously to two modules:

• session_analyzer (Institute for System Programming of the Russian Academy of Sciences) - a utility that calculates the feature vector for each TCP session in real-time based on the 5-tuple principle (src IP:port, dst IP:port, proto). The complete vector contains 118 features; for ML, we selected the 10 most informative ones.

• IPS based on Suricata with up-to-date signatures (~30 MB of rules in JSON). The IPS acted as a “teacher”: its triggers were used to label the dataset—each session was assigned a label of “attack” or “clean traffic.”

The experiment lasted two weeks. In total: 73 million network sessions, of which 57,465 were marked as attacks by Suricata.

Why These 10 Features

The complete vector session_analyzer contains 118 features. For the ML model, we use only 10, selected based on the results of the Institute for System Programming of the Russian Academy of Sciences research on feature significance and experimentally confirmed:

These features describe behavioral patterns of the session, not its content. This is fundamental: the method works independently of traffic encryption.

An important technical detail: all averages and variances are calculated online using Welford's algorithm - without accumulating arrays of packet lengths. This is critical for performance on high-speed networks:

# Welford's Algorithm: online computation of the average without storing an array
def update_welford(count, mean, new_value):
count += 1
delta = new_value - mean
mean += delta / count
return count, mean

# Variance is computed similarly through M2
def update_welford_variance(count, mean, M2, new_value):
count += 1
delta = new_value - mean
mean += delta / count
delta2 = new_value - mean
M2 += delta * delta2
variance = M2 / (count - 1) if count > 1 else 0
return count, mean, M2, variance

Instead of storing all packet lengths - three scalar variables per feature. For a network with hundreds of thousands of parallel sessions, the difference in memory is significant.

From Random Forest to CatBoost: A Story of Iterations

The first iteration of the experiment - Random Forest. The results on the training set looked good (F1 up to 0.99). But on the test set (traffic from another time period), the picture was different:

Model: Random Forest, 100% training data
Test: 73,518,285 sessions (73,460,820 clean, 57,465 attacks)

TP (attacks detected): 16,110
FN (attacks missed): 41,355 <- 72% of attacks missed
FP (false positives): 8,767
TN (true clean): 73,452,053

Precision: 0.647 Recall: 0.280 F1: 0.391 FPR: 0.012%

The model detected only 28% of attacks - this result was unacceptable. The analysis identified three reasons.

Reason 1: "Dirty" labeling. Some of Suricata's alerts are associated with events whose feature vectors are indistinguishable from clean traffic. For example, detection by IP address or SNI (without substantive statistical features of the session). The model was trained on labeled data where "attack" and "clean traffic" had identical behavioral profiles.

Reason 2: Mismatch of observation points. Suricata and session_analyzer saw traffic from different points in the network - before the gateway and after NAT translation. One Suricata event could correspond to two connections in the session_analyzer. Timestamps diverged by up to 100 ms.

Reason 3: Traffic drift. The training and testing data were from different time periods. The nature of the traffic changed along with the load, new IPS rules (which are updated every 2 hours in Ideco NGFW), and changes in user activity.

After switching to CatBoost (gradient boosting over decision trees) and iterative data cleaning, the results changed dramatically.

Finding adversarial examples: the key to quality

The main improvement was the procedure for identifying "bad" sids - those categories of attacks whose feature vectors are indistinguishable from clean traffic in our feature space.

In a 10-dimensional normalized feature space, the maximum geometric distance between two vectors is: dist_max = √10 ≈ 3.162. It was experimentally established that vectors are considered "close" when dist < 0.0003.

All attack vectors were compared with the first 2 million clean vectors. Result: out of 57,465 initial attacks, after filtering those close to clean vectors, there were 16,798 real attacks left. 40,667 events were reclassified as "clean traffic" - they are identified solely based on address features not present in the 10-dimensional behavioral vector.

Final results: four iterations

Final model (generalization capability assessment):

The model was trained on only one time segment (12% clean vectors and 38% of all attacks). It was tested on the entire dataset - 55.5 million sessions from eight different time periods:

Accuracy: 0.9999
Precision: 0.9928
Recall: 0.9661
F1-score: 0.9792
FPR: 0.00066%

This is a key result: the model trained on a portion of the data correctly classifies traffic from time periods in which it was not trained. Generalization capability has been confirmed on 55 million sessions.

Hyperparameters of the final model: CatBoostClassifier, iterations=1000, depth=8, eval_metric='F1', task_type='CPU'. For inference, the model is exported to ONNX and runs through ONNX Runtime - the speed is hundreds of thousands of requests per second.

64 categories of attacks that ML will not detect: method limitations

The study revealed a fundamental limitation that is important to understand when implementing any behavioral ML IDS.

64 categories of Suricata signatures have feature vectors that are almost identical to clean traffic. This is a limitation of the feature space, not a problem of the specific ML model.

What are these events:

The logic is simple: a session to the C2 server is statistically indistinguishable from a regular HTTPS connection. The same interpacket delays, the same packet sizes. ML “sees” only behavioral patterns, but not address context.

There are two solutions:

1. Expand the feature space - add IP reputation and SNI to the vector. This shifts the system to a hybrid mode: behavioral analysis plus reputation.

2. Accept the limitation - position the ML module as a complement to the signature-based IPS in Ideco NGFW, covering precisely what signatures do not see: unknown attacks with atypical behavior.

We chose the second path. This aligns with the architectural principle of all leading vendors.

Other limitations

Model Drift. Interpacket delays and data transfer speeds depend on the state of the network: the number of active users, load, infrastructure changes. With significant changes in the network, the model degrades. For example, Sophos retrains its models every 3 months - we consider this interval reasonable.

We observed this in an experiment: in the 6th time segment (the beginning of a new working week after the holiday season), false positives sharply increased. The likely reason is that the nature of traffic changed due to employees returning and updating IPS rules. After retraining the model on data from the same segment, quality returned to normal.

Initial data accumulation period. For training, at least 2 months of labeled data on the specific client's network is required. During this period, the ML module operates only in observation mode - the signature IPS remains the primary tool.

Quality is determined by the diversity of attacks. If there are few attacks on the client's network, the generalization ability of the model will be weaker. For such cases, we plan to add a "pentest enrichment" mode: controlled generation of attacking traffic during the training phase.

The problem of probability calibration. For Random Forest, the predict_proba values require additional calibration (Platt's method or isotonic regression) for correct probabilistic interpretation. CatBoost provides more accurately calibrated probabilities out of the box - this is one of the arguments in its favor.

ML IPS architecture in Ideco NGFW

Based on the research findings, we developed a phased implementation plan. Here’s how it will be structured from the inside.

Feature collection: VPP and online algorithms

Features for each session (conntrack) will be computed directly in the traffic processing plane - in VPP (Vector Packet Processing). This allows features to be obtained without copying traffic into memory and with minimal overhead.

Upon completion of the session (or after a timer), a vector of 10 features is saved in ClickHouse with a TTL of 1 month. Each row contains flow_id, address information, and 10 feature columns:

{
"flow_id": "01HZ...",
"src_ip": "10.0.0.15",
"dst_ip": "1.2.3.4",
"src_port": 54321,
"dst_port": 443,
"proto": 6,
"average_packet_size": 487.3,
"flow_bytes_per_s": 12450.8,
"max_packet_length": 1460,
"fwd_packet_length_mean": 312.1,
"fwd_iat_min": 148,
"total_length_fwd_packets": 9364,
"fwd_iat_std": 2340.5,
"flow_iat_mean": 18750.2,
"fwd_packet_length_max": 1460,
"fwd_header_length": 320
}

Labeling: IPS as a teacher

IPS triggers are matched with the session table via the unified flow_id. Architecturally: flow_id is generated as ULID in the traffic control module, available within Suricata, and recorded in EVE JSON as ideco_flow_id. In ClickHouse, there are two tables - conntrack statistics and IPS events, linked by key.

Each labeled session includes: class label ("attack" / "non-attack"), sid, name, and category of the IPS event. "Bad" sids identified in the study are automatically excluded from the training dataset.

Training and export to ONNX

After accumulating a minimum dataset (recommended period - 2 months), model training begins. The model is exported to ONNX and executed via ONNX Runtime. The expected size of the Random Forest model is tens of MB. Inference speed is hundreds of thousands of requests per second.

Manual switching between "training → operation" as well as automatic switching is provided.

Inference Daemon and "soft" verdicts

The Inference Daemon receives a vector of 10 features for each completed session and sends it to ONNX Runtime. Instead of a binary response, a probabilistic verdict (predict_proba) is used:

In the security event log, it records not "attack X detected," but "attack X with a probability of 0.87." The administrator sets the trigger threshold. The recommended starting threshold is 0.75. This allows managing the balance between detection completeness and the number of false positives without retraining the model.

A similar approach is implemented at Cisco (EVE Threat Confidence Score) and Palo Alto (configurable confidence thresholds). The probabilistic verdict is the standard for mature ML IPS systems.

Monitoring and retraining

Monitoring of three groups of metrics is provided:

• Model Quality: FPR, TPR, F1 score on verified triggers

• Model Drift: statistical shift in the distribution of input features relative to the training dataset

• Model Health: an aggregate indicator of the model's "health" - whether it's time to retrain

Retraining can be both manual and automatic. The administrator can manually label triggers ("correct" / "incorrect"), and these labels are taken into account in the next training cycle.

What comes next: priorities from global experience

Analysis of the approaches of foreign vendors allows highlighting specific priorities in development:

Multiclass classification. The current model is binary ("attack" / "no attack"). Transitioning to defining the type of attack ("scanning," "C2 channel," "exploiting vulnerabilities," "DDoS") will allow integrating ML IPS with the tactics and techniques of MITRE ATT&CK - as done by Palo Alto and Check Point.

Analysis of encrypted traffic. Cisco EVE's approach - fingerprinting TLS Client Hello - is the next logical step. The ability to detect malicious processes without decrypting traffic is critical for organizations where TLS inspection is not applicable for various reasons.

Expansion of the feature space. Including address information (IP reputation, SNI) in the vector will close part of the 64 "blind" categories and transition the system to a full hybrid mode.

Pre-trained models + fine-tuning. An open question: is it possible to create a "base" model that, after fine-tuning on client traffic, produces results comparable to those of a "natively" trained model? Research by the Institute for System Programming of the Russian Academy of Sciences on transferability provides a cautiously optimistic answer. This could potentially solve the problem of the initial data accumulation period for new installations.

Automatic calibration and adaptation. Automatic detection of Model Drift and launching retraining without administrator involvement is the goal that most mature ML IPS platforms are moving towards.

Conclusion

Research findings: F1 score of 0.979 and FPR of less than 0.001% in a real-world experiment with 55 million sessions of actual corporate traffic. This is not a laboratory synthetic benchmark, but a two-week experiment on live infrastructure.

The main conclusion: An ML model trained on the real traffic of a specific network demonstrates high quality in signature-less attack detection. ML complements signature-based IPS - and this is the architecture used by all leading vendors.

The difference between our approach and foreign leaders: we train the model not centrally with delivery via the cloud, but directly on the client's NGFW, using their real traffic. This provides better adaptation to the specifics of the particular infrastructure and removes cloud dependency - a critical factor for the Russian market.

In upcoming versions of Ideco NGFW, the ML traffic analysis module will appear as an experimental feature. If you work with intrusion detection or want to discuss the details of the research - please write in the comments.