Benchmark for Evaluating LLMs in Security Finding Triage Tasks

Why standard benchmarks are not suitable

THOR is a forensic scanner we develop at Nextron Systems. It searches for signs of compromise, suspicious artifacts, attackers' tools, malware, persistence traces, and other security-relevant evidence. The benchmark uses THOR findings as input data, but the problem itself is not specific to THOR.

A THOR finding is not a programming task, not a mathematical puzzle, and not a creative assignment. It's a piece of forensic context or security context from a real system. It may indicate malware, an attacker's tool, suspicious persistence, a strange cache artifact, a dual-use admin utility, or just a false positive that looks scary at first glance.

The model should make a decision:

• suppress as a false positive

• leave for manual verification

• escalate as a probable incident

These decisions have different error costs. If a model labels a harmless finding as suspicious, it wastes an analyst's time. It's especially bad if there are many such findings; when scanning hundreds or thousands of systems, even a small percentage of false escalations creates a lot of unnecessary work. But it's much worse if a model labels a real incident as harmless. This can hide an attacker and remove the only finding that should have triggered an investigation. A model can excel at code, logical tasks, or writing vulnerability reports but poorly assess real security events. Conversely, a less known model can be more conservative and make better operational decisions for this specific task.

I built a benchmark around THOR finding triage. The goal is not to identify the best LLM in a general sense, but to understand how different models behave when classifying security findings as true positives, false positives, or inconclusive (Inconclusive).

It is important not only whether the model provides the correct answer, but also what happens when the answer is incorrect. A model that sends too many items for review is noisy. A model that suppresses real threats is dangerous.

Benchmark design

The benchmark is intentionally run from a clean slate. Each model is given one enriched THOR finding and must classify it. Context is then reset, and the next finding is evaluated. There is no long-term memory, no analyst feedback, no customer-specific allowlist, no RAG, VirusTotal, sandbox, or SIEM search. This does not mean the model only receives a vague hint. THOR findings are already enriched by design — this is not something we added specifically for the benchmark. THOR uses this same structure in its standard reports. THOR does not simply state "suspicious ShimCache entry" and leave all remaining work to the LLM, but instead parses the forensic artifact, adds detection context, and enriches the event with file information wherever possible.

For example, a raw ShimCache artifact only provides a partial picture: the file path, artifact timestamp, execution flag (if available), and the registry location the entry was sourced from. If the file still exists on disk, THOR can add far more context: file type, size, hashes, first bytes, timestamps, owner, access permissions, and imphash. Also added are the detection rationale, matched data, signature link, assessment, and classification context.

The benchmark does not ask the model to reason from scratch. It asks it to interpret a structured, enriched security finding. We intentionally remove external tools, long-term memory, previous feedback, and customer-specific context.

In production, an LLM can be part of an agent workflow. It can have tools, query VirusTotal, check asset inventory, search for previous cases, view process trees in EDR, inspect events in SIEM, use customer-specific allowlists, or receive analyst feedback from a knowledge base. All of this can improve the final assessment. But this changes what we are measuring. The benchmark stops measuring only the model's triage capability. It starts measuring the entire system: the model, the prompt, tool selection, data quality, extraction quality, customer context, workflow design, and accumulated feedback from previous analyst decisions.

For this benchmark, I wanted a clean baseline. The model receives a finding. Nothing else. This makes the benchmark less similar to a full production workflow, but more comparable. All models work with the same input, the same task, without external assistance.

The current benchmark uses a controlled setup: one detection, one resolution, no tools, no memory, no feedback. This is not the final version of how a large language model will operate within a full SOC automation workflow.

What is THOR finding triage

THOR scans systems for suspicious files, forensic artifacts, attacker tools, persistence mechanisms, suspicious command lines, cache entries, log traces, and many other signals. When THOR finds something relevant, it generates a finding. A finding means: "This looks relevant enough for someone to assess it."

This could be an explicit malicious trigger, a web shell, a credential dumper, a trace of an attacker's tool execution in ShimCache or AmCache. It could also be a legitimate admin utility, a strange vendor binary, a remote administration tool, or a script that looks suspicious but is normal for this environment.

The real task isn't just detection. The real task is triage. For every finding, someone has to decide what it means operationally. In this benchmark we use three classes:

• True Positive — the finding represents a real threat or indicator of compromise

• Inconclusive — uncertainty, the finding must remain visible for manual review

• False Positive — the finding is harmless in this context

True Positive means the finding represents a real threat or indicator of attacker activity. It needs to be investigated and usually escalated.

False Positive means the finding is harmless in this context. It may have triggered by rule, but does not require any analyst action.

Inconclusive - an interesting middle category. Inconclusive means the finding must remain visible for manual review. This could be a legitimate anomaly, a dual-use tool, suspicious admin activity, an unusual persistence mechanism, or an artifact that requires customer-specific context to evaluate. In real scans, most findings are false triggers or harmless matches. A smaller portion are legitimate anomalies: things that are not incidents, but also should not be automatically suppressed. And occasionally those very few findings that are actually important come up - real incident indicators.

A good triage model shouldn't flag everything under the sun. But it also shouldn't suppress that one single finding that points to an attacker.

In this benchmark we are trying to model the decision-making problem: can the model, based on the same extended THOR dataset, make a useful sorting decision? This is not a perfect solution without context. But it can be useful.

Why scoring differs

The first version of the benchmark was simpler. The model classified a finding as TP, FP or Inconclusive. We compared the answer to the ground truth and calculated the score. It sounds simple, but it is insufficient. The problem is that triage errors are not equal in severity. In standard classification tasks, an incorrect answer is just an incorrect answer. For security findings, it matters exactly what kind of error the model made.

A false positive classified as TP puts a strain on analysts. Someone has to review it, possibly open a case, ask the system owner, check several artifacts. It is annoying and costly, especially at scale. But it is much worse when a real threat is classified as FP. Such a finding disappears from the workflow. The model is effectively saying "ignore this". If this was the only trace of an attacker's tool, suspicious persistence or malware execution, the analyst may never see it.

But it is much worse when a real threat is classified as FP. Such a finding disappears from the workflow. The model is effectively saying "ignore this". If this was the only trace of an attacker's tool, suspicious persistence or malware execution, the analyst may never see it. A model that escalates harmless findings is noisy. A model that suppresses real threats is dangerous. These are different types of errors.

Inconclusive means that the finding must remain visible for manual review. A TP→Inc decision is not ideal, but it is safe for review. The model did not clearly identify the incident, but it did not suppress it either. An Inc→FP decision is more problematic. It means that an anomaly worthy of review disappears. This could be a dual-use tool, suspicious admin behavior, unusual persistence or another context-dependent signal that should not be automatically discarded.

Therefore, we added an operational scoring layer. The benchmark measures not only classification quality, but also the practical utility of the decision in a real triage workflow.

Scoring should reflect the cost of the decision. Simplified, it looks like this:

• Critical error (TP→FP): real threat is suppressed

• Over-escalation (FP→TP): harmless finding is marked as a threat

• Anomaly suppression (Inc→FP): finding requiring review disappears

The benchmark uses several metrics:

• classical confidence-weighted scoring

• operational triage scoring

• balanced scoring based on FP, Inc and TP

• critical miss rate

• threat capture rate

• false review load

• cost

• speed

The goal is not to make the benchmark more complex for the sake of complexity. The triage workflow itself is convoluted. This is not only about correctness, but also about how the model makes mistakes in a way that the analyst workflow can withstand.

Naive baselines

It quickly became obvious that such a benchmark needs naive baselines. Otherwise, the numbers may look better than they actually are.

For this task, there are three obvious baselines:

• classify everything as false positive

• classify everything as inconclusive

• classify everything as true positive

No one would implement such strategies, but they are useful as a reference. They show what the model must outperform for its behavior to be considered meaningful.

The "everything as false positive" strategy is dangerous. It suppresses everything. In an environment where most findings are harmless, this may look good on some metrics. It creates almost zero load for analysts. But the problem is obvious: it also suppresses all real incidents.

The "everything as true positive" strategy has the opposite problem. It avoids suppressing real threats, but turns every finding into an incident. On paper this looks safe, but in practice it destroys the workflow. Analysts have to process everything as urgent. This is not triage, it is panic as a classification strategy.

The "everything as inconclusive" strategy is more interesting. It leaves everything visible for manual review. From a security perspective this is attractive, because it does not hide TP or anomalies worthy of review. But this means that the model does not actually reduce the load. It simply passes the entire pile on to the analyst.

This baseline has caused the most problems in scoring. If a benchmark rewards safety too heavily and does not penalize review load enough, "everything as inconclusive" starts to look like a good strategy. But that is not true. It is just a safe fallback. It catches everything but filters almost nothing. A useful triage model must balance between these extremes. It must not behave like "everything is a false positive" because that hides incidents. It must not behave like "everything is a true positive" because that turns triage into alert spam. And it must not behave like "everything is inconclusive" because that just pushes all the workload back to analysts.

The model must do something more complex: keep dangerous and worthy findings visible for review, while suppressing enough harmless noise to be useful.

Why there is no single best model

After adding operational scoring and naive baselines, it became obvious: there is no single best model for this task.

Security triage is not a single fixed optimization task. Different teams care about different types of errors. In some environments, missed incidents cannot be tolerated. Others process so many findings that a model with high review load is not useful, even if it is very safe. Some need on-premises processing. Some care about cost. Some care about speed, as they want to process thousands of events in a short time. Instead of looking for a single global winner, we started looking at model profiles.

The high-safety profile prefers models that almost never suppress true positives. Such models may send many findings to analysts, but that is acceptable in environments where missing an incident is the worst outcome. For example, critical infrastructure, high-value systems, regulated environments, or post-compromise investigations, where the priority is to not miss anything important.

The balanced SOC profile tries to keep both sides under control. A low critical miss rate is needed, but it is also necessary to reduce the workload on analysts. This is probably the most common scenario: the model must help analysts process findings faster, but not suppress too many.

Noise-reduction or high-volume profile is distinct. Here, the main thing is volume. The model must suppress enough harmless findings for the workflow to scale. But this profile is dangerous without guardrails. Low verification load is useful only if the model doesn’t achieve it by suppressing real threats.

There is also a deployment aspect. A model via vendor API can be convenient and often works well, but it is not always applicable. Some environments cannot send security findings to an external provider. Some want a predictable fixed cost. Some want full local control. Some want to run open-weight models on their own hardware, even if it means slower processing or lower quality.

That’s why we divided the results into tiers:

• Closed source / vendor API

• Open source / pro hardware

• Open source / consumer hardware

This makes the benchmark more practically useful. A global leaderboard can say which model performed best overall, but won’t say which model fits your constraints.

First results

When the scoring was in an acceptable form, the most interesting part began: running a large set of models on the same findings and analyzing their behavior.

The first thing I realized: a regular leaderboard is insufficient. A model can look good by the classic confidence-weighted score but be a dubious choice for security triage if it has too many critical errors. In this benchmark, a critical error occurs when the ground truth is TP, but the model classifies it as FP. This is the error that concerns me the most because it means a real threat is being suppressed.

Therefore, the most useful charts are not simple ones stating "model A is better than model B", but trade-off charts. One of the most important ones is Critical Miss Rate vs False Review Load. Critical Miss Rate shows how often a model suppresses real threats. False Review Load shows how many harmless findings still reach an analyst. Ideally, a model should score low on both metrics: it should not hide real incidents, but also reduce noise. The ideal model would be in the bottom-left corner of the chart.

In practice, this is difficult. Some models are safe but noisy. They keep TP visible, but send a lot of FP for review. This may be acceptable for high-safety workflows, but it does not reduce the workload for analysts in high-volume environments. Other models more aggressively reduce review workload. They appear effective, but some of them pay for this with a higher critical miss rate. This is a dangerous trade-off. A model that reduces noise by suppressing real threats is useless.

The next useful chart is Balanced OTS vs False Review Load. Balanced OTS is our operational score, balanced across FP, Inc and TP. It prevents the benchmark from being dominated by the majority class. This is important because the results of real scans are often heavily skewed toward FP. If you only evaluate based on the majority class distribution, a model may appear better than it actually is, being good at suppressing harmless findings but performing poorly on rare critical cases.

Balanced OTS shows whether the model makes useful operational decisions across all classes. False Review Load shows whether it reduces the load on analysts. Again, a trade-off is important. A model with high Balanced OTS but high False Review Load may be safe but noisy. A model with low False Review Load but weak Balanced OTS may be effective but too risky or superficial. Models that maintain operational safety while reducing noise are of interest.

Another graph compares CW% and Balanced OTS. It is useful because it shows how classic classification quality and operational utility can diverge. CW% rewards closeness to the expert answer, taking into account confidence. This is still valuable but does not fully reflect the operational cost of different errors. Some models look good in terms of CW% but are less attractive when considering Critical Miss Rate, anomaly suppression, and false review load. Others may not look impressive in terms of classic scoring but behave more conservatively in aspects useful for triage.

The initial results revealed something unexpected: the largest or most expensive model is not always the best choice for this workflow. Some lighter models performed surprisingly well. In some cases, they were a better operational choice than larger pro models from the same family. My current hypothesis: some of these models behave more cautiously. They may be less prone to making strong assertions, which helps in security triage, where suppressing a real threat is the worst mistake.

Early results also made tier-based segmentation more important. Closed-source API models are convenient and often powerful, but they are not always applicable in environments with strict data control requirements. Open-source models running on pro hardware may be more attractive for controlled deployments. Smaller open-source models running on consumer hardware may be appealing for local triage or workflows with limited budgets, even if they do not top the global leaderboard.

Why we do not publish raw benchmark data

There are two reasons why we publish results, graphs, and scoring methodology, but not raw benchmark data.

The first reason is benchmark contamination. If we publish exact findings, prompts, expected classifications, and ground truth labels, this data may end up in future training datasets. Even if model providers do not intentionally train on this benchmark, public repositories are scanned, copied, mirrored, and used in various ways. Once raw benchmark data becomes public, I can no longer be certain that future models have not seen it.

This will gradually make the benchmark less useful. It will measure not only how well the model handles this security triage task, but also how much of the benchmark or similar data has leaked into training materials.

The second reason is more practical: some reports and findings are based on real cases or results from actual scans. Even if the data is scrubbed, I do not want to publish raw findings if I am not sure that all sensitive context has been removed. Paths, hostnames, usernames, hashes, tool locations, timestamps, internal software names, and surrounding forensic context can reveal more than expected.

Therefore, the public repository contains what I consider useful and safe to publish:

• scoring approach

• result tables

• charts

• benchmark methodology

• model comparisons

• operational profiles

It does not contain raw THOR reports, private ground truth files, or detailed case material accompanying the findings.

How to use the benchmark

I would not use this benchmark as a simple leaderboard. The first question should not be "which model is in first place?", the better question is: which model fits my workflow and risk tolerance?

For security triage, I would start with the type of error I can tolerate the least. In most workflows, this is a critical miss: a TP classified as an FP. If a model suppresses real incidents, I would not justify this with cost, speed, or a high overall score. Therefore, the first metrics to check are the Critical Miss Rate and Threat Capture Rate. The model must keep TPs visible, either as clear TPs or as Inc cases that still reach a human.

Next, I would look at the False Review Load. This will show whether the model reduces the workload for analysts. A model can be safe, but still send too many items for review. This may be acceptable in high-safety workflows, but it does not solve the problem of high-volume triage.

Only after the security and load profile looks acceptable would I look at cost and speed.

Deployment constraints are also important. Some teams may use vendor APIs. Others need local processing, open-weight models, or predictable fixed costs. Therefore, the benchmark divides models into tiers. The best global model may not be applicable if it does not fit your data control or infrastructure constraints.

My practical order of analysis:

1. choose a deployment tier that you can use

2. choose an operational profile that matches your risk tolerance

3. discard models with unacceptable Critical Miss Rate

4. check Threat Capture Rate and Anomaly Capture Rate

5. compare False Review Load

6. then compare Balanced OTS, cost, and speed

This benchmark should narrow down the search space. It should not replace your own testing. If you want to use LLM in a real SOC or forensic workflow, test it on your alerts, false positives, internal tools, and risk tolerance.

Current status and next steps

This benchmark is already mature enough for publication but is not yet finished. Current results already show useful patterns. Some models are safe but noisy. Some are more aggressive in reducing review load. Some look better by classical scoring than by operational security. And some small models work better than expected for this specific workflow.

Next step - carefully expand the dataset. More findings are useful, but only if they add real signal. Adding hundreds of simple FP will make the benchmark larger but not better. Diversity is more important: more TP, more uncertain/legitimate anomalies, more operating systems, more artifact types, more weak signals, and more realistic edge cases.

The most challenging part is TP. They need to look and behave like real findings, not like artificial test objects. If a sample only looks suspicious but doesn't have a real malicious function, the result becomes ambiguous. One model might aggressively flag it, while another might examine it more closely and classify it lower. This doesn't necessarily mean one model is better. It's possible the test case wasn't clean enough. I'll also be adding new models as they become available. Models change, provider defaults change, prices change, open-source models improve rapidly. Scoring might evolve too, but the basic idea should remain: critical errors should be more expensive than false escalations, and anomalies worth checking shouldn't be treated as one-off noise.

Raw findings and expert ground truth will remain private. The public part is the methodology, the scoring approach, graphs, and aggregated results. That's the part I consider useful to publish. This benchmark is a baseline. It shows how models behave on the same enriched findings, without external tools, memory, feedback, or customer-specific context. A real deployment might work better with context from SIEM, EDR telemetry, allowlists, analyst feedback, or RAG memory. But then you're measuring the entire workflow, not just the model.

Use this benchmark to narrow down candidates, not to outsource the final decision. Final testing should still happen in your environment, with your tools, your data constraints, your false positives, and your tolerance for errors.