Response Pain: How Information Security Processes Affect Incident Investigation

The effectiveness of the investigation depends on three key factors: the competence of the response team, the availability of tools for analysis and data collection, and the consistency of the information security processes of the customer organization as a whole.

The last factor has a significant impact on the course of the investigation, including the accuracy, completeness, and reliability of the final investigation results. In particular, the lack of established information security processes can lead to changes in key artifacts, complicate the identification of the intrusion source, and reduce the likelihood of detecting new indicators of compromise (IOCs).

Throughout numerous investigations, the Solar 4RAYS team has encountered such "obstacles." Having analyzed them, we decided to describe the problems that arise in one information security process or another, which negatively impact the incident investigation.

When Problems Arise

Classical investigation models divide the response process into six stages: preparation, detection, containment, eradication, recovery, and lessons learned.

In reality, the interaction between stages may not be linear, so within this article we will consider problems at three conditional stages of the investigation:

1. Before the incident investigation.

2. During the investigation.

3. After the investigation.

Before the Incident Investigation

Yes, We Accepted the Risks

Every organization has a cyclical process known as information security risk management. It is aimed at identifying, assessing, prioritizing, and reducing risks associated with potential incidents. There are several management strategies for this process, one of which is "Risk Acceptance."

"Accepting risk" is a strategy that involves a conscious decision to ignore measures to reduce the likelihood of a risk occurring or to eliminate a threat. This approach may be driven by economic factors, i.e., it may be applied if the expected damage from the realization of the threat is lower than the cost of preventing it. However, within the framework of information security risks, we see a distortion of this strategy. Often, it is accompanied by a mental consensus: "Why spend money on something that may not happen?"

This approach inevitably leads to a situation in which the "accepted" risk occurs, an incident happens, and the organization is completely unprepared for it. The consequences of such unpreparedness, especially in the current realities of the cyber threat landscape, lead to inevitable economic and reputational costs. Moreover, the costs in most cases exceed any expectations of the organization.

In practice, this manifests itself in the form of ignoring credible warnings. We at Solar 4RAYS track the activity of known groups and, when we see signs of an attack, warn the attacked organization about the compromise of its infrastructure. However, these alerts often remain unanswered. Sometimes, in response, they ask to provide indicators of compromise, but no further investigation is conducted. Often, such inaction has led to the development of an incident and its public disclosure.

At the same time, the "Acceptance" strategy is not an error in itself. However, its application must be a conscious decision based on an honest analysis of the potential damage in the event of an incident, and not just an assumption that the incident may not happen. The practice of our investigations shows that literally any organization can become the target of an attack by a professional group.

What? Where? When?

The Solar 4RAYS team has encountered situations where, during an incident, the customer did not know exactly what their own infrastructure consisted of. This most often occurs due to a poorly configured asset management and inventory process. Similar problems also arise with access control and account administration. As a result, a situation occurs where the victim, together with the investigation team, conducts inventory on the fly, essentially engaging in live auditing, with the particularity of partially disconnected infrastructure.

Lack of information about infrastructure segments has a negative impact on the investigation process. In particular:

1. Creates the need for live auditing. In the absence of up-to-date information about the infrastructure, the IR team has to identify web servers, domain controllers, privileged accounts, etc., instead of the investigation itself. This takes time that investigators may not have. For example, in cases where attackers still maintain a presence in the infrastructure and can cause damage.

2. Leads to a lack of consistency in isolation. The response team may encounter a situation where the wrong things are "isolated" due to poor awareness of the infrastructure.

No matter how obvious this may sound, the only way to know what your infrastructure consists of is a well-established asset management and inventory process. This is good practice even outside the context of cyber incident investigation, but in the event of a cyberattack, it significantly increases the efficiency and speed of investigators' work.

It is extremely important that, in an incident situation, it does not turn out that the attacker knows your infrastructure better than you do yourself.

Example from a real case

The attackers discovered an outdated domain controller in the victim's infrastructure that, although decommissioned, maintained a connection with other network components. Upon initial access, the threat actors deployed 3proxy in the system, which allowed them to tunnel traffic. Subsequently, the attackers downloaded ProcDump to dump memory processes, specifically lsass.exe.

Subsequently, attackers used a domain controller as an intermediate node for lateral movement across the entire infrastructure. For the customer, this "discovery" came as a complete surprise.

If I don't see it, that means it doesn't exist?

The network infrastructure of any organization is a complex and dynamic system that requires constant attention both from a technical standpoint and a security perspective. Given the steady increase in the number of attacks, it is impossible to maintain infrastructure security without a well-organized monitoring process.

The essence of the monitoring process is not just simple data collection, but continuous analysis of that data using additional tools. The goal of the entire process is to detect anomalies indicating potential malicious activity before actual damage occurs. The result of the analysis is a notification about a suspicious event, which is in turn assessed directly by the recipient of the notification (the customer).

Unfortunately, there are serious risks involved in this process. Even when reliable alerts are present, the notification recipient may ignore them, assess them incorrectly, or misinterpret them (false positive). As a result, this can lead to catastrophic consequences such as data encryption or data leaks.

In fact, ignoring an alert or incorrectly assessing it due to negligence or insufficient awareness not only fails to protect the organization, but also creates a false sense of security. A mindset takes hold: "Some events are going on, the process is set up, which means I'm safe and don't need to do anything".

Another risk is insufficient monitoring coverage of key infrastructure segments, which creates blind spots for the monitoring service.

The consequences of the above-mentioned risks impact the investigation process, specifically:

1. A significant delay between alerting and responding can lead to the loss of critical artifacts: event logs, activity traces, temporary files.

2. The presence of blind spots can slow down the detection of infected systems and monitoring of IOCs.

3. In the event of infrastructure destruction (e.g., as a result of encryption), the investigation begins to depend on the availability of backups and monitoring logs. The lack of monitoring (and, as a result, its logs) can negatively affect the effectiveness of the investigation.

The only way to eliminate such risks is a responsible approach to monitoring. It is important to escalate clearly malicious activity to a responsible person, even if the latter previously stated that the activity was "legitimate". If the event turns out to be a false positive, the consequences of such escalation are minimal, while ignoring a real threat can result in huge losses of funds and time.

In turn, the person who received the notification is obliged to have insight. To do this, it is important that they regularly familiarize themselves with fresh reports on the activities of hacking groups, imagine what the use of certain tactics and techniques by attackers looks like.

It is important to cover with monitoring the key segments of the organization's infrastructure, namely: the server segment, with web servers, databases, etc., the demilitarized zone (DMZ), which connects the internal network segment with the external one, and others. The presence of gray areas can allow attackers to hide in them.

Similar logic applies to other security tools: WAF, EDR, AVPO. Ignoring their alerts is also a mistake.

Saw, got scared, deleted

Imagine that an organization's employee has discovered anomalous activity on their system. For example, an unusual directory on the desktop that was not there before, or an executable file with a suspicious name or location. Upon quick assessment of such activity, the user realizes that it is malware and simply deletes the suspicious object, believing that they have thereby prevented an incident. At best, they will then run a scan using antivirus software. However, as practice shows, such actions do not prevent an incident, but can only delay it.

We have called this approach "Saw, Got Scared, Deleted". In its essence, it is erroneous not so much in the deletion procedure itself, but in the lack of a systematic approach to responding to such events. At the same time, the greatest danger is not the attempt to resolve the incident independently, but the lack of escalation of detection to a specialized specialist.

For the investigation team, this approach poses the following risks:

1. When a suspicious file is deleted, a sample of malware and/or a new C2 server of the attackers may be lost forever. In this case, it is often impossible to recover the file due to various circumstances.

2. From the moment of "Saw, Got Scared, Deleted" to the moment of "Call the Response Team", enough time may pass for the rotation of various artifacts. All this negatively affects the efficiency and accuracy of the analysis of the events that occurred in the system.

3. The user may forget or intentionally hide (for example, out of fear of disciplinary action) that something suspicious was happening on the system, and will not report the "interesting findings" during the investigation.

Solving this problem requires a comprehensive approach consisting of two parts. Firstly, the implementation of mandatory monitoring will avoid ignoring the incident or not saving the fact of the event. Secondly, training employees in at least the basics of information security allows for the implementation of a meaningful approach to threat detection.

The ideal scenario for employee behavior in the above cases may include the following actions:

1. The most important thing is to escalate what happened to a specialist in the field. Hiding what happened can lead to catastrophic consequences.

2. After escalation, you should not take any independent actions. They may interfere with determining the cause of the incident.

In other words, "Saw, got scared, deleted" should turn into "Saw, reported to Information Security, followed instructions from specialists."

It is worth mentioning separately the scenario in which an organization chooses to respond independently. In this case, it is necessary to document in detail the discovered artifacts of malicious activity. In the future, when investigating new incidents, a base of "historical" knowledge will help either to trace the development of an old incident and understand where the taken protective measures failed, or to establish that the activities are not related and not waste time on a dead-end branch of the investigation.

A detailed report on independent response will help not only you, but also a third-party response team, if it becomes necessary to involve it.

Cases when a customer or their employee deleted traces of infection according to the principle "Saw, got scared, deleted" are quite common in the practice of the Solar 4RAYS team. In a typical case, an administrator or any other employee sees a notification from the anti-virus protection system indicating the fact of infection. Acting according to the above scenario, they do not save any data about the event and its environment. Deleting one sample of malicious software, as a rule, does not limit the capabilities of attackers in any way, and a more serious incident, of which the deleted malware was a part, sooner or later occurs.

The heads of the organization ask themselves: "How was the system infected?" — and decide to conduct an analysis of the system. As a result of this amateur activity, some of the data is destroyed by the organization's employees, and the incident investigator is left to hypothesize, based on disparate facts, about how the attackers penetrated the network.

We should separately note the cases when APT groups use malware that exists only in the system's RAM. And the customer, after being informed that the systems are infected, simply performs an emergency shutdown of the entire infrastructure. As a result, after turning on the systems are already cleaned of malware and there is simply nothing to analyze. But this by no means implies that the group has not retained access to the infrastructure.

In the course of the investigation

Incident Management

Sometimes at the initial stages of investigations, our team encountered the problem of the customer's unpreparedness to respond to the incident. We have repeatedly witnessed the customer being torn apart, trying to simultaneously find the person responsible for the infrastructure, start helping us in the investigation, and begin to restore the infrastructure.

Coordination of actions during an incident is a rather rare phenomenon, and it is observed only in those organizations where there is a pre-developed response plan.

Developing such a plan is not just a formal compliance with requirements. It is a practical step-by-step scenario of interaction between key participants in the process (from technical specialists to management), which will be very useful in a situation where it seems that everything is lost. The basis of such a plan is the information security incident management policy, which defines responsibility, notification procedures, incident classification, and other actions.

At the same time, it is necessary to separately work out a plan for interacting with external investigation teams. Such teams can provide recommendations on certain actions with your infrastructure to help restore it and preserve significant artifacts for investigation. The lack of an interaction plan inevitably leads to problems during the investigation.

Let's give a typical example:

During the investigation of the incident, a trail was discovered leading to a compromised system. But in response to the requested data, the customer sent a clean triage/image with a freshly reinstalled system. We requested more systems, and they turned out to be exactly the same. It turned out that the customer had already partially restored the systems from a golden image, without even thinking to warn us about their intention.

In single-threaded mode

Usually, on the customer's side, a very small team is allocated for communication with the response team. Often, a single employee - a system administrator - becomes the direct executor of the operational instructions of the investigation team. They have all the necessary rights to collect data, and also have more detailed information about the infrastructure "on the ground".

This creates some problems. In most organizations, the number of administrators is limited. Their main duty after an incident is to restore infrastructure, update access, change configurations, and they cannot always effectively allocate time between collecting data for response experts and their direct responsibilities.

To avoid overloading the employee, the best practice is to allocate a small amount of time for them to solve tasks from the investigation team. Many tasks that are set are often related to data collection, i.e., launching utilities or executing specific commands. After that, the employee can be free until the data collection is completed.

In turn, the investigation team is obliged to set specific tasks and at the same time explain how to perform one or another action in order to facilitate the administrator's work.

Everything is stored behind seven locks!

Our team has encountered cases where the provision of data for research was complicated by the customer's fear of possible loss or leakage. Sometimes there were attempts to delete some artifacts from the collected data. However, in our opinion, such concerns are excessive. Moreover, an attempt to delete any collected data may affect the results of the investigation.

First and foremost, the incident investigation and response service implies that the parties sign a non-disclosure agreement (NDA) that stipulates all necessary obligations and terms. Moreover, the non-disclosure agreement specifies the retention periods for incident data.

From a technical standpoint, the Solar 4RAYS team stores all data provided by the customer exclusively in secure storage facilities with a multi-level access control system based on the principle of least privilege. Access to data is granted to a specialist for a limited time, and all actions with files are recorded in an audit log.

As a result, data transfer and processing within the scope of the investigation are completely secure. And the customer’s "doubts" can negatively impact the investigation process.

After incident investigation

Who needs these recommendations of yours…

Recommendations are the quintessence of the incident investigation results. They systematize specific well-founded actions necessary for eliminating the consequences of compromise, restoring infrastructure integrity, and reducing the risk of reinfection. They include not only technical instructions but also organizational recommendations to strengthen overall security. Special attention is paid to compromise indicators that must be monitored even after the incident.

Practice shows that recommendations are sometimes not given due attention, or they are completely ignored. Such "oversights" not only reduce the effectiveness of the investigation, they directly contribute to repeated compromise.

And no matter how obvious this may sound, the most effective way to prevent repeat incidents is strict compliance with the recommendations. They are not a formality. Each of the actions specified in the recommendations is conditioned by threat analysis, attacker actions, and investigation outcomes. Each step is part of a sequential chain, and breaking this chain can leave the system vulnerable to similar or related attacks.

Let's give an example from incident investigation practice:

The customer requested a limited incident investigation. During the process, multiple traces of APT group activity were discovered.

As a result, we provided a number of recommendations: what to do with the infrastructure, which indicators of compromise to monitor and how. However, relatively soon the same customer came to us again — during monitoring, their specialists discovered connections to the attackers' C2, mentioned in our report.

It turned out that the recommendations were implemented only six months after the investigation. During this time, the attackers continued to infect the extensive infrastructure and began to use a new C2.

Conclusions

The above-mentioned problems individually do not always hinder the investigation, but together they can prevent it or distort the final results.

In modern conditions of a high level of cyber threats, a systematic approach to information security processes is not a recommendation, but a harsh necessity.

Every organization, regardless of size and industry, must have a clear plan of action in case of an information security incident. In addition to the plan, there must be established processes for asset management, employee training in information security basics, and monitoring.

Particular attention should be paid to the principle of preserving digital traces related to the incident. The traces left by the attackers are key evidence of malicious activity. Their alteration or loss can complicate the investigation process and hinder the accuracy of the results.

All this will significantly reduce the risk of an incident, and in case of its occurrence, established processes will help to conduct a high-quality investigation in the shortest possible time.

____________________

The team of the Solar 4RAYS cyber threat research center Solar 4RAYS