How one developer prevented the largest cyberattack: the story of the XZ Utils hack

00:31
24.05.2025
klimensky
19

Exactly one year has passed since the world watched in awe as the investigation of one of the most sophisticated backdoors in Linux history unfolded. The story of the XZ Utils library resembled a thriller: infiltration under a real name, trust from the community, backdoors in the code — and a random discovery on an ordinary working day.

On March 29, 2024, programmer Andres Freund woke up early, as usual. The coffee machine was already hissing in the kitchen, and the laptop was blinking with a familiar update indicator. Andres loved the morning hours: while the city was just waking up, he was already immersed in his routine—tests, logs, CPU usage graphs.

In the morning, he ran his standard set of tests. Everything looked normal: the graphs were stable, the CPU wasn’t overloaded, no bugs in sight. And suddenly—a peculiarity. An insignificant error, but not the kind you just ignore. Andres frowned. "What was that?" he muttered. He connected to the server via SSH to check the details and noticed another anomaly: a 500 millisecond response delay. Half a second. For most, no big deal. But for Andres—it was the first warning sign. He started digging deeper.

Background

Meanwhile, thousands of kilometers from Andres, someone was clearly rubbing their hands in satisfaction. Their plan was nearly complete: step by step, almost imperceptibly, they had prepared one of the most sophisticated attacks on open source infrastructure. The target was the XZ Utils project—barely noticeable, yet critically important for most modern Linux distributions.

XZ Utils and its underlying library liblzma are used for data compression and are integrated into system utilities like ssh, systemd, and many others. A subtle failure in their operation could spell disaster for servers powering hospitals, government agencies, telecom networks, and cloud platforms.

According to CISA, a vulnerability in XZ could allow an attacker to gain remote access to such systems. It truly was a key to the digital infrastructure of entire countries—and possibly one of the most disturbing incidents in open source history.

Created in 2005 by software engineer Lasse Collin, the .xz file format uses the LZMA compression algorithm, providing 30–40% better compression than gzip. The format quickly gained popularity and is now actively used for packing tar files as well as compressing system archives and other important data.

Why XZ Utils Was Attacked

Because it ended up as part of many other popular tools. One of them—OpenSSH—serves as the de facto standard for secure remote server access. Millions of systems worldwide rely on it, from home servers to critical infrastructure.

It's key to understand that OpenSSH (more precisely, its server component sshd) doesn’t directly use the liblzma library, which is part of XZ Utils. However, in some popular distributions, such as Debian, sshd can be built with systemd support—the system manager that handles service initialization upon Linux startup.

Here is where it gets interesting. systemd may call liblzma when processing compressed configuration files. That is, if malicious code is injected into liblzma, it could be triggered before sshd starts and affect its operation.

Thus, to embed a backdoor, the attacker does not need to break into sshd or systemd directly — that would attract attention. Instead, they find a vulnerability in a less obvious library that is included "in chain." Namely, in liblzma, which turns out to be the perfect "entry point."

The hack that was planned for years

It sounds simple enough, but to exploit this vulnerability, a long journey was required. It all began back in 2021. At that time, a mysterious hacker registered on GitHub under the name Jia Tan (JiaT75, Jia Tan) and began sending patches to the xz-devel mailing list. Gradually, taking his time, he gained trust.

His first patch in this mailing list added a file called ".editorconfig." It was quite harmless and did not raise suspicion. In November of the same year, Jia Tan once again attracted attention by asking to fix an obvious build issue.

At the same time, the hacker (or group of hackers) began using subtle social engineering tactics. The main goal was to convince the XZ Utils maintainer, Lasse Collin, that handling the project alone was becoming increasingly difficult, and it was time to bring in someone else from the "trusted" helpers.

Lasse had been working on XZ Utils almost single-handedly for many years — out of interest, by a calling. But at some point, the passion started to feel like a heavy duty. The workload was increasing, while time and energy were decreasing. Especially with personal health problems. Objectively, help was indeed needed.

And this is exactly what Jia Tan took advantage of.

In April 2022, he sent another seemingly insignificant patch. However, it was with this that a new character appeared on the stage — Jigar Kumar. Formally, he had no connection with Jia Tan, but quickly began to promote the idea that Lasse needed a "partner" so the project wouldn't die. And Jia was an excellent candidate.

Jigar Kumar begins to criticize Lasse Collin, the maintainer of XZ Utils, carefully but persistently. He hints that the development has almost stalled, updates are not being accepted, and all this is slowing down the project's progress:

Patches have been sent to this mailing list for years. There is no reason to believe that anything will appear in the near future.

The subtext is simple: if things continue like this, the project could completely stall. And therefore, Lasse should pass some of the responsibility to someone more "efficient." For example, Jia Tan.

Later, accusations were joined by someone named Dennis Ens. He sent an email to xz-devel asking about XZ support for Java and also expressed dissatisfaction with the work of the maintainer.

Lasse Collin apologized for the delay in responses and explained that he was currently working on many requests. But the conflict did not end there.

In June 2022, a discussion on the pace of development and the need to involve new participants unfolded on the XZ Utils developers' forum. On June 7, a participant under the pseudonym Jigar Kumar wrote:

There will be no progress until a new maintainer appears. XZ for C also has rare commits. Dennis, you’d better wait for a new maintainer or fork it yourself. Sending patches here these days is pointless. The current maintainer has lost interest or no longer wants to maintain it. It’s sad to see such a repository.

To this, Lasse Collin replied:

I have not lost interest, but my ability to work on the project has been quite limited, mainly due to prolonged mental health issues, but also because of some other things. Recently, I worked a bit with Jia Tan on XZ Utils, and perhaps in the future, he will have a more significant role, we’ll see. Also, keep in mind that this is an unpaid hobby project.

After several weeks of pressure and a flow of emails urging the project to be handed over to new hands, a commit from the user JiaT75 appears in the xz repository. At first glance, it seems like an innocuous patch. But this became the turning point: Jia Tan started advancing quickly, soon becoming the second most active participant in the project.

Meanwhile, Jigar Kumar and Dennis Ens disappear from sight. Their names no longer appear online, except for a couple of mentions in emails to Collin. But Jia Tan has firmly established himself. By October 2022, he is already part of the Tukaani organization on GitHub, and Lasse increasingly assigns him key tasks.

Andres Freund and the Prevention of the Attack

The impending disaster was prevented by engineer-programmer Andres Freund. He is 38 years old, lives in San Francisco, and works at Microsoft. His job involves developing part of the open-source software for PostgreSQL.

The story began in the winter of 2024 when Freund was returning from Germany, where he had flown to visit his parents. While reviewing the automated test log, he noticed several strange error messages. At the time, he had a terrible headache, was suffering from jet lag, and the messages didn't seem urgent, so he postponed them.

But a few weeks later, on March 29, 2024, while conducting more tests at home, he noticed a delay in SSH performance when running an unstable version of Debian.

Andres himself describes his thoughts at that moment:

There was no graphical user interface or anything else running. So, it was like there shouldn't have been any CPU usage at all because it was idle. But sometimes there were these 500-millisecond and 700-millisecond periods of high CPU usage that then dropped again. That was unexpected. So I ran the profiler.

And this was the first moment when the strange symptoms became noticeable. Usually, when profiling a system with all debug symbols installed, you should have attribution for where time is being spent, including the symbol name. But in this case, for most of it, there was no visible symbol name at all.

There could have been various reasons for this: maybe I forgot to install something, or there was some kind of JIT compilation, or something else that could have caused it. But none of that was true. So I started looking — what could be the cause. And at first, I didn’t think it was a backdoor or anything like that. It just seemed like a weird symptom. But the more time I spent investigating, the weirder it got.”

The trail led him to the XZ Utils utility — and then he remembered: similar bugs had already popped up before. Maybe these were all parts of the same story?

Silhouette of a man in a dark room with a laptop, the background wall covered with schematics and diagrams

Freund discovered that XZ Utils had been deliberately modified with malicious code over the last three years. The changes were small and unnoticeable, but when combined with other parts of the Linux operating system, they essentially gave the hacker full access to the entire server. The backdoor was being prepared for inclusion in Linux versions that would have reached almost every modern internet-connected server.

“It felt surreal,” he said. “There were moments when I thought I must have just slept badly and was seeing fever dreams.”

But his investigation kept turning up new evidence, and on March 29, Freund posted a message about the backdoor to the Openwall mailing list.

This news caused an explosion in the tech world. The problem was fixed, and Freund became a hero who prevented a massive cyberattack. In an interview, Andres said that this kind of fame was disorienting: “I find it very strange. I’m a pretty private person who just sits in front of a computer tinkering with code.”

The essence of the backdoor

In the very first days, researchers rushed to study the backdoor’s structure. It was initially assumed that the goal of this backdoor was to bypass SSH authentication. However, further analysis showed that the backdoor provides remote command execution (RCE) with the privileges of the sshd daemon, operating at the pre-authentication stage.

If the information from the remote certificate matches the backdoor data, it’s decrypted using the ChaCha20 algorithm. After successful decryption, the data is passed to the system() function, allowing attackers to execute commands before the authentication process is completed. This makes the vulnerability much more serious than simply bypassing authentication with public keys.

Technically, the backdoor was implanted during the creation of the tarball for the release, but it cannot be found in the XZ repository on GitHub. The malicious code was hidden from developers because it was added only into the source tarball releases, not public repositories. This ensured relative stealth for the backdoor, despite its use in builds of dependent projects.

Researchers describe in detail the construction of the backdoor, which consists of several components injected through different commits. The malware uses the IFUNC mechanism to intercept symbol resolution functions and includes an obfuscated shared object hidden in test files. During the library build, a set of scripts is triggered to extract this shared object, which is not included in the repository and is added to .gitignore. Additionally, network access blocking is disabled, which is a security measure to limit process privileges.

The backdoor execution process involves several stages: first, the build-to-host.m4 script is launched, which decodes the “test” file bad-3-corrupt_lzma2.x into a bash script. This script then performs complex operations to decode another file and extracts the shared object liblzma_la-crc64-fast.o for compiling the liblzma library. This construction makes the backdoor difficult to detect without direct code analysis.

Unlike simple authentication bypass methods, this backdoor uses remote code execution, making it harder to detect with traditional admin tools. For instance, code strings related to audit hooks do not appear in the exploit binary. Instead, a trie is used to store strings, which further complicates the task of identifying the malicious code.

The shared object is compiled into the liblzma library and interferes with the dynamic linking mechanism. When a process is launched, function pointers such as RSA_public_decrypt from OpenSSH are replaced with addresses of the malicious function. This function extracts commands from the client certificate that has passed authentication and passes them to system() for execution, allowing arbitrary code execution (RCE) even before authentication is complete.

The Secret of Jia Tan

The backdoor process has been thoroughly studied, but the perpetrator behind the attack has not yet been identified. Who is Jia Tan? Is it a person or a group? And what motivates their actions? At this point, it remains one of the most intriguing mysteries in information security.

Almost nothing is known about Jia Tan’s identity. Independent security reporter Brian Krebs writes that he was unable to find “any trace” of Jia Tan’s email addresses: “The email addresses used for at least a couple of years by the parties involved have absolutely no trace in any data breach or database outside of Github/Gitlab, and possibly Tukaani, Debian, and a few mailing lists. Normally, when I see that, I assume we’re dealing with a disposable or single-purpose email address, which was created either for fraud, or because someone is extremely paranoid about privacy.” A similar picture is observed with the accounts of Jigar Kumar, Dennis Ens, and Hans Jansen.

But researchers did manage to gather some information. Jia Tan was present in the #tukaani IRC channel on Libera.Chat. A /whois revealed the connecting IP and traced activity on March 29.

Running Nmap on the IP shows many open ports, which likely indicates a proxy, hosting provider, or something similar. It appears that Jia Tan was connected from the Singaporean IP address 185.128.24.163, but this Singaporean IP is associated with the Witopia VPN service.

Next, the researchers analyzed the attacker’s time zone. At first glance, Jia Tan certainly looks like someone living in East Asia. The time zone of their commits is UTC+8: that's China’s time zone, and it’s only one hour different from North Korea’s.

Huntress researcher, working under the pseudonym “Alden”, published a visualization of Jia Tan’s malicious commits in XZ Utils.

However, an analysis by two researchers, Rhea Karthi and Simon Henniger, suggests that Jia Tan might have simply changed the time zone on their computer to UTC+8 before each commit.

“Another sign that he is not from China is the fact that he worked during well-known Chinese holidays,” Karthi and Henniger say. They note that Jia Tan also didn’t push new code on Catholic Christmas or New Year’s Day.

“We notice that Jia’s work schedule and holidays seem to match an Eastern European more than a Chinese person.

Chinese public holidays (considering only 2023):

Worked on September 29, 2023: Mid-Autumn Festival;
Worked on April 5, 2023: Tomb Sweeping Day;
Worked on January 26, 22, 23, 24, 26, and 27, 2023: Lunar New Year.

Eastern European holidays:

Didn't work on December 25: Christmas;
Didn't work on December 31 or January 1: New Year.

Additionally, during the investigation it was found that most of Jia Tan’s work was done between 9 a.m. and 5 p.m. for an Eastern European or Middle Eastern time zone. The most common workdays were Tuesday (86), Wednesday (85), Thursday (89), and Friday (79).

“The commit time range suggests this wasn’t just some project done outside of work hours,” the experts believe.

Security researchers agree that a nation-scale hacker group is hiding behind the Jia Tan persona.

Costin Raiu, former head of the analysis team at Kaspersky Lab, believes that a state-level structure with long-term strategic goals, ready to invest time and resources in deep infiltration of open-source projects, is behind this activity.

While Raiu allows for the possibility that the backdoor design could have been created by American hackers, he considers it unlikely. He is convinced that the National Security Agency would have used more sophisticated cryptographic methods.

Costin Raiu suggests that the attack was carried out by non-American groups such as the Chinese APT41, North Korean Lazarus Group, or Russian APT29.

Researcher Joe Christian suspects the North Korean Lazarus Group or Diamond Sleet: “This particular group has actively exploited vulnerabilities in software supply chains and has targeted CI/CD (Continuous Integration/Continuous Deployment) infrastructure for the past four years.”

But all of this remains only speculation and conjecture.

Chris Stangl, former FBI cyber division agent, stated that law enforcement is likely investigating the incident. He noted that CISA and the FBI are studying the situation to prevent similar cases in the future and to understand what motivated those involved.

The NSA does not comment on the situation, neither confirming nor denying that an investigation is underway.

Conclusions

The US Cybersecurity and Infrastructure Security Agency (CISA) assigned the vulnerability the code CVE-2024-3094. The backdoor made its way into test releases of Linux distributions, such as Debian Sid, Fedora 41, and Fedora Rawhide, but was discovered before it could reach widely used stable versions.

Although the consequences of this incident were not catastrophic, it exposed the vulnerability of the entire open-source infrastructure. As it turned out, even a small project with a single tired maintainer can become a point of entry into critical systems around the world.

CISA urges companies using open-source not to rely solely on volunteers, but also to invest in the development and support of such projects. Without this, it will become increasingly difficult to prevent similar threats.

The problem is especially acute given that many tech giants, making billions from using free software, are far from always willing to invest in maintaining it. As a result, vital tools end up in the hands of enthusiasts without resources or support and are prone both to burnout and to outside attacks.

The FFmpeg developer bitterly noted on Twitter:

The xz fiasco showed how dependence on unpaid volunteers can lead to serious problems. Corporations with trillions in turnover expect free and urgent support from volunteers.

After the incident, there were proposals for how exactly to strengthen the open-source infrastructure. One of the authors of the XML specification, Tim Bray, suggested creating “Open Source Quality Institutes” (OSQI) — independent organizations capable of supporting critical projects, paying maintainers, and providing independent audits. However, over the past year, this initiative has not gained significant traction, and no major systemic changes have yet taken place.

“We were incredibly lucky this time,” said Andres Freund in one of his posts on Mastodon. “But relying on luck in the future is a bad strategy.”

Over the past 12 months, the community, researchers, and journalists have not been able to find new facts revealing Jia Tan's identity or motivation. There are plenty of theories: from working on behalf of government agencies to a well-thought-out solo trolling campaign. But all of this remains speculation. The irony is that the main threat wasn't even about who the attacker was, but about how easily he managed to break through all the barriers.

The moral of the XZ Utils story isn’t exposure, but a warning: if nothing changes, something like this will happen again. And next time, we might not be so lucky.

A UFO came by and left a promo code here for our blog readers:

15% off any VDS order (except the Progrev plan) — HABRFIRSTVDS