When Trust Became a Vulnerability: What Recent Studies Say About AI's Impact on Cybersecurity

Let’s go through the main points of the Cyber Risk Report 2025, released by Resilience in collaboration with its analytical division, ROC (Resilience Risk Operations Center).

Cyber Risk Report 2025: AI is used for social engineering and deepfakes

Ransomware has shifted from encryption to data theft. Attacks based solely on data theft, without encryption, have noticeably increased: their share among ransomware incidents rose from 49% in the first half of 2025 to 65% in the second. Analysts believe that businesses have learned to better handle traditional ransomware incidents: many companies now have stronger backup and recovery systems. In response, hackers have quickly adapted and started targeting not only infrastructure but also the data itself. In this brave new world, a data breach can cause significant damage to an organization! Therefore, defenses based solely on recovery after failure are no longer sufficient.

Data theft and exposure are especially painful for the healthcare, retail, and manufacturing sectors. The fact of a leak immediately turns into problems: lawsuits, “letters of happiness” from regulators, and the company being bashed in media headlines — naturally, in a bad light.

Social engineering, amplified by AI, has become unprecedentedly (!) effective. Phishing has become the main problem: its contribution to recorded losses rose from 21% in 2024 to 50% in 2025. The authors say that not every attack can be directly attributed to AI, but they believe that the increase in success is precisely due to AI helping make attacks more convincing.

After a decline in 2024, phishing losses rose again in 2025, exceeding $1.6 million per insurance case. More and more phishing incidents in 2025 led not to minor consequences, but to serious extortion. Naturally, losses grew even more.

The authors also note the increase in the quality of AI-assisted phishing. They refer to research by Harvard and the Harvard Kennedy School, which showed that AI phishing succeeded in 54% of cases compared to 12% for traditional phishing — meaning it was about 4.5 times more effective. Quite an improvement. AI tools allow attackers to craft flawlessly worded messages, convincingly impersonate victims’ executives and colleagues, without the “translation difficulties” that previously made them easy to detect.

To deal with these risks, besides employee training, technical measures are needed: phishing-resistant multi-factor authentication, email authentication, behavioral analytics. But training also helps: according to Resilience, companies with implemented anti-phishing training programs and regular security drills reduced potential risk of damage by more than $100,000.

The new stage of threats — deepfakes. The authors warn that AI-generated audio and video increase the danger of scenarios involving forgery of executives’ faces in video calls and real-time voice synthesis. Because of this, the reliability of traditional verification methods via video or voice calls has decreased. Clients who implemented stricter mechanisms for protecting bank transfers, such as dual approval, reduced the overall risk of damage from deepfakes by an average of $795,000.

Example: interview recording where a deepfake was used. Suspecting that the person applied an AI-based filter, the interviewer asked the applicant to bring their hand to their face. They refused, and Mochadlo ended the interview. The AI used for the deepfake was not very good… A more modern and better model would have handled the task.

Another example — the story of how an IT worker passed four video interviews, background checks, and was hired. Later, the employer, the cybersecurity company KnowBe4, revealed that it was a hacker from North Korea using a stolen American identity and AI-processed photo. As a result, he successfully passed four video interviews using a deepfake, passed reference and background checks, and received a work Mac. Suspicions arose after the device was issued: security teams noticed suspicious activity, malware downloads, and attempts to manipulate session histories.

The very nature of extortion has changed. It has become multi-stage: first, criminals demand money to decrypt locked data, then for not disclosing stolen information, and then they may start pressuring clients or partners to extract additional payments. This cycle can continue indefinitely—as long as you remain on the scammers' hook. The authors note that attackers often still sell the data they promised not to disclose to make more money off your misfortune.

The future, dark and bleak…

What does the coming year hold for us? An intensification of the trends outlined in 2025. The same terrible gears will keep spinning faster.

Extortion will become hybrid and multi-layered. Attackers will pressure not only the company but also its subsidiaries, suppliers, and clients. The sequential application of multiple tactics—encryption, data theft, DDoS, psychological pressure—will become the norm rather than the exception. Therefore, response plans must account for cascading effects across the entire ecosystem, not just the direct damage to a single organization. It is also worth modeling supply chain disruptions separately.

The popularity of unencrypted extortion models will continue to grow. If, in 2025, the share of attacks with data theft without encryption grew from 49% in the first half of the year to 65% in the second, by the end of 2026, such schemes may become dominant.

The largest incidents will increasingly occur due to vendors and providers. The main issue for CISOs (Chief Information Security Officers) is dependence on the external ecosystem. Therefore, having a plan for supplier failures becomes just as important as assessing their security. Insurance policies should also account for the company’s actual dependence on contractors.

Credential compromise will accelerate due to infostealers (malicious software that steals information from victims' devices). They will massively collect valid sessions. Session hijacking and the abuse of OAuth will make standard multi-factor authentication insufficient against social engineering phishing. Moreover, most of the extortion victims, according to the authors, will be visible in the infostealer logs even before the main attack. Hardware tokens and passkeys will become mandatory — SMS and push notifications for multi-factor authentication will no longer be enough, forget about them. Companies need to regularly look for signs of compromise by infostealers and assume that if they are found, the breach has already occurred. Activate the action plan when all the bad stuff has already happened!

Deepfakes will become indistinguishable from reality and will allow scaling social engineering to a new level. The risks of impersonating executives in video calls and attacks with real-time voice synthesis will increase, and it will become easier for attackers to bypass traditional verification methods. So think about what to do with this.

The largest incidents of 2026 will be associated not so much with AI attacks, but with the hasty integration of AI into business. That is, the threat is created not only by malicious actors with AI, but also by companies themselves that rush to implement these tools bypassing data usage rules. The risks for private and sensitive data are especially notable due to the improper use of AI by employees.

“The threats associated with deepfakes and their use in social engineering started being discussed a couple of years ago, but despite significant AI advancements, these predictions have not yet materialized. Perhaps the reason is that the very scheme of deceiving a person using a deepfake depicting their colleague, manager, or friend implies a targeted nature of the attack. This limits the scalability of deepfake-based social engineering and prevents it from becoming widespread. Moreover, many people can be deceived by simpler methods—so why would cybercriminals complicate the attack?”

Vladimir Zuev

Technical Director of the RED Security SOC monitoring and response center

IBM X-Force: Is it easier to become a hacker now?

IBM highlights similar trends in its report:

— AI lowers the barrier to attacks and accelerates processes. Less experienced hacker groups have gained the ability, thanks to AI, to conduct operations that previously required deep technical expertise. According to IBM, this is the real shift in the threat landscape: reducing costs and simplifying entry into offensive operations.

— Current offensive AI capabilities are the result of training models on large code datasets. And this skill will continue to develop. Although neural networks were not originally designed as tools for developing malware or conducting complex attacks, they are gradually moving in this direction. The main risk is that advanced AI labs, private teams, and other players are already working on specialized datasets for offensive purposes—specifically, datasets for training and arming models to find vulnerabilities for attacks and operate against network and web infrastructure. As such datasets emerge, models will be able to conduct security testing faster, on a larger scale, and more sophisticatedly than they can today.

— IBM does not believe that the future of attacks is entirely dependent on AI. Instead, the market is moving towards a model where attacks are built as a mix of human work and AI. Both autonomous security testing using AI and semi-autonomous offensive cyber operations and vulnerability hunting using AI will evolve. In conjunction with specialized datasets for offensive tasks, this will enhance cyber operations and ransomware attacks with encryption, data theft, and other pressure scenarios.

— Attackers bypass defenses by breaking malicious tasks into small fragments that individually appear harmless; they format requests as security testing scripts; distribute interactions across multiple sessions — all of this so that the content of requests doesn’t attract attention. From this, IBM draws an important conclusion: prompt checking alone is not enough, and the vulnerability lies deeper — in the architecture of the systems themselves.

By the way, in 2025, a data-stealing malware revealed over 300,000 ChatGPT credentials, demonstrating that AI platforms have reached the same level of leak risk as other corporate SaaS solutions.

— The gap between open and closed systems of offensive AI is widening. IBM believes this division is becoming one of the defining features of the new threat market. While large labs are tightening restrictions on releasing models and publishing new advanced features less frequently, more serious players in the dark web may shift to privately trained, forked, or entirely closed systems, which are beyond regular oversight. As a result, a multi-layered ecosystem is forming, where groups of attackers gain access to different levels of AI capabilities. Open models provide broad accessibility but often lag in power, depth of further training, and the ability to bypass defenses. Meanwhile, closed systems can significantly accelerate the development of exploits, automated target reconnaissance, and orchestration of multi-stage attacks.

— The gap is growing between defenders and attackers, and in favor of the latter. Defenders have a more limited set of tools at their disposal, whereas their opponents can freely enhance their closed models. That is, the problem is not only that AI is becoming more powerful, but also that the conditions of its use for attackers and defenders are fundamentally different.

— Models are increasingly taking on routine but critically important tasks: domain rotation, adapting malicious payloads for specific targets, organizing infrastructure deployment, masking network traffic, maintaining infrastructure resilience. This reduces the operational burden on attackers and allows even less skilled groups to deploy resilient and adaptive infrastructure that previously required significant experience and knowledge. For stronger players, the next step is dynamic reconfiguration of infrastructure in response to detection and on-demand generation of new variants of malicious content.

— The evolution from text-based models to multimodal systems is changing the nature of attacks. Such models can interpret code, natural language, screenshots, network diagrams, event logs, audio, and video. For attackers, this means the ability to automate reconnaissance and environment mapping at a level that previously relied on humans. By correlating different types of data, the system can more quickly identify configuration errors, analyze traffic, find weak points, and in the future even generate adapted exploits, test attack paths, and collect physical, digital, and social signals into a unified picture.

What will help security professionals?

IBM lists the areas it considers particularly important for the future of AI in offensive cyber operations. Among them: autonomous security testing, semi-autonomous cyber operations, searching for and researching vulnerabilities using AI, accelerating exploit development, creating offensive tools with AI, searching for and analyzing targets with AI, OODA loops — observe, orient, decide, and act — in multi-stage attacks, as well as agent-based AI systems capable of performing related tasks independently. IBM does not claim that all of this is already common practice, but it believes that offensive AI is moving in this direction now.

“Does AI lower the barrier to entry for cybercriminal activity? Both yes and no. To create malware with the desired functionality using AI, you still need to have a mental image of the outcome. Without it, it is impossible to assess how well AI has performed the task, refine, and detail the prompt. And without understanding how the software works, it will be impossible to use it as intended during a cyberattack. So AI will not help people with no knowledge at all to ‘become a hacker.’ But what AI does handle well is automating routine and templated tasks, which, of course, can help attackers act faster and more efficiently all else being equal.

Currently, AI is more useful to attackers than to defenders. Attackers have clear tasks for it where they can evaluate the result: writing malicious scripts, exploits, and so on. In theory, a cyberattack monitoring center could ‘feed’ AI security events and ask it to make decisions about them. But even if the decision is correct, we cannot be sure that the correct logic led to it, and therefore we cannot entrust this work entirely to artificial intelligence. So in defensive security, AI is still mainly a tool for automating basic tasks.”

Vladimir Zuev

Technical Director of the RED Security SOC Cyberattack Monitoring and Response Center

ReliaQuest: new tools — many worries?

Thanks to AI, hackers use legitimate tools as weapons. Therefore, defense can no longer rely on finding a single malicious file or indicator of compromise. It is important to notice in time when a trusted account starts behaving strangely.

The researchers recommend making every device and every access path visible and manageable for the SOC (Security Operations Center), continuously managing the external attack surface (especially during changes and migrations), and strengthening identity controls where trust is easiest to exploit. For example, standardizing more robust verification for support requests, access configurations, reducing privileges, and making privileged access phishing-resistant, so that stolen credentials and social engineering do not give the hacker a high level of control over the environment.

ReliaQuest writes that the average time for attackers to move beyond the initially compromised system in 2025 decreased by 29% — from 48 to 34 minutes. At the same time, average metrics no longer reflect the most extreme scenarios: in some cases, attackers moved laterally (when an intruder, already inside one system, begins to move further across internal infrastructure) in just four minutes, and exfiltrated data within six minutes of initial access. At the same time, the authors note, slow operations have not disappeared — especially from state-linked groups that maintain a presence in infrastructure for months, expand access, and mask themselves as normal activity. In other words, defenders have to operate in two modes simultaneously: countering attacks unfolding at tremendous speed and resisting the prolonged hidden presence of an intruder in the system.

More problems?

AI tools for pentesting will turn into attackers' weapons. ReliaQuest writes that in 2026, cracked versions of AI penetration testing tools will appear on criminal forums, allowing vulnerabilities to be found and exploited faster. The article notes that such tools had already gained noticeable popularity in 2025, automating vulnerability discovery, exploitation, and attack simulation. As an example, the authors mention Xbow, which, according to them, reached the top of the HackerOne leaderboard and discovered CVE-2025-27888. At the same time, after appearing on the black market as cracked versions, they may follow the same path that Cobalt Strike did: from a specialist tool to a widely used attack instrument. ReliaQuest separately notes that Cobalt Strike was involved in 50% of client incidents and expects a similar trajectory for AI pentest solutions.

In 2025, forums were already selling services for creating deepfakes and Blackhat ChatGPT, i.e., a jailbroken model promoted on forums like DarkNet Army. From this, the authors conclude that the commercialization of AI solutions has already begun, and the appearance of cracked AI tools may radically shorten attack timelines — from initial access to deploying ransomware in just minutes — while simultaneously lowering the entry barrier for less skilled attackers.

In 2026 specialized cyber groups will emerge that will deliberately target not companies directly, but the people responsible for maintaining open-source packages and libraries. The authors explain this forecast by the structural weakness of the open-source ecosystem: out of 11.8 million open-source projects, about seven million are maintained by a single person, and almost half of the 13,000 most downloaded npm packages rely on a single maintainer. This creates a convenient attack point: compromising just one maintainer is enough to inject malicious code into a trusted package and achieve large-scale propagation down the chain.

If an update from a compromised maintainer is automatically pulled into the CI/CD pipeline, malicious code may steal cloud API keys, service tokens, and other sensitive credentials from build servers. A single infected library, the company writes, can affect hundreds of applications across an enterprise, and can then be used to steal data, extort, spread spyware, or deploy cryptocurrency miners. For users, this means service outages, feature shutdowns, and loss of trust once it is discovered that their data was compromised long before the incident was localized.

One example is the events of September 2025, when the Qix maintainer, whose packages had two billion downloads a week, fell victim to phishing despite having MFA enabled. Afterward, the attackers published malware in his repositories to steal cryptocurrency. In the same month, as ReliaQuest writes, accompanying PyPI maintainers received phishing emails from fake pypi[.]org domains, and the Shai-Hulud worm infected over 500 npm packages via fake collaboration invitations. From this, the authors conclude that future groups will monetize not only account theft, but the very compromise of the maintainer as a mass distribution channel for malicious code.

"Although dependency compromise as part of supply-chain attacks has been known for a long time, the 'human' vector for its implementation remains an exotic one. Nevertheless, the hypothesis that it could spread as part of targeted attacks seems quite plausible. For example, there was a case where an attacker infiltrated the open-source software development community for several years, only to eventually publish a malicious library in the repository. Targeted phishing against maintainers is from the same category: the person is used as the weak link in the IT systems chain."

Vladimir Zuev

Chief Technology Officer of the RED Security SOC Cyberattack Monitoring and Response Center

What should be done?

Artificial intelligence has made good old cybercrime even more effective. It has lowered the entry barrier for attackers, sped up preparation and scaling of attacks, and helped automate phishing, social engineering, reconnaissance, and infrastructure work. As a result, attackers act faster and more confidently, attacks have become cheaper, and defenders have even less time to react.

The most dangerous change is that trusting video or voice is no longer an option. Not only a password, endpoint, or server can be faked, but also the person themselves or their voice on the other side of the screen. Internal procedures, support requests, calls from a manager, emails from colleagues, requests to urgently confirm a transfer, collaboration invitations—all of this can be fraudulent… And AI has made creating such forgeries even easier and more believable. Before you know it, a maintainer has gained access to a package and created a trusted session that no one found suspicious.

Attacks increasingly start with a valid account, look like normal activity, live inside a trusted environment, and cause damage not at the moment of intrusion but afterward—through leaks, blackmail, legal claims, regulatory pressure, and reputational loss. The main risk now lies in the consequences, which can continue for a very, very long time.

Cyber threats are no longer just a technical problem for IT security teams. Now it’s a matter of corporate governance. Because it’s no longer just about patches, antivirus, and SOC, but about the quality of internal procedures, AI usage rules, dependency on contractors, maturity of access control, legal readiness, and the company’s ability to make quick decisions when an attack unfolds in minutes.