250 documents: hackers found a vulnerability that kills any AI

Unfortunately, this belief is misguided, as convincingly demonstrated by a joint study by Anthropic, the UK Centre for AI Safety, and the Alan Turing Institute. Their conclusion is that a small, fixed number of malicious samples can implant a backdoor (hidden vulnerability) in large language models (LLMs), regardless of their size.

The Anthropic article “A Small Number of Samples Can Poison LLMs of Any Size” explains this phenomenon with an unusual candor and clarity for corporate communications: embedding about 250 carefully crafted documents into the training corpus induces the model to develop dormant behavior that can be activated by a specific trigger. The experiment they chose is not the most spectacular, but paradoxically, it makes everything happening more frightening: this is a denial-of-service type backdoor that, upon detecting a keyword, causes the model to generate nonsense as if it has broken from within. This is not an attack designed to destroy the model, steal money, or influence elections: for the most part, it is a demonstration of control in the spirit of “I can make your model do this at my whim.”

An important detail here is not the nonsense, but the metrics. Until now, a threat model based on percentages was assumed: to poison a large model, an attacker would need to control a significant percentage of the training, which becomes practically unfeasible when it comes to hundreds of billions of tokens. This research turns everything upside down: models were trained on sizes ranging from 600 million to 13 billion parameters using the optimal amount of data according to the Chinchilla rule (more for larger models), and it was observed that the attack does not scale with size: the same 250 documents equally compromised all models. Essentially, the poison does not dissolve as expected: it learns to survive.

This creates systemic fragility. LLMs are mainly trained on publicly available texts from the internet, and the Turing Institute highlights the consequences: anyone can post content with the intent to enter these datasets, and if the actual threshold is around 250 documents, the barrier to entry is not particularly high. You don’t need to control some part of the internet: it’s enough to just find cracks in the data supply chain. And here’s the key point: in 2026, concerns about LLMs are no longer just about hallucinations or bias, but also integrity. Where does what the model “knows” come from? Who interfered in the process? What are the incentives for interference? Could this become a way to seize control over narratives?

In fact, a backdoor that generates nonsense is almost a toy compared to what has already been discovered in the scientific literature: backdoors for lowering security, provoking malicious behavior, or bypassing alignment systems. The Anthropic paper itself refers to research on backdoors that function as a sort of universal command to obtain malicious responses when a certain trigger occurs. Anyone who thinks, “This is an alignment and RLHF issue, not pretraining,” is mistaken: the overarching lesson is absolutely the same. In a system that learns to correlate, intentionally embedded malicious correlations may prove more resilient than millions of harmless ones.

The most troubling parallels arise when moving to areas where the cost of a mistake is not a meme but potentially human lives. In 2024, a study in Nature Medicine modeled a poisoning attack on The Pile, one of the flagship datasets of the ecosystem, by introducing medical misinformation: by replacing just 0.001% of tokens with plausible lies, the resulting model became more prone to spreading medical errors, and the worst part was that it still, seemingly, “worked just as well” in benchmarks.

This debunks another widespread fantasy: that an "objective" assessment will save us. If the attack is targeted, if it is designed to activate on a trigger or to influence a specific subset of issues, the model can pass all standard tests and still remain compromised. In the security field, this has been talked about for decades: systems that "seem" correct under normal conditions fail when someone knows how to press the right button. What's new is that now this system writes, advises, programs, summarizes, negotiates, translates, and more frequently acts as a cognitive intermediary in thousands of human decisions.

Therefore, it is not surprising that recent risk assessment systems clearly include this threat. OWASP, in its list of risks for applications with LLM, defines Data and Model Poisoning as an integrity violation vector with clear consequences: backdoors, intentionally introduced biases, behavior degradation, and attacks that are difficult to detect because the model can behave "normally" until the trigger is pulled. The NIST taxonomy on adversarial machine learning includes categories such as backdoor poisoning and supply chain attacks to encourage the industry to think in terms of lifecycles, not just the current model. When the most reliable standardization bodies start talking this way, it usually means the problem is no longer theoretical and has become a plausible risk.

The question is no longer whether this can harm the reputation of LLMs, but what happens to their reliability as they become infrastructure for an increasing number of things. Because if we accept that a model can be trained on opaque data with unclear provenance and imperfect control, and that a patient attacker can implant several dozen or hundreds of fragments designed to survive this process, then the model stops being merely probabilistic and becomes potentially falsified. And here the social contract changes: a system that "sometimes makes mistakes" is manageable, but a system that can be manipulated without any visible signs becomes politically toxic and very difficult to regulate.

Is there a way out? There is, but it's not convenient, it's not cheap, and it certainly doesn't fit with the "move fast and break things" culture that got us to this point. What the research asks us to acknowledge is this: LLMs need what we have taken for granted in software for years - a supply chain with mechanisms for control, auditing, traceability, and verification. It's not enough to simply filter "bad content" or remove duplicates. We're talking about verifiable provenance, reproducible curation processes, anomaly monitoring during training, tests designed to detect conditional behavior, continuous red teaming (penetration testing), and the acknowledgment that certain critical use cases will require much more closed, specialized, and controlled models and datasets. And that costs money.

Let's not kid ourselves: this is not some bug that can be fixed with a patch. This is a symptom of a deeper reality: we're building machines of statistical generalization on an informational substrate - a web space that increasingly resembles a battlefield, strewn with corpses and debris. If training means absorbing the internet, then the security of your model depends on the security of the internet, and that is very dangerous, because we all know what's out there. The Anthropic article is a warning: in the upcoming world, the answer to the question of whether I can trust what a particular model says will not lie in the realm of accuracy metrics, but in a much more uncomfortable realm: "Can I trust how it was created, the data it was trained on, and those who worked with that data?"