A joint study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute warns that very small levels of data poisoning can compromise large language models (LLMs). The team shows that inserting around 250 malicious documents during training is enough to implant a reliable backdoor—regardless of the model’s size.
Why this matters
Conventional wisdom says big datasets dilute bad samples. Anthropic’s results challenge that, showing attack success depends on the absolute number of poisoned samples, not on their percentage of the total corpus. In other words, scale doesn’t save you.
What the researchers did
- Trained models at 600M, 2B, 7B, and 13B parameters on clean, Chinchilla-optimal data.
- Injected 100, 250, or 500 poisoned documents.
- Tested a backdoor/DoS-style trigger causing gibberish output when a hidden token appears, while keeping normal behavior otherwise.
Findings:
- ~100 poisoned docs: generally insufficient.
- ~250+ poisoned docs: consistently successful across all sizes.
- Varying total training volume and random seeds did not meaningfully change outcomes.
Important caveat
The experiment targets a relatively narrow denial-of-service backdoor (trigger → nonsense). It doesn’t demonstrate more dangerous outcomes (e.g., data exfiltration, safety bypass, or malicious code). Whether the “constant poison count” result holds for higher-stakes backdoors in frontier systems remains an open question.
Takeaways for builders & safety teams
- Harden the data pipeline:
- Use multi-stage data provenance, supplier allowlists, and content hashing.
- Apply active curation and statistical anomaly detection for pretraining corpora.
- Use multi-stage data provenance, supplier allowlists, and content hashing.
- Adopt poison-aware training:
- Consider differentially private or robust training methods that reduce sensitivity to outliers.
- Incorporate trigger sweeping and canary tests during and after training.
- Consider differentially private or robust training methods that reduce sensitivity to outliers.
- Continuously evaluate models:
- Maintain red-team suites for backdoor probes.
- Monitor drift and unexpected trigger correlations in deployment logs.
- Maintain red-team suites for backdoor probes.
- Segment risk:
- Separate high-trust from low-trust data sources.
- Gate high-impact features behind policy enforcement and runtime guardrails.
- Separate high-trust from low-trust data sources.
Context & related developments
- Anthropic is expanding globally and plans an India office in Bengaluru (2026).
- The company recently touted Claude Sonnet 4.5 as its most capable coding model to date.
- Microsoft has begun integrating Anthropic models into parts of Copilot.
Quick FAQ
Does a larger model resist poisoning better?
Not in this study. Model size had negligible effect once the attacker reached ~250 poisoned samples.
Is 250 a hard threshold?
It’s an empirical result for the tested backdoor and training setup. Different tasks or defenses may shift the number—but the trend is what matters.
Can this enable data theft or safety bypass?
This paper did not test those outcomes. The risk is plausible; further research is needed.
