Data & Model Poisoning Cheat Sheet: Backdoors & Defenses
A model is a mirror of its data. Data & model poisoning (OWASP LLM05) is tampering with what a model learns — during pretraining, fine-tuning, or through the embeddings it retrieves — to plant a bias, a backdoor, or a quiet failure that only the attacker knows how to trigger. Unlike a live attack, this one is baked in before the model ever answers a prompt.
Test your own data and models only. Studying poisoning to defend a pipeline you own or are authorized to assess is security work; deliberately poisoning a shared dataset or someone's model is an attack on everyone downstream.
Why it's so hard to see
A poisoned model passes normal tests. A backdoored one is perfectly well-behaved until it meets its trigger; a biased one just looks slightly off in ways easy to rationalise. And it only takes a little: research has repeatedly shown that a tiny fraction of poisoned examples can implant a reliable backdoor. That's why the defense is mostly about the data going in, not testing the model coming out.
The threats
Every entry point where bad data can reach a model.
| Threat | How it works |
|---|---|
| Training-data poisoning | Malicious content is seeded into the web-scale data a model trains on, so the model quietly learns the attacker's slant or a hidden behaviour. |
| Backdoor / trojan trigger | A specific phrase or pattern is trained in as a secret switch — the model behaves normally until it sees the trigger, then does the attacker's bidding. |
| Fine-tuning poisoning | A handful of crafted examples in a fine-tuning set is enough to bend a model — far less data is needed here than in pretraining. |
| Embedding / RAG poisoning | Bad documents are pushed into the vector store so they surface as “trusted” context. Overlaps with LLM09. |
| Feedback-loop poisoning | Thumbs-up/down or user-submitted data feeds back into training; an attacker games it to drift the model over time. |
Keep the data clean
Govern what goes in, and watch what comes out.
- Know your data's provenance. Track where every training and fine-tuning source came from; prefer curated, verifiable data over raw web scrapes for anything sensitive.
- Validate and clean datasets. Deduplicate, filter, and run anomaly detection before training — outliers and odd clusters are where poison hides.
- Control the fine-tuning set. It's small and high-leverage, so review it closely; a few bad rows here do more damage than millions in pretraining.
- Guard the feedback loop. Don't pipe raw user submissions or thumbs-up/down straight back into training — rate-limit, filter, and human-review first.
- Red-team for backdoors. Actively probe a model for trigger phrases and unexpected behaviour before you ship it, not just standard accuracy tests.
- Monitor after deploy. Watch live behaviour for drift and sudden anomalies — a backdoor may only reveal itself in production.
Go deeper
- OWASP LLM Top 10 — LLM05 in the full risk map.
- AI supply chain — poisoned data often arrives via a third-party dataset or model.
- RAG security — embedding poisoning, the runtime cousin of this.
- LLM security lab — its data-poisoning challenge (“The Sleeper Phrase”) is a backdoor trigger, hands-on.
Aligned to OWASP LLM05:2026 Data & Model Poisoning; a plain-language distillation of current practice. Something changed? Tell me.