Data poisoning

What is data poisoning?

Data poisoning is a type of attack where someone deliberately corrupts or manipulates the data used to train an AI model. By injecting false, misleading, or malicious information into the training set, the attacker can make the model behave in unintended ways—producing wrong outputs, embedding hidden backdoors, or systematically skewing its decisions. Unlike hacking a model after it’s built, data poisoning targets the model’s ‘education,’ meaning the damage is baked in from the start [1].

A useful analogy

Imagine you’re studying for an exam using a textbook, but someone has secretly replaced several key pages with convincing but false information. You’d study diligently, feel confident, and still give wrong answers—not because you’re a bad student, but because you learned the wrong lessons. Data poisoning works the same way: the AI learns from corrupted material and then confidently produces incorrect or harmful outputs [2].

How data poisoning works

Modern AI models learn by processing enormous datasets—billions of text passages, images, or other data points. During training, the model identifies patterns in this data and uses them to make predictions. Data poisoning exploits this process by inserting carefully crafted malicious samples into the training data [2].

The OWASP Top 10 for LLM Applications (2025) classifies data poisoning as LLM04:2025, defining it as an integrity attack that can target pre-training data, fine-tuning data, or embedding data [1]. This matters because modern large language models go through multiple training stages—pre-training on general text, fine-tuning for specific tasks, and alignment to follow instructions—and each stage presents an opportunity for an attacker to insert poisoned data.

Three types of data poisoning attacks

Security researchers categorize data poisoning attacks by their method and goal:

1. Untargeted poisoning (label flipping)
The attacker modifies labels in the training data—marking spam emails as safe, or toxic content as acceptable. The model learns these wrong associations and becomes less accurate overall. The goal is to degrade general performance rather than trigger a specific behavior [2].

2. Targeted poisoning (backdoor attacks)
The attacker inserts a specific trigger pattern into the training data along with a desired output. The model behaves normally on regular inputs but responds to the trigger with a predetermined action—for example, classifying any image containing a particular symbol as ‘safe’ regardless of its actual content. These backdoors can persist even through safety training [3].

3. Clean-label poisoning
The most sophisticated approach. The attacker adds samples that look completely legitimate to human reviewers—correctly labeled, natural-looking data—but that subtly shift the model’s decision boundaries. Because the poisoned samples appear normal, they’re much harder to detect through standard data quality checks [2].

The Microsoft Tay example

In March 2016, Microsoft launched Tay, an AI chatbot designed to learn from conversations on Twitter. Within 16 hours, coordinated users fed Tay massive amounts of racist, misogynistic, and antisemitic content. The chatbot absorbed this poisoned input and began reproducing it, posting over 95,000 tweets before Microsoft shut it down. Tay remains a textbook example of what happens when a model learns from unvetted, adversarial data without adequate filtering or oversight [4].

Why LLMs are especially vulnerable

Large language models face compounding data poisoning risks for several reasons:

Scale creates opportunity: LLMs train on billions of tokens from diverse sources—web scrapes, books, code repositories, user-generated content. The sheer volume makes comprehensive vetting nearly impossible [5].

Multi-stage training multiplies attack surfaces: Pre-training, fine-tuning, and alignment each use different data sources. An attacker can target any stage, and poisoning at later stages (like fine-tuning) can be particularly effective because it modifies a model that has already learned general patterns [5].

Supply chain trust: Many organizations use pre-trained models or third-party datasets without full visibility into how that data was collected or processed. The NIST AI 600-1 framework identifies ‘Value Chain and Component Integration’ as a key risk category—poisoned data can enter through any link in the supply chain [5].

Persistence through safety training: Research from Anthropic published in January 2024 (‘Sleeper Agents’, arXiv:2401.05566) demonstrated that deceptive behaviors introduced during training can survive standard safety techniques like RLHF and adversarial training. The backdoors don’t wash out—they hide [3].

Mitigation strategies

Defending against data poisoning requires a layered approach spanning the entire model lifecycle:

Data provenance tracking: Maintain a clear record of where every piece of training data came from, how it was processed, and who handled it. Tools like OWASP CycloneDX and ML-BOM (Machine Learning Bill of Materials) help organizations track data lineage [1].

Vendor vetting: Rigorously evaluate third-party data providers and pre-trained model sources. Blind trust in upstream suppliers is a primary vector for supply chain poisoning [1].

Data version control: Use version control systems designed for datasets (like DVC) to track changes over time, enabling rollback if poisoning is detected [1].

Anomaly detection: Apply statistical methods to identify unusual patterns, outliers, or distributional shifts in training data that may indicate tampering [1].

Sandboxing and isolation: Test models in isolated environments before deployment to detect unexpected behaviors that might indicate hidden backdoors [1].

Human oversight: Maintain human review at critical points in the training pipeline—especially for fine-tuning data and alignment datasets [1].

Despite these defenses, researchers have shown that modern attacks using generative models and steganography can create poisoned samples that evade detection while preserving the model’s performance on normal tasks, making the arms race between attackers and defenders increasingly sophisticated [2].

Common questions

Is data poisoning the same as a cyberattack on an AI system?
Not exactly. Traditional cyberattacks target a system’s infrastructure—stealing data or disrupting service. Data poisoning targets the model’s knowledge itself, corrupting what it learned during training. The damage isn’t a breach you can patch; it’s baked into the model’s understanding of the world.

Can a poisoned model be fixed after deployment?
It’s difficult. Research shows that backdoors introduced through poisoning can survive standard safety training techniques. Prevention through data provenance, vetting, and anomaly detection is far more effective than trying to ‘clean’ a model after the fact.

How is data poisoning different from prompt injection?
Data poisoning attacks the training phase—corrupting the data the model learns from before it’s deployed. Prompt injection attacks the inference phase—tricking the model into ignoring its instructions during a live conversation. One corrupts the education; the other exploits the conversation.

Who is most at risk from data poisoning?
Organizations using third-party datasets or pre-trained models without full supply chain visibility face the highest risk. Healthcare, finance, and other high-stakes domains are particularly vulnerable because poisoned outputs can have real-world consequences—misdiagnoses, bad financial decisions, or compromised safety systems.

Can users accidentally poison an AI model?
In theory, yes—if a model is designed to learn from user interactions (like Microsoft Tay was). Accidental poisoning through low-quality user data is a real concern for systems with online learning capabilities. This is why most production LLMs don’t learn from individual conversations in real-time.