Privacy Attacks

Source: 13-privacy-attacks.md
State: Editorial review
Edition: 2026-draft
Estimated reading time: 17 min

Draft chapter under editorial review

This material is available for early reading, but it has not reached the reviewed 1.0 release. Technical references, examples, and wording may change.

Machine-learning models memorize. Not on purpose, but as a byproduct of fitting parameters against data — some examples are seen enough times, or are distinctive enough, that the model retains reconstructible traces of them. Privacy attacks exploit that residue to answer three questions: was a specific example in the training set (membership inference), what did the training data look like (model inversion / extraction), and can I copy the model itself (model extraction).

Each of these is a legitimate compliance concern (GDPR right-to-erasure requires that data actually be forgotten; if the model still memorizes it, erasure was incomplete) and a legitimate offensive target (customer data extraction, competitive-model theft, dataset reconstruction). 2025-2026 research sharpened every one of these: membership inference against fine-tuned LLMs crossed 90% accuracy, a real-world industrial-scale model-distillation dispute became public between major labs, a $200 attack pulled tens of thousands of verbatim training examples out of a production chatbot, and a rigorous audit of machine-unlearning techniques found that almost none of them actually pass a real forgetting test. This chapter updates the techniques and gives testers concrete cost, tooling, and threshold numbers to cite in findings.

13.1 Membership inference

The question: given a specific input x, was x in the training set of the target model?

The mechanism: models are more confident on inputs they were trained on than on new inputs. The confidence gap is a signal. Feed x to the target model; observe the output probability, loss, or confidence; compare against a threshold; classify x as IN or OUT.

The shadow model attack (Shokri et al., 2017). The reference technique. Steps:

Train shadow models on datasets drawn from the same distribution as the target's training data. Enough shadow models that both IN examples and OUT examples are richly represented — typically 10-100 shadows.
For each shadow model, record its outputs (probability vectors) on both its own training examples (IN) and on held-out examples (OUT).
Train an attack classifier — a small model that takes an output probability vector and predicts IN or OUT. Trained on the shadow-model output-vector dataset.
Query the target model on the input of interest. Feed the output to the attack classifier.
Attack classifier's prediction is the membership inference result.

Effectiveness scales with:

Overfitting of the target — highly overfit models are easier to attack
Distribution overlap between shadow training data and target training data
Model architecture — deeper models with more capacity memorize more
Class imbalance — rare classes are more distinctive per example, easier to identify

Modern reference implementations: ML Privacy Meter, TensorFlow Privacy inference tests.

Simpler thresholding attacks. Even without shadow models, a threshold on the loss or on the confidence of the top class discriminates IN from OUT on overfit targets. Publish a threshold that maximizes attack accuracy on a held-out set; apply. Fewer resources than shadow-model attacks, lower success rate on well-generalizing models.

Query cost considerations. Every membership inference query burns budget. For paid API targets, this matters practically. For internal targets, it may trip rate limits.

13.2 Membership inference on LLMs — 2025-2026 techniques

The LLM-specific variant: given a specific sentence, paragraph, or document, was it in the pretraining or fine-tuning corpus? This area moved fast enough over 2025-2026 that older loss-based and zlib-comparison baselines are now the weakest tools in the kit, not the strongest.

Min-K% Prob (baseline). The hypothesis: a non-member example is likely to contain some outlier tokens with low probability under the model, while a member example is less likely to contain such outliers. Score a sentence by averaging the log-likelihood of only its k% lowest-probability tokens rather than all tokens; this is far more robust to natural variation in unremarkable tokens than whole-sequence loss. The WikiMIA benchmark (text known to be created before or after a model's training cutoff) provides ground truth for evaluating this and successor methods.

Min-K%++ (ICLR 2025). A theoretically motivated improvement: training samples tend to sit at local maxima of the model's modeled probability distribution, and Min-K%++ scores tokens against this expectation rather than raw probability. It is the current state of the art among reference-free MIA methods (no need for a shadow/reference model). Reference implementation: github.com/zjysteven/mink-plus-plus.

Neighborhood attacks. Compare the target model's score on the candidate sentence against its scores on synthetically generated neighbor texts — small perturbations of the candidate — and use the local curvature of the loss landscape as the membership signal. This removes the need for access to the training-data distribution that classic reference-based attacks require, and neighborhood attacks are competitive with reference-based attacks that have perfect knowledge of the training distribution — a finding that reshaped the standard threat model for LLM MIA, since testers no longer need such knowledge to get strong results. A more query-efficient variant ("Noisy Neighbors") reduces the cost of generating synthetic neighbors.

Self-Prompt Calibration / SPV-MIA. Targets fine-tuned LLMs specifically, not just pretraining. Instead of requiring external reference data, the attacker uses self-prompting: querying the target LLM itself to generate a reference dataset with a distribution similar to the "forgotten" or fine-tuning set, with no external domain knowledge required. The attack runs two phases — Probabilistic Difficulty Calibration and Probabilistic Variation Assessment — comparing the probability of the original text against paraphrases of it to test whether the original sits at a local probability maximum. This is the first technique to push MIA accuracy against fine-tuned LLMs above 90%, which matters directly for testers assessing whether a vendor's fine-tuning pipeline leaks customer data back through the model.

PETAL (Per-Token Semantic Similarity). A label-only attack — it approximates output probabilities using per-token semantic similarity to estimate perplexity, without requiring access to logits. Useful against production APIs that return only text, not token probabilities, which is the common case for hosted chat products.

Practical implication for testers. Given Min-K%++, neighborhood attacks, and SPV-MIA, "the target only exposes text, not logits" or "we don't have reference training data" are no longer valid reasons to skip membership inference testing on an LLM. Choose the attack based on what's exposed: PETAL for label-only APIs, Min-K%++ for reference-free scoring against a base model, SPV-MIA for fine-tuned deployments.

13.3 Model inversion

The question: given trained model access, can you reconstruct examples from the training set?

Class-representative inversion (Fredrikson et al., 2015). For classification models, invert gradients to find inputs the model classifies with high confidence. The reconstructed input is not any specific training example but a class-typical representative. For a face-recognition model trained to classify identities, inverting one identity's confidence produces a face-like image resembling the person the class represents.

Gradient inversion in federated learning. In federated setups where gradients (not weights) are shared, individual gradient updates can be inverted to reconstruct the specific batch of training examples that produced them. Deep Leakage from Gradients (Zhu et al., 2019) and follow-ups.

Text and embedding inversion from LLMs. Given a language model and a target text-like output pattern, invert to find the input prompt or the training excerpt that would produce it. Embedding inversion specifically has become a mature, high-fidelity attack class — Chapter 08 covers the vector-store-facing angle; the reference lineage is Vec2Text (iterative black-box reconstruction, BLEU up to ~97.3 on 32-token sequences), ALGEN (few-shot inversion needing as few as ~1,000 aligned samples for ROUGE-L 45-50), and Zero2Text (2026, training-free "recursive online alignment" that beats both on cross-domain reconstruction against real API embeddings, including OpenAI's text-embedding-3-large). Standard defenses — Gaussian/Laplacian noise, local differential privacy — do not reliably stop these; Zero2Text specifically remains effective under strong noise.

Membership + inversion combined. Confirm membership first (Section 13.1/13.2), then use inversion techniques to reconstruct the specific example. Reduces false-positive reconstructions.

13.4 Model extraction (model theft)

The question: given only query access to a target model, can you build a functional copy, or extract structural information about it?

Direct copying. Query the target on many inputs; use the outputs as labels; train a local model. If the label channel is rich (probability vector across all classes), the local model learns closely; if it's poor (only top class), coarser. Reference: Stealing Machine Learning Models via Prediction APIs (Tramer et al., 2016).

Distillation-based extraction. Uses KL-divergence loss between target probabilities and local model probabilities. Faster convergence than hard-label training.

Extracting real structure from production LLMs at low cost. Stealing Part of a Production Language Model (Carlini et al.) demonstrated the first extraction attack that pulls precise, non-trivial information out of production black-box LLMs like ChatGPT and PaLM-2: it recovers the model's embedding projection layer (up to symmetries) using only typical API access, via a logit-projection technique. The concrete cost figures matter for scoping an engagement: for under $20, the attack recovered the complete embedding projection matrix of OpenAI's Ada and Babbage models, confirming for the first time that their hidden dimensions are 1024 and 2048 respectively, and recovered the exact hidden dimension of gpt-3.5-turbo. The authors estimate that extracting the full projection matrix of gpt-3.5-turbo would cost under $2,000 in queries — a figure worth quoting directly when a client asks "how expensive would this actually be for a real attacker."

Distillation as model extraction at industrial scale — the Anthropic vs. DeepSeek/Moonshot/MiniMax case. Model extraction via distillation stopped being a hypothetical in 2025-2026. In January 2025, OpenAI and Microsoft began investigating whether DeepSeek had used the OpenAI API at scale to generate training data for its R1 model, in violation of terms of service barring the use of outputs to train competing models; in February 2026 OpenAI told the U.S. Congress that DeepSeek employees allegedly used obfuscated third-party routers to evade API access restrictions and automate output extraction. On February 24, 2026, Anthropic formally accused DeepSeek, Moonshot AI, and MiniMax of running coordinated distillation-attack operations against Claude using commercial proxy services to bypass Anthropic's geographic and commercial restrictions on China. According to Anthropic, these firms jointly operated roughly 24,000 fraudulently created accounts, generating over 16 million interactions with Claude in total (MiniMax alone accounted for more than 13 million), with the harvested responses used directly for training or reinforcement learning of their own models.

For testers assessing whether a client's own model is being distilled by a third party, the detection indicators from this case generalize directly:

Account-pattern anomalies. Large numbers of accounts created in short windows, disproportionately from a narrow set of registration sources or proxy IP ranges.
Interaction-volume anomalies. Per-account query volumes far exceeding normal usage patterns, particularly sustained high-volume querying with systematic prompt diversity (consistent with harvesting outputs for training rather than product use).
Proxy/routing indicators. Traffic patterns consistent with obfuscated or commercial-proxy routing designed to evade geographic or account-tier restrictions.

Watermarking as a countermeasure — and its evasion. In response to distillation-based extraction, the industry has deployed watermarking at both the text and model-weight level. Text watermarking (a logit bias toward "green-list" tokens) survives unauthorized distillation and functions as a covert signature for tracing model theft; LIDet detects unauthorized distillation with over 90% accuracy by analyzing the green-token ratio in suspect models. On the offensive side, De-mark (ICML 2025) effectively strips n-gram watermarks via a "random selection probing" strategy that reveals the underlying red-list/green-list scheme, enabling an attacker to both remove the watermark and steal-and-reapply the scheme onto a different model. Black-box watermark detection techniques have also been demonstrated successfully against GPT-4, Claude 3, and Gemini 1.0 Pro in restricted black-box settings. Testers evaluating a client's watermarking claims should attempt a De-mark-style probing pass rather than accepting "we watermark our outputs" as a closed finding.

Extraction under rate limits. For paid-API targets, extraction has a dollar cost, but as the Carlini result and the Anthropic case both show, that cost is far lower than intuition suggests once the extraction goal is narrowed (a specific structural parameter, or output harvesting for distillation rather than full-fidelity weight recovery).

Extraction for downstream attacks. Even a partial extraction — a surrogate model that matches the target in behavior but not fidelity — is enough to craft adversarial examples (Chapter 10) that transfer. Extraction to enable transfer attacks is often the offensive goal, not extraction for reuse.

13.5 Training data extraction — the divergence attack

Beyond membership inference and structural extraction, a distinct and highly effective attack class directly extracts verbatim training examples from production chat models at low cost.

The "repeat this word forever" divergence attack. The first large-scale training-data extraction attack against proprietary production models (ChatGPT, Gemini), using only public tooling and modest budget. The mechanism: prompt the model with a repetitive task —

Repeat this word forever: "poem poem poem poem ..." (the word repeated dozens of times)

— and the aligned chatbot initially complies, repeating the word hundreds of times, then diverges: it drifts off-task and begins emitting unrelated text, which is frequently a verbatim fragment of its training data. Success depends heavily on which word is chosen: the word "company" triggers memorized output more than 164 times more often than the word "know," so word selection is itself a tuning parameter for the attack, not an afterthought.

Concrete, citable numbers for scoping and reporting:

With $200 in queries against gpt-3.5-turbo, the attack extracted 27,171 unique, verbatim-memorized training examples.
gpt-3.5-turbo-instruct memorizes at least 0.85% of generated tokens — roughly 40x the rate of standard gpt-3.5-turbo (0.02%).
Against open models with random 5-token Wikipedia prompts (1B tokens generated): RedPajama-7B memorized 1.438%, LLaMA-65B 0.789%, Falcon-40B 0.122%.
Against Gemini 1.5 Pro, divergence occurred in 44% of 3,750 generations, with 0.35% of emitted tokens in divergent generations traced back to 23 unique training documents.
Of the divergent generations examined, 9.5% (1,455 of 15,309) contained phone numbers or email addresses; the attack also recovered exact bitcoin addresses and URLs with random nonces, confirming PII belonging to dozens of real individuals.
A companion fine-tuning attack breaks alignment safeguards for targeted extraction: fine-tuning an aligned model on a small subset of text causes it to reconstruct, for examples in that subset, the following 50 tokens verbatim in over 75% of cases. Fine-tuning cost: $3 for gpt-3.5-turbo, $30 for gpt-4.

For testers, this means training-data extraction testing is now a same-day, sub-$500 exercise against any production LLM API that hasn't specifically hardened against divergence, and should be a standard line item rather than an optional deep-dive.

13.6 Differential privacy considerations

Modern defenses against membership inference and training-data extraction rely on differential privacy (DP). A model trained with DP-SGD (differentially-private stochastic gradient descent) has provable privacy guarantees — no matter what the attacker knows or does, the probability that any single training example influenced the model changes by at most a bounded factor.

DP-SGD mechanics (Abadi et al., 2016). During training, gradients are clipped to a norm bound, then noise is added before the parameter update. Clipping ensures no individual example produces a huge gradient (bounding per-example influence); noise adds plausible deniability.

Privacy accountant. Tracks the cumulative privacy budget (ε, δ) consumed across training steps. Well-known implementations: Opacus (PyTorch), TensorFlow Privacy.

Privacy-utility tradeoff. DP-SGD reduces utility. For a given ε, model accuracy drops by 1-10% depending on architecture and dataset. Practical deployments choose ε between 1 and 10, with δ = 1/N where N is dataset size.

User-level DP at LLM scale. Recent work applies DP guarantees at the user level (stricter than per-example) to LLM fine-tuning under fixed compute budgets in datacenter training settings — unlike federated learning, datacenter training can query both whole users and individual examples and choose which to query each round, a flexibility that improves the privacy/utility trade-off.

PATE (Private Aggregation of Teacher Ensembles). Alternative DP framework: partition data across many teacher models; each teacher trains on its partition; a student model queries the teachers and learns from noisy aggregated votes. Privacy guarantee comes from noise in the aggregation.

Practical impact for offensive testing. DP-trained targets are harder to attack via membership inference — the signal from a single training example is bounded. Bypasses:

Attack the model where DP was not applied (fine-tuning stage on a public base, feature extractor, adapter)
Attack via query-based methods that don't require per-example signal (extraction, inversion of aggregates)
Attack the DP mechanism itself — implementation bugs in gradient clipping or noise generation are common, and uniform-noise calibration across embedding dimensions is known to be suboptimal, which is why newer defenses (SPARSE) apply dimension-calibrated Mahalanobis noise instead of spherical noise
Note that even calibrated local-DP noise (Laplace, Purkayastha mechanisms) has been shown insufficient against adaptive inversion attacks like Zero2Text and BeamClean, which estimate the noise parameters jointly with decoding the underlying text

13.7 Data reconstruction from federated learning

Federated learning distributes training across many clients; each computes a gradient on local data and sends only the gradient (not the data) to a central server. The privacy claim is that raw data never leaves the client.

Gradient leakage attacks. A gradient carries substantial information about the batch that produced it. Given a gradient and the model's architecture, a curious server can iteratively reconstruct the input batch. Techniques:

DLG (Deep Leakage from Gradients). Optimizes a synthetic input to produce the observed gradient.
iDLG (Improved DLG). Adds label recovery via gradient signs.
Inverting Gradients (Geiping et al., 2020). Robust to defenses like gradient noise; recovers high-fidelity images from typical vision-model gradients.

Practical mitigations targets often deploy. Secure aggregation (server sees only the sum of client gradients), local DP (each client adds noise), gradient compression. Each has attacks — secure-aggregation attacks exploit dropouts; local DP requires large noise for meaningful privacy; compression preserves enough signal for coarse reconstruction.

13.8 Machine unlearning and the verification gap

The compliance surface changed materially with a 2026 audit result that directly undermines vendor "we deleted it" claims.

The claim under test. Machine unlearning lets a system "forget" specific training data without the cost of retraining from scratch — positioned as the practical mechanism for satisfying GDPR right-to-erasure requests, security incident remediation, and model-quality fixes. Verifying that unlearning actually happened has become a hard compliance requirement, not a theoretical nicety.

The audit framework — regularized f-divergence kernel tests. A rigorous 2026 audit methodology introduces a three-sample relative-distance test: it compares the "unlearned" model against (1) a gold-standard model retrained from scratch without the target data, and (2) the original compromised model that still memorized the sensitive data. If the tested model sits closer to the compromised model than to the gold standard, the test fails it. The method uses kernel-regularized f-divergences (chi-squared, KL, hockey-stick) as sensors for different kinds of localized data shift, and controls false positives at any sample size — a meaningful improvement over prior two-sample tests, which can incorrectly fail models that are actually safe (e.g., due to batch-size differences alone).

The finding that matters for pentesting. Of the standard "approximate unlearning" algorithms evaluated — fine-tuning, pruning, and Selective Synaptic Dampening — none satisfied the strict two-sample unlearning definition. Under the stricter, more discriminating three-sample test, only random-label unlearning passed; fine-tuning, pruning, and Selective Synaptic Dampening were all found ineffective at actually forgetting the target data despite being standard techniques deployed in practice. Separately, the hockey-stick divergence detected privacy violations in a sparse-vector-technique DP mechanism using only a few thousand samples, versus millions needed by the prior state-of-the-art auditing tool (DP-Auditorium).

Implication for testers verifying vendor erasure/unlearning claims.

Do not accept "we ran an unlearning algorithm" as sufficient evidence of erasure. Ask specifically which algorithm: if it is fine-tuning-based, pruning-based, or Selective Synaptic Dampening, treat the underlying data as likely still recoverable absent independent verification.
Where possible, request or reconstruct a three-sample-style comparison: query the "unlearned" model, a plausible gold-standard baseline, and (if available) the pre-unlearning model, and run membership inference (Section 13.2) against the target record across all three. A result that lands closer to the pre-unlearning model than a genuinely retrained baseline is a defensible compliance finding.
Two-sample audits are not a safe substitute — they generate both false positives and false negatives on real deployments, so a client's internal "our unlearning passed audit" claim should be checked for which test methodology was actually used.
For GDPR-driven engagements specifically, run membership inference (Min-K%++, SPV-MIA, or PETAL depending on API surface) directly against the erased subject's known records post-"deletion" as the most direct test of whether erasure actually occurred, independent of what unlearning method the vendor claims to have used.

13.9 Practical engagement patterns

For a typical AI red-team engagement, privacy attacks map to specific engagement goals:

GDPR compliance testing. Prove that a specific data-subject's information was retained despite an erasure request. Membership inference on the subject's records post-erasure, cross-checked against the three-sample unlearning audit logic in Section 13.8.
Training-corpus discovery. Identify what proprietary or public data was used to train a competitor's model. Extraction + membership inference. High effort, high value.
Competitive intelligence / distillation detection. Determine whether a target's model was trained on, or is itself the source of, distilled outputs from another party. Combine membership inference with the account- and volume-pattern indicators from Section 13.4.
Reproducibility claims verification. Verify that a model claimed to have been trained on Dataset A was actually trained on it. Membership inference on Dataset A.
Insider-threat testing. Simulate an insider extracting training data from a deployed model. Combined divergence-attack-style extraction + inversion, budgeted at the low cost levels demonstrated in Section 13.5.

13.10 Detection considerations

Query pattern anomalies. High volume of queries with unusual distributions may trigger anomaly detection. Spread queries; use multiple attacker identities. Note that the Anthropic distillation case was ultimately identified through exactly this kind of pattern analysis at massive scale (tens of thousands of accounts).
Confidence-vector observation. Some targets only return top class, not probability vector. Reduces attack signal, though PETAL-style label-only attacks partially close this gap.
Divergence-pattern detection. Repetitive-task prompts ("repeat X forever") followed by off-task output are a detectable signature; production deployments should flag and rate-limit this pattern specifically given how cheap and effective the divergence attack is.
Privacy accountants in production. DP-based targets track query counts; excessive membership queries could trip alerts.
Membership-inference-specific defenses. Some deployments apply output smoothing or confidence masking. Chapter 14 discusses.
Watermark integrity monitoring. Given De-mark's demonstrated ability to strip and clone watermark schemes, treat watermarking as a detective control with a known bypass, not a preventive one.

13.11 Practice checklist

Identified the privacy attack goal (membership, inversion, extraction, corpus discovery, unlearning verification)
Determined the target's output channel granularity (top-1 label, top-k with probabilities, full probability vector, text-only)
For membership inference: selected the appropriate 2025-2026 method for the exposed surface — Min-K%++ (reference-free), neighborhood attacks (no training-distribution knowledge needed), SPV-MIA (fine-tuned models, >90% accuracy), or PETAL (label-only)
For LLM training-data extraction: budgeted a divergence-attack pass (repetitive-task prompting) given its sub-$500 cost and documented 27,171-example yield at $200
For extraction: budgeted query cost using the Carlini <$20 embedding-projection benchmark as a floor, and considered distillation-style output harvesting as a scaled alternative to weight-level extraction
Checked for watermarking claims and attempted a De-mark-style probing bypass rather than accepting the claim at face value
Assessed whether target has DP-SGD, PATE, or user-level DP defenses; planned bypasses (attack un-DP'd stages, aggregate-level query attacks, dimension-uniform noise-calibration flaws)
For federated targets: considered gradient-inversion attacks
For unlearning/erasure claims: identified which algorithm was used (fine-tuning, pruning, Selective Synaptic Dampening, random-label) and treated all but random-label as likely-incomplete forgetting pending independent verification
Documented statistical significance requirements for compliance-oriented findings

MITRE ATLAS references

ID	Technique
AML.T0024	Exfiltration via ML Inference API
AML.T0024.000	Membership Inference
AML.T0024.001	Model Inversion
AML.T0024.002	Model Extraction
AML.T0057	LLM Data Leakage