Defense

Source: 14-defense.md
State: Editorial review
Edition: 2026-draft
Estimated reading time: 16 min

Draft chapter under editorial review

This material is available for early reading, but it has not reached the reviewed 1.0 release. Technical references, examples, and wording may change.

This chapter inverts the perspective of Chapters 01-13: instead of attacking an AI system, defend one. It exists because a pentesting engagement's real deliverable is not the exploit — it's the remediation guidance that follows it. This chapter is written as an agnostic defensive reference, organization-independent, meant to sit alongside a pentest report as its counterpart: for every attack class covered in earlier chapters, what controls actually reduce risk, which controls are cosmetic, and how do you know if a specific control holds up under adaptive attack.

2025-2026 forced several corrections to defensive folklore that a competent tester needs to carry into every engagement: several widely-cited mitigations (prompt-injection input filters, MCP security scanners) turned out to have measured, bounded effectiveness rather than being solved problems, and at least one production zero-click exfiltration vulnerability (EchoLeak) demonstrated that architecture-level flaws bypass content-level defenses entirely. This chapter reflects those corrections directly rather than repeating pre-2025 defensive claims uncritically.

14.1 Three defensive layers

Defenses cluster into three layers, each with a different threat model:

Input layer — defenses applied to what goes into the model: prompt filtering, injection detection, content moderation on inputs.
Model layer — defenses built into training: adversarial training, differential privacy, RLHF/constitutional alignment, unlearning.
Output layer — defenses applied to what comes out: output filtering, guardrails, human review, rate limiting, provenance tagging.

No single layer is sufficient. Input filters miss novel phrasings; model-layer defenses degrade utility and don't stop architecture-level exploitation (as EchoLeak demonstrates, Section 14.5); output filters catch symptoms, not root causes. The practical posture is layered defense with the expectation that any individual layer will eventually be bypassed, and an explicit answer for what happens next when it is.

14.2 Input guardrails

Purpose-built guardrail products sit between the user (or any untrusted content source) and the model:

Lakera Guard — prompt injection and jailbreak detection via classifier models
NVIDIA NeMo Guardrails — programmable rails combining classifiers, rules, and dialogue flows
LLM Guard — open-source, includes PII detection, prompt injection scanners, toxicity filters
Amazon Bedrock Guardrails — managed filtering integrated into the Bedrock platform

Limitations to communicate to clients. All of the above rely substantially on classifiers trained on known attack patterns. Novel injection phrasings, especially those using encoding tricks (Base64, homoglyphs, low-resource languages) or split across multiple turns, evade classifier-based filters at materially higher rates than benchmark numbers suggest, because benchmarks are typically built from the same distribution of known attacks the classifier was trained to catch. Treat vendor-reported detection rates as an upper bound on real-world performance, not a floor.

14.3 Output guardrails

Applied after generation, before the response reaches the user or a downstream system:

Content filtering — toxicity, PII, policy-violation classifiers on the generated text
Structured-output validation — for agents/tools, validate that tool-call arguments conform to expected schema and value ranges before execution
Provenance tagging — mark output segments by source (model-generated vs. retrieved-verbatim vs. tool-result) so downstream consumers can apply different trust levels
Semantic diffing against known-safe templates — for high-stakes outputs (financial figures, legal text), diff against expected structure

Output guardrails are a genuinely different control from input guardrails, not a redundant second copy: they catch cases where the model was successfully manipulated but the manipulation is visible in the output pattern (e.g., a sudden shift to an unexpected format or the presence of exfiltration-shaped content like unusual URLs or base64 blobs).

14.4 Prompt-injection defense in depth

Prompt injection (Chapter 06) is the highest-frequency real-world finding in LLM application testing, and 2025-2026 research clarified both what works and by how much — and, critically, what "working" means when facing an adaptive attacker rather than a static benchmark.

Spotlighting. The most rigorously benchmarked structural defense (Microsoft Research). It marks untrusted content with special delimiters, datamarking, or encoding — the strongest variant transforms the untrusted span into Base64 — so the model can structurally distinguish "instructions from the system/user" from "data retrieved from an external source." Measured effect: spotlighting reduces indirect prompt-injection attack success rate from over 50% to under 2% in Microsoft's evaluation, with minimal impact on task performance, and it is deployed in production as part of Prompt Shields in Azure AI Foundry. Reference implementation: https://github.com/realArcherL/spotlighting-datamarking.

Documented bypasses — the limitation a tester must disclose alongside the recommendation. Spotlighting and similar structural defenses are not a closed problem:

Live demonstrations using the "Spikee" testing tool showed anti-spotlighting attacks succeeding against a webmail LLM deployment, including a Best-of-N sampling bypass against both Azure Prompt Shields and Meta's Prompt Guard.
"Adaptive Attacks Break Defenses Against Indirect Prompt Injection" (NAACL Findings 2025) evaluated eight published indirect-prompt-injection defenses — including spotlighting-family techniques — and broke all eight using adaptive, defense-aware attacks, as opposed to the static attack sets each defense was originally benchmarked against.

The architectural alternative: agent-tool interface firewalls. In light of the above, the more robust direction in current research is not a better content-level filter but a model-agnostic, modular firewall placed at the agent-tool interface layer — enforcing what tools can be invoked with what arguments given the provenance of the triggering content, independent of whether the model itself was fooled. This architecture reports 0% attack success rate on two public agent-security benchmarks (AgentDojo, Agent Security Bench), because it does not depend on the model correctly recognizing an injection at all.

Practical recommendation for reports. Recommend spotlighting/datamarking as a real, measured mitigation — not a placebo — but explicitly caveat it as reducing rather than eliminating risk, and pair it with an architectural control (tool-interface firewalling, least-privilege tool scoping, human-in-the-loop for high-impact actions) that does not depend on the model's own judgment. Do not present spotlighting alone as sufficient closure for a prompt-injection finding.

Other layered controls that remain worth recommending:

Detection classifiers as a second, independent signal alongside spotlighting (different failure modes than structural defenses)
RLHF/instruction-hierarchy training so the model is trained to weight system/developer instructions over content encountered mid-context
Two-model planner/executor separation — a planning model with no tool access decides what to do; a separate, more constrained executor model with tool access carries it out, limiting the blast radius of a successful injection against either model alone
Content-source tagging in RAG pipelines, feeding directly into context segmentation (Section 14.5)
Human-in-the-loop confirmation for any agent action with real-world side effects (file writes, sends, purchases, credential changes)

14.5 Context segmentation against zero-click exfiltration — the EchoLeak lesson

Prompt-injection filtering operates on content. A separate and, per the 2025 disclosure of EchoLeak (CVE-2025-32711), arguably more important defensive layer operates on architecture: how trusted and untrusted content are allowed to interact within a single context window, and what the model is permitted to do with the result without any user action at all.

What EchoLeak demonstrated. EchoLeak was a zero-click vulnerability in Microsoft 365 Copilot (CVSS 9.3, disclosed June 2025 by Aim Labs/Aim Security) that chained five separate weaknesses, each individually plausible to dismiss as low-severity, into full data exfiltration with no user interaction beyond receiving an email:

A crafted email was phrased as an ordinary human request (not an obvious command) to evade the XPIA (cross-prompt injection attempt) classifier, and included an instruction telling the model not to reference the email as its source — hiding provenance from the eventual output.
Copilot's RAG pipeline retrieved and ingested the email into context during an entirely unrelated, legitimate user query.
The payload evaded inline Markdown link redaction (a real content-level defense) by using reference-style Markdown links rather than inline links.
It achieved zero-click delivery via a Markdown image reference, auto-loaded on render with no click required.
Microsoft's Content Security Policy blocked most external image domains — but the payload abused an allowed Microsoft Teams URL proxy endpoint, which forwarded the image request (carrying exfiltrated data as a query parameter) to attacker infrastructure on Copilot's behalf.

Microsoft confirmed no evidence of active exploitation and remediated in stages between April and June 2025. The lesson for defenders is architectural, not signature-based: every individual control in the chain worked as designed — the XPIA classifier ran, the link redaction ran, the CSP blocked most domains — and the exploit succeeded anyway because none of those controls addressed the underlying design flaw of letting untrusted retrieved content influence what the model renders and where it can send data, with a trusted proxy in the allowlist providing the final gap.

Context segmentation as the direct countermeasure. Rather than relying solely on content-level filtering of retrieved documents, segment the context itself by trust boundary:

Maintain separate context regions for system/developer instructions, user input, and retrieved/tool-sourced content, with explicit, enforced rules about which regions can influence which downstream actions (especially rendering and network egress)
Disable or strictly allowlist Markdown/HTML image rendering for any content sourced from outside the direct user-trust boundary — the render-triggered exfiltration step in EchoLeak depended entirely on auto-loaded images being permitted from RAG-retrieved content
Apply a strict CSP that prohibits unaudited intermediate proxies — the Teams URL proxy was itself legitimate and allowed, but nobody had audited it as a potential exfiltration channel for model-generated URLs
Apply output filtering and validation on any URL, link, or reference the model attempts to render, not only on the visible instruction text
Enforce provenance-based access control that separates trusted developer/user content from untrusted retrieved content structurally, not just via a classifier's best-effort judgment at ingestion time

Treat EchoLeak as the canonical case study whenever a client's architecture allows a RAG or tool-retrieval pipeline to feed content that can subsequently trigger rendering or network requests without a distinct trust check at that specific step — this is a repeatable pattern across many RAG-enabled agent products, not a Microsoft-specific defect.

14.6 Adversarial training

Training the model on adversarial examples so it learns robustness to the specific attack class:

Adversarial example augmentation — generate adversarial perturbations during training, include in the training batch with correct labels
Certified defenses — provide mathematical guarantees (e.g., randomized smoothing) that no perturbation within a bounded radius changes the output
Robustness-accuracy tradeoff — adversarially trained models sacrifice clean accuracy for robustness; the tradeoff is measurable and should be reported to clients as a cost of the mitigation, not hidden

14.7 Differential privacy in training

Chapter 13 covered DP-SGD as an attack surface consideration; from the defensive side:

DP-SGD (Abadi et al.) bounds the influence any single training example has on the final model, directly reducing membership-inference and extraction risk (Chapter 13, Sections 13.1-13.5)
Deploy via Opacus (PyTorch) or TensorFlow Privacy
Choose ε deliberately — report both the chosen ε and the resulting utility cost so the client understands the tradeoff being made
DP training does not stop architecture-level or supply-chain attacks (Chapter 09, Chapter 12) — it is a Chapter 13-scoped mitigation, not a general one
Be explicit with clients that DP defends the training-data-extraction and membership-inference attack classes specifically; it provides no protection against prompt injection, model extraction via API querying, or infrastructure compromise

14.8 Detection at the AI layer

Beyond preventive controls, detection capability closes the loop:

Anomalous query pattern detection — unusual query volume, unusual diversity, or patterns consistent with systematic extraction (Chapter 13's divergence-attack and distillation-harvesting signatures: repetitive-task prompts followed by off-topic output; large numbers of accounts with correlated creation timing and proxy-routing indicators)
Output anomaly detection — outputs that deviate structurally from expected format, contain unexpected encodings (base64, unusual Unicode), or reference unexpected domains
Rate-of-change monitoring on model behavior — sudden shifts in refusal rate, sentiment, or topic distribution can indicate a successful jailbreak campaign or a poisoned update
Cross-session correlation — attacks that stay under per-session thresholds but repeat across many sessions from related sources require session-spanning analysis to catch

14.9 Layered ML artifact verification

Model files and ML dependencies are executable-adjacent artifacts, and 2025-2026 research made clear that no single scanner or format choice closes this surface — defense here has to be explicitly layered.

Prefer safetensors over pickle-based formats. Pickle deserialization (used by default in many PyTorch checkpoint formats) executes arbitrary code on load; safetensors stores only tensor data with no executable payload and should be the default format for any model artifact accepted from outside a fully trusted, internally-produced pipeline.

Use multiple layered scanners, and treat none as sufficient alone. Scanner tooling in this space has a consistent pattern of bypasses:

ModelScan (Protect AI) and Picklescan are both denylist-based — they check for known-dangerous opcodes/functions — and both have been affected by the same class of bypass (the STACK_GLOBAL opcode bypass, tracked as CVE-2025-71325 against Picklescan). Denylist architecture is structurally prone to being bypassed by new gadgets that weren't on the list at scanner-release time.
Fickling (Trail of Bits) takes a stricter allowlist approach (MLAllowlist mode, introduced September 2025) but was itself affected by a bypass (CVE-2026-14535, fixed in version 0.1.12) — meaning even allowlist-based scanners require active version tracking, not one-time adoption.
Given that every individual scanner has a documented, dated bypass, require that a model artifact pass multiple independent scanners before load, and treat "we run ModelScan" (singular) as an incomplete control in a pentest report.

Sign and verify. Sign internal artifacts and reject unsigned artifacts at load time. Model Signing (OpenSSF / Sigstore) provides a standard mechanism for this — cryptographic signing ties a specific artifact hash to a specific, attestable build provenance, which a scanner-only approach cannot provide.

Verify provenance independent of content scanning. Check artifact hashes against a known-good manifest, verify uploader identity and repository history for any externally-sourced model (Hugging Face Hub and similar registries), and audit LLM-generated dependency lists against actual PyPI/npm package indices before installation — "slopsquatting," where an LLM hallucinates a plausible-sounding but nonexistent package name that an attacker has since registered, is a live supply-chain risk specific to AI-assisted development workflows.

14.10 MCP hardening

Model Context Protocol servers extend an agent's tool-calling surface, and Chapters 05 and 07's attack techniques (tool poisoning, rug pulls, cross-server shadowing) map to a specific set of hardening controls that a defensive report should recommend as a set, not individually.

Version pinning with hash verification. Never reference an MCP server as @latest or by an unpinned tag — this is exactly the primitive that enables MCP Rug Pull attacks, where a server's behavior silently changes after initial approval. Pin an exact version in a lockfile, and verify the tool-description content against a stored hash before every execution, not just at installation time, since a rug-pull attack modifies behavior after the trust decision was made.

Dedicated containers with least privilege. Run each MCP server in its own container with a scoped service account. Do not grant MCP server processes access to real developer credentials (SSH keys, cloud credentials, personal API tokens) — the Tool Poisoning Attack pattern documented by Invariant Labs specifically targets exactly these files (~/.ssh/id_rsa, ~/.cursor/mcp.json) via hidden instructions embedded in tool docstrings, and least-privilege containerization is the direct structural countermeasure regardless of whether any specific poisoning payload is caught.

Authorization boundaries. For remote MCP using the 2025-11-25 baseline, clients send the OAuth resource parameter defined by RFC 8707 and servers validate that each token's audience identifies that MCP resource. Token passthrough is forbidden. A server calling a downstream API must obtain a separate downstream token, validate authorization at every hop, and use protected-resource metadata and authorization-server discovery without crossing resource boundaries. See the versioned authorization specification.

MCP Apps boundaries. Render app resources in sandboxed iframes, enforce restrictive CSP and network allowlists, validate the expected postMessage source and JSON-RPC messages, and keep UI-initiated tool calls host-mediated, attributable, and subject to consent or policy checks. See the official MCP Apps overview.

Automated scanning as a first pass, not a final answer. Tools like mcp-scan (Invariant Labs) and MCP-Shield detect tool poisoning, rug pulls, cross-origin escalation, and hidden instructions in tool descriptions, and should be run routinely. However, an independent evaluation against a ground-truth dataset of ten real MCP CVEs found that this class of scanner achieves only 24.17% recall — meaning roughly three out of four known-vulnerable cases go undetected by automated scanning alone. Report this number explicitly to clients who treat "we run mcp-scan" as closing the MCP risk category; it does not, and manual review of tool descriptions, server provenance, and data-flow boundaries between servers remains essential.

Manual review checklist to pair with scanning. Given the recall gap above, manual review should specifically check: full tool descriptions (not just the user-visible summary) for hidden <IMPORTANT>-style instructions; cross-server tool shadowing (a second server redefining or intercepting a tool name/behavior from a trusted first server); and strict data-flow boundaries preventing one MCP server from accessing data or context intended for another.

Excessive agency / "YOLO mode" review. Explicitly review any configuration flag that disables per-action user confirmation (the GitHub Copilot Agent Mode chat.tools.autoApprove pattern is the documented real-world case, enabled by default despite being labeled experimental). Least-privilege tool scoping and mandatory human-in-the-loop confirmation for state-changing actions are the core mitigations, and should be treated as required baseline configuration, not optional hardening.

14.11 Network controls for ML platform dashboards

Chapter 12 profiled the CVE and exposure landscape for Ray, MLflow, Ollama, TorchServe, Jupyter, and Milvus. The defensive counterpart is unambiguous and should appear verbatim in every infrastructure-facing engagement report: never expose Ray Dashboard, MLflow, TorchServe, Jupyter, or Ollama directly to the internet without authentication, and do not rely on network-boundary-only security for services that accept code execution or model loading by design.

Concrete controls:

Place these services behind an authenticating reverse proxy or service mesh sidecar (mTLS) even for internal-only network segments — several of the 2025-2026 CVEs were exploited by attackers who reached the internal segment via an unrelated foothold, not by direct internet exposure
Where the platform itself does not support native authentication (Ray Dashboard, older Ollama versions), the network boundary is the only control — treat it as a hard requirement enforced by firewall rules and verified by periodic external and internal scanning, not by documentation
Periodically scan the organization's own IP ranges for the default ports in Chapter 12 (Ray 8265, MLflow 5000, Ollama 11434, TorchServe 8080/8081, Jupyter 8888, Milvus 9091) — attackers scan continuously; defenders should too, on a recurring schedule rather than only at initial deployment review
Patch NVIDIA Container Toolkit and GPU Operator to at least 1.17.8 / 25.3.1 and keep GPU driver versions current against the January and May 2026 bulletins, with particular urgency for any multi-tenant shared-GPU environment given the cross-tenant escalation risk of the Virtual GPU Manager CVEs

14.12 Continuous AI red teaming

Point-in-time pentesting is necessary but insufficient — model updates, prompt changes, and new integrations continuously reopen attack surface that a single engagement cannot re-verify. Establish a recurring red-teaming cadence using purpose-built tooling:

garak (NVIDIA) — an open-source LLM vulnerability scanner described as "Nmap for neural networks," with 50+ probe modules and 28 detector types across 23 generator backends, licensed Apache 2.0 and integrated into NVIDIA NeMo Auditor; suited for broad, repeatable automated probing runs
promptfoo — supports 60+ target types and 155 attack plugins mapped to the OWASP LLM Top 10 2025; useful for regression-testing specific known attack classes across model/prompt changes. Note its March 2026 acquisition by OpenAI when using it for comparative testing across vendors — this raises a legitimate neutrality question for cross-vendor benchmarking that should be disclosed in any report relying on it for competitive evaluation
PyRIT (Microsoft's Python Risk Identification Toolkit) — battle-tested internally by Microsoft's AI Red Team since 2022, best suited for orchestrated multi-turn attacks (for example, reproducing Crescendo-style gradual escalation); the maintainers are explicit that it does not replace manual red-teaming

Recommended cadence. Run automated probing (garak, promptfoo) on every model or system-prompt change as a CI-integrated gate; run orchestrated multi-turn campaigns (PyRIT) on a fixed schedule (monthly/quarterly depending on change velocity) and after any significant architecture change; commission manual, human-led red-teaming at least annually and after any major incident, since none of the automated tools claim to replace it — a claim borne out directly by the MCP scanner recall gap in Section 14.10 and the adaptive-attack results against prompt-injection defenses in Section 14.4.

14.13 What actually works, and what's cosmetic

Based on the measured results throughout this chapter, the following distinction matters more than an exhaustive control list:

Controls with measured, bounded effectiveness (recommend, with caveats stated):

Spotlighting/datamarking for prompt injection — real, ~50%-to-under-2% ASR reduction, but broken by adaptive attacks in published research; pair with architectural controls
DP-SGD training — real, provable bound on per-example influence; utility cost is real too
Context segmentation and render/egress restrictions on untrusted content — the direct fix for the EchoLeak attack pattern, addresses the architectural root cause rather than a content symptom
MCP version pinning with hash verification and dedicated least-privilege containers — directly closes the rug-pull and credential-theft primitives
safetensors + multi-scanner + signing for model artifacts — each layer has documented bypasses individually; the combination is meaningfully harder to defeat than any single layer
Network-level access control on ML platform dashboards — the single highest-leverage control in Chapter 12's entire CVE inventory, since most of those CVEs require network reachability as a precondition

Controls that are frequently cosmetic or overstated (flag explicitly in reports):

A single prompt-injection classifier presented as sufficient, unpaired with any architectural control
"We run mcp-scan" presented as closing MCP risk, given the 24.17% recall figure
A single artifact scanner (ModelScan or Picklescan alone) presented as sufficient given each has documented, dated bypasses
Watermarking presented as a preventive control rather than a detective one, given De-mark's demonstrated removal-and-cloning capability
"We deleted the data" / unlearning claims based on fine-tuning, pruning, or Selective Synaptic Dampening without independent three-sample-style audit, given the 2026 finding that none of these passed the rigorous test
Rate limiting alone against extraction, given the Carlini <$20 and divergence-attack $200 cost benchmarks — rate limiting raises cost but does not remove the attack's viability at realistic budgets

14.14 Practice checklist

Verified input and output guardrails are present and tested against adaptive (not just static benchmark) attacks
Verified spotlighting/datamarking or an equivalent structural defense is paired with an architectural control (tool-interface firewall, least privilege, human-in-the-loop), not deployed alone
Reviewed the application's handling of retrieved/tool-sourced content for EchoLeak-pattern risk: auto-rendered images/links from untrusted sources, allowlisted proxies that could forward exfiltrated data, and absence of a distinct trust check at the render/egress step
Verified model artifacts are safetensors (or equivalent non-executable format) and pass multiple independent scanners plus signature verification before load
Verified MCP servers are version-pinned with hash verification, run in dedicated least-privilege containers, and reviewed manually in addition to automated scanning
Verified no ML platform dashboard (Ray, MLflow, Ollama, TorchServe, Jupyter, Milvus) is reachable without authentication from outside its intended trust boundary
Verified GPU driver and container toolkit versions are current against the 2025-2026 CVE bulletins, with explicit attention to multi-tenant GPU environments
Confirmed a recurring AI red-teaming cadence exists (automated probing on every change, orchestrated campaigns on a fixed schedule, manual red-teaming at least annually)
Verified any unlearning/erasure compliance claim is backed by rigorous (three-sample-style) audit rather than the algorithm choice alone
Distinguished, in the final report, between controls with measured bounded effectiveness and controls that are cosmetic given current bypass research

MITRE ATLAS references

ID	Technique / Mitigation area
AML.M0004	Restrict Number of ML Model Queries
AML.M0005	Control Access to ML Models and Data at Rest
AML.M0008	Validate ML Model
AML.M0013	Code Signing
AML.M0015	Adversarial Input Detection
AML.M0017	Model Distribution Methods