Prompt Hardening & Jailbreak Prevention

Threats mitigated

• T1 Memory Poisoning

  Attackers inject malicious data into the agent's memory to manipulate future decisions, affecting any memory type from in-agent session to cross-agent cross-user memory.

• T5 Cascading Hallucination Attacks

  Foundation models generate incorrect information that propagates through the system, affecting reasoning quality and being stored in memory across sessions or agents.

• T6 Intent Breaking & Goal Manipulation

  Attacks that manipulate the agent's core decision-making to achieve unauthorized goals, breaking control flow, abusing shared context, or interfering with isolated data.

• T7 Misaligned & Deceptive Behaviors

  Model alignment issues leading to unintended behaviors that impact users, organizations, or broader populations through subtle reasoning or tool usage misalignments.

• T15 Human Manipulation

  Models exploit human trust to manipulate users, leveraging reasoning capabilities or operational access to influence human decisions.

OWASP AISVS coverage

The AISVS 1.01 chapters and verification requirements this control helps satisfy.

C2 Input Validation

• C2.1.1 Level 1

  Verify that input normalization is applied before tokenization or embedding.

• C2.1.2 Level 1

  Verify that encoding and representation smuggling in inputs is detected and mitigated. Approved mitigations include canonicalization, strict schema validation, policy-based rejection, or explicit marking.

• C2.1.3 Level 1

  Verify that all inputs that could steer model behavior are treated as untrusted and screened by a prompt injection detection ruleset or classifier, with flagged inputs blocked.

• C2.1.6 Level 2

  Verify that the system enforces an instruction hierarchy in which system and developer messages override user instructions and other untrusted inputs, even after user instructions have been processed. Untrusted content cannot acquire higher instruction priority through retrieval, tool processing, or persistence across conversation turns.

• C2.1.7 Level 2

  Verify that reserved special tokens are encoded as literal characters and cannot be injected into the model context.

• C2.1.8 Level 3

  Verify that the system can detect many-shot jailbreaking patterns.

C11 Adversarial Robustness

• C11.1.3 Level 1

  Verify that models are evaluated against known adversarial attack techniques relevant to their modality.

C12 Monitoring, Logging & Anomaly Detection

• C12.2.1 Level 1

  Verify that the system detects and alerts on known jailbreak patterns, prompt injection attempts, and adversarial inputs.

• C12.2.3 Level 2

  Verify that custom rules detect AI-specific threat patterns for coordinated jailbreak attempts, prompt injection, and system prompt extraction attempts.

Design phase

Architecture, threat modelling and policy decisions made before code is written. 22 tasks.

Prompt Architecture Design

• Design clear separation between system prompts and user inputs using structured delimiters
• Plan hierarchical instruction priority system (IMMUTABLE > SYSTEM > USER preferences)
• Design behavioral constraint frameworks with explicit forbidden action lists
• Plan prompt versioning and A/B testing strategies for security improvements
• Design context isolation mechanisms to prevent instruction bleeding
• Plan prompt template security with parameterized input handling
• Design multi-language prompt security for international deployments
• Plan prompt monitoring and anomaly detection strategies

Defensive Prompt Engineering

• Design XML tag isolation patterns for clear instruction boundaries
• Plan instruction re-contextualization techniques to treat user input as data
• Design few-shot example patterns for desired and undesired behaviors
• Plan constitutional AI integration for principle-based responses
• Design prompt complexity limits and input validation rules
• Plan output validation patterns for generated responses
• Design prompt injection detection heuristics and ML-based classifiers
• Plan prompt recovery mechanisms for failed security checks

Security Layer Architecture

• Design multi-layered defense with input filtering, prompt hardening, and output validation
• Plan real-time prompt injection detection and response mechanisms
• Design prompt audit trails and security event logging
• Plan prompt security policy enforcement points
• Design prompt testing and validation frameworks
• Plan prompt security metrics and monitoring dashboards

Build phase

Implementation, testing and hardening while the agent is developed. 27 tasks.

Structural Prompt Defenses

• Implement XML tag isolation with clear instruction boundaries:
  • Use <system_instructions> and <user_query> tags for separation
  • Implement instruction hierarchy with priority levels
  • Deploy delimiter validation and enforcement
  • Use escaping mechanisms for special characters in user input
• Build instruction re-contextualization frameworks:
  • Implement "treat as data" instructions for user inputs
  • Build context switching mechanisms for different input types
  • Deploy input normalization and canonicalization
  • Implement input length limits and complexity controls

Behavioral Constraint Implementation

• Deploy constitutional AI frameworks for principle-based responses
• Build explicit deny-lists for forbidden actions and topics
• Implement behavioral monitoring with real-time constraint enforcement
• Deploy response validation against security policies
• Build escalation mechanisms for policy violations
• Implement adaptive constraint adjustment based on threat intelligence

Injection Detection & Prevention

• Build ML-based prompt injection classifiers using transformer models
• Implement rule-based detection for known jailbreak patterns
• Deploy input anomaly detection for unusual prompt structures
• Build real-time prompt analysis with threat scoring
• Implement prompt input sanitization and normalization
• Deploy adversarial prompt detection using embedding analysis

Output Validation Framework

• Build response validation against intended functionality
• Implement content filtering for harmful or policy-violating outputs
• Deploy response consistency checking across multiple generations
• Build output attribution and provenance tracking
• Implement response audit trails for security analysis

Operations phase

Monitoring, response and continuous improvement once the agent is live. 24 tasks.

Prompt Security Operations

• Monitor prompt injection attempts and attack patterns in real-time
• Execute regular prompt security assessments and penetration testing
• Maintain prompt security baselines and deviation detection
• Deploy automated prompt security policy updates
• Execute prompt vulnerability scanning and remediation
• Maintain prompt security metrics and reporting dashboards
• Deploy incident response procedures for prompt security breaches

Continuous Improvement

• Execute A/B testing for prompt security improvements
• Analyze prompt security logs for attack pattern identification
• Update prompt security policies based on threat intelligence
• Execute red team exercises against prompt security defenses
• Maintain prompt security knowledge base and best practices
• Deploy automated prompt security testing in CI/CD pipelines

Threat Intelligence Integration

• Monitor public disclosure of new jailbreak techniques
• Update detection rules based on emerging attack patterns
• Execute threat hunting for novel prompt injection methods
• Maintain threat intelligence feeds for prompt security
• Deploy automated threat intelligence integration for prompt defenses
• Execute collaborative threat sharing with security community

Performance Monitoring

• Monitor prompt processing latency and performance impact
• Execute capacity planning for prompt security infrastructure
• Deploy performance optimization for security controls
• Maintain service level agreements for prompt processing
• Execute scalability testing for prompt security systems

Tools and frameworks

Prompt Security Frameworks

• Prompt Management: LangChain Prompt Templates, OpenAI Evals, Promptfoo for testing
• Injection Detection: Rebuff.ai, LLM Guard, custom ML classifiers using HuggingFace
• Constitutional AI: Anthropic Constitutional AI, IBM Watsonx governance
• Guardrails: NVIDIA NeMo Guardrails, Microsoft Guidance, Guardrails AI

Detection & Analysis

• ML Frameworks: scikit-learn, TensorFlow, PyTorch for injection detection models
• NLP Libraries: spaCy, NLTK, Transformers for text analysis
• Anomaly Detection: Isolation Forest, One-Class SVM, DBSCAN clustering
• Pattern Matching: Regular expressions, Aho-Corasick algorithm implementations

Testing & Validation

• Security Testing: Garak LLM scanner, PyRIT for red teaming, custom fuzzing tools
• Prompt Testing: Promptfoo, LangChain evaluations, custom test harnesses
• Benchmarking: HELM benchmark, OpenAI Evals, custom security benchmarks
• Monitoring: Weights & Biases, MLflow, custom prompt monitoring solutions

Integration & Deployment

• API Gateways: Kong, Istio, AWS API Gateway with custom security plugins
• Monitoring: Prometheus, Grafana, DataDog for prompt security metrics
• Logging: ELK Stack, Splunk, custom structured logging solutions
• CI/CD Integration: GitHub Actions, GitLab CI, Jenkins with security testing plugins

Related controls

• M11 Content Security & Output Filtering shares T5, T7, T15, T6, T1
• M13 Secure Development Lifecycle Integration shares T1, T5, T6, T7, T15
• M5 Multi-Stage Reasoning Validation shares T5, T6, T7, T1
• M16 Behavioral Monitoring & Ethics Framework shares T7, T15, T6, T5

References

• Prompt Injection Guide
• Constitutional AI Paper
• OWASP Top 10 for LLM Applications 2025