Assistant Memory Poisoning

How the attack unfolds

1. 1. The setup

   The target is a Knowledge-Intensive Agent Architecture. Agents focused on leveraging and processing large volumes of external data, typically using sophisticated RAG approaches. Employs chain of thought for reasoning over retrieved knowledge and data connectors for accessing persistent knowledge stores.

2. 2. The attack

   An attacker implants a user assistant's memory via Indirect Prompt Injection, compromising current and future sessions.

   Threat T1 · Component KC4 · High

3. 3. Technique: Data Injection

   Injecting malicious payloads into session memory. The threat catalog rates this T1 vector high.

   Threat T1 · Component KC4 · High

4. 4. Threat: T1 Memory Poisoning

   T1 Memory Poisoning: Attackers inject malicious data into the agent's memory to manipulate future decisions, affecting any memory type from in-agent session to cross-agent cross-user memory.

   Threat T1 · Component KC4 · High

5. 5. Also in play: T6 Intent Breaking & Goal Manipulation

   T6 Intent Breaking & Goal Manipulation: Attacks that manipulate the agent's core decision-making to achieve unauthorized goals, breaking control flow, abusing shared context, or interfering with isolated data.

   Threat T6 · Component KC2 · High

6. 6. Why the Knowledge-Intensive pattern is exposed

   The primary attack vector is injecting malicious content into the knowledge base or RAG data sources. Attackers can plant misinformation, prompt injection payloads, or biased content that gets retrieved and influences the agent's outputs, potentially affecting many users.

   Threat T1 · Component KC4 · High

Threats involved

• T1 Memory Poisoning: Attackers inject malicious data into the agent's memory to manipulate future decisions, affecting any memory type from in-agent session to cross-agent cross-user memory.
• T6 Intent Breaking & Goal Manipulation: Attacks that manipulate the agent's core decision-making to achieve unauthorized goals, breaking control flow, abusing shared context, or interfering with isolated data.

Controls that stop it

• M4 Prompt Hardening & Jailbreak Prevention

  Implement multiple layers of prompt security including structural defenses, behavioral constraints, runtime detection, and advanced techniques to prevent prompt injection, model manipulation, and jailbreaking attempts.

• M5 Multi-Stage Reasoning Validation

  Implement comprehensive validation of agent reasoning processes using multiple checkpoints, external validators, consistency checks, and fact verification to ensure goal alignment, prevent manipulation, and detect hallucinations.

• M11 Content Security & Output Filtering

  Implement comprehensive filtering and validation of agent outputs to prevent harmful content generation, data leakage, manipulation attempts, and ensure compliance with security policies and regulatory requirements.

• M13 Secure Development Lifecycle Integration

  Embed security practices throughout the AI and agent lifecycle with security design reviews, automated testing gates, dependency scanning, and continuous compliance checks.

• M1 Memory Validation & Sanitization

  Implement comprehensive validation and sanitization of AI memory systems, including vector databases, embeddings, and training data to prevent poisoning attacks and ensure data integrity throughout the AI lifecycle.

• M12 Secure Model Training & Validation

  Comprehensive security practices for model training pipelines including data integrity verification, training environment hardening, model validation against adversarial inputs, and ensuring reproducibility of training processes.