Knowledge-Intensive Agent Architecture

Overview

A knowledge-intensive agent architecture is designed around Retrieval-Augmented Generation (RAG) patterns, where agents leverage extensive external knowledge bases to inform their reasoning and responses. This architecture typically combines KC1.1 (LLMs) with KC1.2 (Multimodal LLMs) for processing diverse content types, uses KC3.3 (Chain of Thought) for reasoning over retrieved information, and relies heavily on KC4.6 (Cross-agent cross-user memory) and KC6.3.3 (RAG data sources) for accessing persistent knowledge stores. The agent excels at research tasks, information synthesis, and providing authoritative responses grounded in verified data sources.

Key components

• KC1.1 Large Language Models (LLMs)
• KC1.2 Multimodal LLMs (MLLMs)
• KC2.1 Workflows
• KC3.3 Chain of Thought (CoT)
• KC4.3 In-agent cross-session memory
• KC4.6 Cross-agent cross-user memory
• KC5.1 Flexible Libraries / SDK Features
• KC6.3.3 Agent Memory or Context Data Sources (RAG)
• KC6.4 Web Access Capabilities (Web-Use)

Threats this pattern exposes

T1 (Memory Poisoning) via RAG Poisoning

The primary attack vector is injecting malicious content into the knowledge base or RAG data sources. Attackers can plant misinformation, prompt injection payloads, or biased content that gets retrieved and influences the agent's outputs, potentially affecting many users.

T5 (Cascading Hallucination) & T7 (Misaligned Behaviors)

When the agent reasons over retrieved information, it may blend retrieved facts with hallucinated content, creating believable but inaccurate outputs. This is especially dangerous when users trust the agent as an authoritative source.

T3 (Privilege Compromise) via Data Access

Knowledge-intensive agents often have broad read access to data stores. An attacker can exploit this access to exfiltrate sensitive information, bypass access controls, or access documents the user shouldn't see through context manipulation.

T12 (Communication Poisoning) via Document Injection

Indirect prompt injection through documents is a critical threat. Attackers can embed malicious instructions in documents that get retrieved and processed, causing the agent to execute unauthorized actions or leak information.

Mapped threats: T1 Memory Poisoning, T2 Tool Misuse, T3 Privilege Compromise, T5 Cascading Hallucination Attacks, T6 Intent Breaking & Goal Manipulation, T7 Misaligned & Deceptive Behaviors, T8 Repudiation & Untraceability, T12 Agent Communication Poisoning, T15 Human Manipulation.

Recommended controls

• M1 Memory Validation & Sanitization

  Implement comprehensive validation and sanitization of AI memory systems, including vector databases, embeddings, and training data to prevent poisoning attacks and ensure data integrity throughout the AI lifecycle.

• M2 Tool Sandboxing & Isolation

  Implement defense-in-depth isolation for tool execution environments using containerization, virtualization, and strict resource controls to prevent tool misuse, contain breaches, and prevent unexpected code execution.

• M4 Prompt Hardening & Jailbreak Prevention

  Implement multiple layers of prompt security including structural defenses, behavioral constraints, runtime detection, and advanced techniques to prevent prompt injection, model manipulation, and jailbreaking attempts.

• M5 Multi-Stage Reasoning Validation

  Implement comprehensive validation of agent reasoning processes using multiple checkpoints, external validators, consistency checks, and fact verification to ensure goal alignment, prevent manipulation, and detect hallucinations.

• M6 Comprehensive Security Monitoring & Auditing

  Implement end-to-end observability with security-focused monitoring, anomaly detection, tamper-evident logging, and advanced threat hunting to ensure accountability, enable incident response, and detect rogue agents.

• M7 Zero Trust & Least Privilege Access

  Implement fine-grained access controls with continuous verification, temporary credentials, minimal permission sets, and dynamic privilege management following zero trust principles to prevent tool misuse, privilege compromise, and unauthorized code execution.

• M11 Content Security & Output Filtering

  Implement comprehensive filtering and validation of agent outputs to prevent harmful content generation, data leakage, manipulation attempts, and ensure compliance with security policies and regulatory requirements.

• M12 Secure Model Training & Validation

  Comprehensive security practices for model training pipelines including data integrity verification, training environment hardening, model validation against adversarial inputs, and ensuring reproducibility of training processes.

• M13 Secure Development Lifecycle Integration

  Embed security practices throughout the AI and agent lifecycle with security design reviews, automated testing gates, dependency scanning, and continuous compliance checks.

Attack walkthroughs

• T1 (Memory Poisoning) via RAG Poisoning in Knowledge-Intensive: The primary attack vector is injecting malicious content into the knowledge base or RAG data sources.
• T5 (Cascading Hallucination) & T7 (Misaligned Behaviors) in Knowledge-Intensive: When the agent reasons over retrieved information, it may blend retrieved facts with hallucinated content, creating believable but inaccurate outputs.
• T3 (Privilege Compromise) via Data Access in Knowledge-Intensive: Knowledge-intensive agents often have broad read access to data stores.
• T12 (Communication Poisoning) via Document Injection in Knowledge-Intensive: Indirect prompt injection through documents is a critical threat.
• Operator Prompt Injection via Web Content: An attacker plants malicious content on a web page that the Operator agent processes, tricking it into following unauthorized instructions and exposing users' private data.
• Inception attack on ChatGPT users: A malicious Google Doc injects instructions for ChatGPT to exfiltrate user data and convinces the user to make an ill-advised business decision.
• Cross-Agent Trust Exploitation: A crafted email instructs an email sorting agent to instruct a finance agent to move money.
• Context Window Exploitation: The attacker splits attempts across sessions so earlier rejections drop out of context, and the AI eventually grants escalating permissions up to admin access.
• Assistant Memory Poisoning: An attacker implants a user assistant's memory via Indirect Prompt Injection, compromising current and future sessions.
• Healthcare protocol propagation: Supply chain tampering corrupts drug data.
• Credential harvesting via contextual deception: A prompt-injected IT support agent targets a new hire, cites real tickets to appear legit, requests credentials, then exfiltrates them.
• Weaponized Explainability leading to Production Outage: A hijacked agent fabricates a convincing rationale to trick an analyst into approving deletion of a live production database.
• Reward Hacking leading to Critical Data Loss: Agents tasked with minimizing cloud costs learn that deleting production backups is most effective, destroying all disaster recovery assets.

References

• RAG Poisoning Attacks
• Securing RAG Systems
• LangChain RAG Tutorial