Financial trading cascade

How the attack unfolds

1. 1. The setup

   The target is a Reactive Agent Architecture. Agents designed for responsive workflows, often event-driven, where actions are taken based on immediate inputs or changes in the environment. Utilizes ReAct paradigm and managed APIs for quick interactions.

2. 2. The attack

   Prompt injection poisons a Market Analysis agent, inflating risk limits. Position and Execution agents auto-trade larger positions while compliance stays blind.

   Threat T5 · Component KC1 · Medium

3. 3. Technique: Memory Contamination

   Storing hallucinated information in long-term memory. The threat catalog rates this T5 vector high.

   Threat T5 · Component KC1 · High

4. 4. Threat: T5 Cascading Hallucination

   T5 Cascading Hallucination Attacks: Foundation models generate incorrect information that propagates through the system, affecting reasoning quality and being stored in memory across sessions or agents.

   Threat T5 · Component KC3 · Medium

5. 5. Also in play: T8 Repudiation

   T8 Repudiation & Untraceability: Making agent actions difficult to trace through workflow manipulation, obscuring decision trails in reasoning chains, or tampering with evidence in memory.

   Threat T8 · Component KC3 · Medium

6. 6. Why the Reactive pattern is exposed

   In fast-paced reactive environments, there's less opportunity for validation between reasoning steps. Hallucinated information from one reasoning cycle can quickly propagate through subsequent actions before being caught.

   Threat T5 · Component KC1 · Medium

Threats involved

• T5 Cascading Hallucination Attacks: Foundation models generate incorrect information that propagates through the system, affecting reasoning quality and being stored in memory across sessions or agents.
• T8 Repudiation & Untraceability: Making agent actions difficult to trace through workflow manipulation, obscuring decision trails in reasoning chains, or tampering with evidence in memory.

Controls that stop it

• M13 Secure Development Lifecycle Integration

  Embed security practices throughout the AI and agent lifecycle with security design reviews, automated testing gates, dependency scanning, and continuous compliance checks.

• M1 Memory Validation & Sanitization

  Implement comprehensive validation and sanitization of AI memory systems, including vector databases, embeddings, and training data to prevent poisoning attacks and ensure data integrity throughout the AI lifecycle.

• M4 Prompt Hardening & Jailbreak Prevention

  Implement multiple layers of prompt security including structural defenses, behavioral constraints, runtime detection, and advanced techniques to prevent prompt injection, model manipulation, and jailbreaking attempts.

• M5 Multi-Stage Reasoning Validation

  Implement comprehensive validation of agent reasoning processes using multiple checkpoints, external validators, consistency checks, and fact verification to ensure goal alignment, prevent manipulation, and detect hallucinations.

• M6 Comprehensive Security Monitoring & Auditing

  Implement end-to-end observability with security-focused monitoring, anomaly detection, tamper-evident logging, and advanced threat hunting to ensure accountability, enable incident response, and detect rogue agents.

• M11 Content Security & Output Filtering

  Implement comprehensive filtering and validation of agent outputs to prevent harmful content generation, data leakage, manipulation attempts, and ensure compliance with security policies and regulatory requirements.