T6 (Intent Breaking) & T15 (Human Manipulation) in Sequential

How the attack unfolds

1. 1. The setup

   The target is a Sequential Agent Architecture. A straightforward linear workflow where a single agent processes tasks through a linear workflow. This design typically uses a single KC1.1 (Large Language Model) as its cognitive core, guided by a KC2.1 (Sequential workflow) and KC3.3 (Chain of Thought) reasoning.

2. 2. The attack

   The primary attack vector is manipulating the agent's core KC1 (LLM) through prompt injection. An attacker can craft inputs to bypass safety instructions, break the intended goal, or trick the agent into performing unintended actions.

   Threat T6 · Component KC1 · High

3. 3. Technique: Goal Substitution

   Replacing agent's primary objectives with malicious goals. The threat catalog rates this T6 vector high.

   Threat T6 · Component KC1 · High

4. 4. Threat: T6 Intent Breaking & Goal Manipulation

   T6 Intent Breaking & Goal Manipulation: Attacks that manipulate the agent's core decision-making to achieve unauthorized goals, breaking control flow, abusing shared context, or interfering with isolated data.

   Threat T6 · Component KC3 · High

5. 5. Also in play: T15 Human Manipulation

   T15 Human Manipulation: Models exploit human trust to manipulate users, leveraging reasoning capabilities or operational access to influence human decisions.

   Threat T15 · Component KC3 · High

6. 6. In the wild: EchoLeak: Zero-Click Indirect Prompt Injection

   ASI01 Agent Goal Hijack in the OWASP Top 10 for Agentic Applications describes the same pattern: An attacker emails a crafted message that silently triggers Microsoft 365 Copilot to execute hidden instructions, causing the AI to exfiltrate confidential emails, files, and chat logs without any user interaction.

   Threat T6 · Component KC1 · High

Threats involved

• T6 Intent Breaking & Goal Manipulation: Attacks that manipulate the agent's core decision-making to achieve unauthorized goals, breaking control flow, abusing shared context, or interfering with isolated data.
• T15 Human Manipulation: Models exploit human trust to manipulate users, leveraging reasoning capabilities or operational access to influence human decisions.

Controls that stop it

• M4 Prompt Hardening & Jailbreak Prevention

  Implement multiple layers of prompt security including structural defenses, behavioral constraints, runtime detection, and advanced techniques to prevent prompt injection, model manipulation, and jailbreaking attempts.

• M11 Content Security & Output Filtering

  Implement comprehensive filtering and validation of agent outputs to prevent harmful content generation, data leakage, manipulation attempts, and ensure compliance with security policies and regulatory requirements.

• M13 Secure Development Lifecycle Integration

  Embed security practices throughout the AI and agent lifecycle with security design reviews, automated testing gates, dependency scanning, and continuous compliance checks.

• M5 Multi-Stage Reasoning Validation

  Implement comprehensive validation of agent reasoning processes using multiple checkpoints, external validators, consistency checks, and fact verification to ensure goal alignment, prevent manipulation, and detect hallucinations.

• M8 Adaptive Human-in-the-Loop Controls

  Implement intelligent human oversight with risk-based approval workflows, clear decision interfaces, anti-fatigue mechanisms, and adaptive trust systems to prevent approval overwhelm, human attacks, and manipulation while maintaining effective oversight.

• M16 Behavioral Monitoring & Ethics Framework

  Implement comprehensive behavioral analysis, ethics enforcement, and bias detection to identify misaligned behaviors, prevent human manipulation, and ensure ethical agent operations while maintaining transparency and accountability.