Adaptive Human-in-the-Loop Controls

Threats mitigated

• T6 Intent Breaking & Goal Manipulation

  Attacks that manipulate the agent's core decision-making to achieve unauthorized goals, breaking control flow, abusing shared context, or interfering with isolated data.

• T10 Overwhelming Human in the Loop

  Bypassing human oversight in workflows by creating excessive activity requiring approval, leading to notification fatigue and reduced scrutiny.

• T14 Human Attacks on Multi-Agent Systems

  Exploits trust relationships between agents and workflows to manipulate human operators.

• T15 Human Manipulation

  Models exploit human trust to manipulate users, leveraging reasoning capabilities or operational access to influence human decisions.

OWASP AISVS coverage

The AISVS 1.01 chapters and verification requirements this control helps satisfy.

C9 Orchestration & Agentic Security

• C9.2.1 Level 1

  Verify that the agent runtime blocks execution of privileged or irreversible actions until explicit human approval is received and verified.

• C9.2.2 Level 2

  Verify that approval requests display canonicalized and complete action parameters, such as diffs, commands, recipients, amounts, resources, and scopes, without truncation or unsafe transformation.

• C9.2.3 Level 2

  Verify that each high-impact action has a trusted reversibility classification, such as read-only, reversible, externally reversible, or irreversible.

• C9.2.4 Level 2

  Verify that the agent runtime enforces reversibility classifications by blocking, requiring approval, or restricting actions based on their impact and ability to be reversed.

• C9.2.8 Level 3

  Verify that approvals are cryptographically bound to action parameters, requester identity, execution context, and a unique single-use nonce, and are rejected if any bound value changes or the nonce is reused.

• C9.2.10 Level 3

  Verify that approval gates for multi-step or multi-agent action chains enforce the highest-impact reversibility classification present anywhere in the chain, where the chain is established by a component outside the control of the agent being gated.

• C9.6.2 Level 2

  Verify that when a human-approval gate is not satisfied within the defined approval time, the system blocks the pending action.

C12 Monitoring, Logging & Anomaly Detection

• C12.4.2 Level 2

  Verify that audit logs capture security-critical proactive actions, including approver identity, timestamp, action parameters, and decision outcomes.

Design phase

Architecture, threat modelling and policy decisions made before code is written. 25 tasks.

Risk-Based HITL Architecture

• Design action classification system based on risk levels and impact:
  • HIGH RISK: Financial transactions, system administration, data deletion
  • MEDIUM RISK: Bulk operations, configuration changes, external communications
  • LOW RISK: Read-only operations, routine tasks, cached responses
• Plan adaptive approval thresholds based on context and agent performance
• Design escalation mechanisms for complex or unusual requests
• Plan human reviewer assignment based on expertise and availability
• Design approval interface optimized for quick, informed decision-making
• Plan approval workflow automation with intelligent routing
• Design approval performance metrics and quality assessment
• Plan approval audit trails and decision accountability

Anti-Fatigue & UX Design

• Design intelligent batching and grouping of similar approval requests
• Plan context-rich approval interfaces with relevant information presentation
• Design decision support systems with recommendations and risk assessment
• Plan approval time limits and automatic escalation mechanisms
• Design reviewer rotation and workload balancing systems
• Plan approval quality feedback and learning mechanisms
• Design mobile-friendly interfaces for timely approvals
• Plan approval delegation and backup reviewer systems

Trust & Performance Adaptation

• Design agent trust scoring based on historical performance and behavior
• Plan dynamic approval requirements based on trust levels and risk assessment
• Design learning systems that adapt approval requirements over time
• Plan approval bypass mechanisms for trusted agents and routine operations
• Design trust recovery mechanisms for agents with degraded trust scores
• Plan trust calibration based on human reviewer feedback and outcomes

Build phase

Implementation, testing and hardening while the agent is developed. 37 tasks.

Risk Assessment & Classification Engine

• Build automated risk assessment for all agent actions:
  • Financial impact analysis with configurable thresholds
  • Data sensitivity classification and access impact assessment
  • Operational risk evaluation with business impact scoring
  • Compliance and regulatory impact assessment
  • External dependency and integration risk analysis
• Implement machine learning models for risk prediction:
  • Historical outcome analysis for similar actions
  • Context-aware risk scoring based on current environment
  • Anomaly detection for unusual request patterns
  • Risk correlation analysis across multiple factors

Intelligent Approval Interface

• Build context-rich approval dashboards with comprehensive information:
  • Agent identity, trust score, and historical performance
  • Action details with parameters, scope, and potential impact
  • Risk assessment summary with key factors and mitigation options
  • Similar historical actions with outcomes and lessons learned
  • Time constraints and escalation information
• Implement approval decision support systems:
  • Recommendation engines based on policy and precedent
  • Risk mitigation suggestions and alternative approaches
  • Expert consultation and collaboration features
  • Decision templates and guided workflows

Adaptive Approval Engine

• Build dynamic approval routing based on expertise and availability:
  • Subject matter expert identification and assignment
  • Workload balancing across multiple reviewers
  • Escalation to senior reviewers for high-risk or complex requests
  • Cross-functional review for interdisciplinary impacts
• Implement approval performance optimization:
  • Response time tracking and optimization
  • Approval quality metrics and feedback integration
  • Reviewer training and capability development
  • Approval process continuous improvement

Anti-Fatigue Mechanisms

• Build intelligent request batching and prioritization:
  • Similar request grouping for efficient batch processing
  • Priority-based ordering with urgent request handling
  • Request deduplication and consolidation
  • Automated pre-screening and filtering of routine requests

Operations phase

Monitoring, response and continuous improvement once the agent is live. 27 tasks.

HITL Operations Management

• Monitor human reviewer performance and decision quality metrics
• Execute regular training and capability development for reviewers
• Maintain reviewer availability and workload balancing
• Deploy reviewer fatigue monitoring and prevention measures
• Execute approval process optimization and improvement initiatives
• Maintain approval quality assurance and audit procedures
• Deploy backup and contingency procedures for reviewer unavailability

Approval Workflow Operations

• Monitor approval queue performance and processing times
• Execute regular review of approval policies and thresholds
• Maintain approval decision audit trails and compliance reporting
• Deploy approval escalation and resolution procedures
• Execute approval process testing and validation
• Maintain approval system health and availability monitoring
• Deploy approval system backup and disaster recovery procedures

Trust & Adaptation Operations

• Monitor agent trust scores and performance trends
• Execute regular calibration of trust and risk assessment models
• Maintain trust score transparency and explainability
• Deploy trust recovery and improvement programs for underperforming agents
• Execute trust model validation and bias detection
• Maintain trust metrics reporting and stakeholder communication
• Deploy trust education and awareness programs

Quality Assurance Operations

• Execute regular approval decision quality audits and reviews
• Monitor approval outcome correlation with risk assessments
• Maintain approval decision feedback loops and learning systems
• Deploy approval process benchmarking and best practice sharing
• Execute approval system security and integrity monitoring
• Maintain reviewer certification and competency assessment

Tools and frameworks

Workflow & Process Management

• BPM Platforms: Camunda, Activiti, jBPM, Microsoft Power Automate
• Approval Systems: ServiceNow, Jira Service Management, custom workflow engines
• Task Management: Asana, Monday.com, Airtable with approval workflows
• Collaboration: Slack, Microsoft Teams, Discord with approval bots

Risk Assessment & ML

• Risk Analytics: SAS Risk Management, IBM OpenPages, custom ML models
• Decision Support: IBM Watson Decision Platform, Microsoft Cognitive Services
• ML Frameworks: scikit-learn, TensorFlow, PyTorch for trust scoring models
• Analytics: Tableau, Power BI, Grafana for approval analytics dashboards

User Interface & Experience

• Dashboard Frameworks: React, Vue.js, Angular with custom approval interfaces
• Mobile Apps: React Native, Flutter, native iOS/Android for mobile approvals
• Notification Systems: Twilio, SendGrid, Firebase for approval notifications
• Real-time Updates: WebSocket, Server-Sent Events, Socket.io for live updates

Integration & API

• API Gateways: Kong, Istio, AWS API Gateway for approval service integration
• Message Queues: RabbitMQ, Apache Kafka, Redis for approval request processing
• Database: PostgreSQL, MongoDB, Redis for approval state management
• Authentication: OAuth 2.0, SAML, JWT for reviewer authentication and authorization

Related controls

• M13 Secure Development Lifecycle Integration shares T6, T10, T14, T15
• M16 Behavioral Monitoring & Ethics Framework shares T15, T14, T6
• M4 Prompt Hardening & Jailbreak Prevention shares T6, T15
• M11 Content Security & Output Filtering shares T15, T6

References

• Human-AI Interaction Design
• Human-Centered AI