As agentic AI systems capable of autonomous decision-making and task execution emerge, current governance frameworks face unprecedented challenges. Based on Oliver Patel's forward-looking analysis, this guide explores the novel risks and essential considerations for responsible agentic AI deployment.
AI systems that "pursue goals and complete tasks on behalf of users" (Google), acting "autonomously, making decisions, and adapting dynamically to complex environments" (Kieran Gilmurray).
Key Characteristic
Proactive and autonomous completion of tasks, moving beyond human-intermediated predictive and generative AI models.
Current Status
Limited enterprise use cases today, but rapidly advancing with potential for widespread adoption to outpace governance frameworks.
The Governance Gap
Lack of Frameworks
"Not yet any laws, standards, frameworks, or guidelines which directly address or stipulate how the novel risks of agentic AI should be mitigated."
Creeping Adoption
Unlike generative AI's "watershed moment" with ChatGPT, agentic AI adoption will likely "creep up on us," increasing the risk of failing to address challenges in time.
Potential Consequences
Organizations could be left "exposed, with an AI governance and risk management framework that is no longer fit for purpose in the agentic AI era."
Human Out of the Loop Risks
Reduced Oversight
Automation of tasks leads to significant reduction in human oversight, undermining a "foundational pillar of AI governance."
Excessive Trust
"The key risk is that we become overly trusting of agentic AI systems and take the human out of the loop to a degree which becomes dangerous."
Accountability Challenge
"If the AI agent executes the action, it may become even harder to determine which human, or entity, should be held accountable for it."
Cascading Hallucinations & Reliability Concerns
Initial Hallucination
AI agent generates inaccurate information
Reinforcement
Inaccuracy is reinforced through memory, tool use, or multi-agent interactions
Amplification
Misinformation amplifies across multiple decision-making steps
Severe Consequences
Particularly dangerous "if that agent is being trusted to execute tasks in a high-risk domain"
Adaptiveness & Unpredictability Challenges
Unpredictable Behavior
"Virtually impossible to predict how they will behave"
Risk Assessment Complexity
Significantly complicates "AI risk assessment, and therefore AI risk mitigation"
Continuous Monitoring Need
Requires more "continuous and comprehensive monitoring of AI performance"
Data, Privacy & Security Risks
Unauthorized Data Access
Agents could "easily mine and retrieve data from sources which they were not supposed to have access to," including copyrighted material or sensitive personal data.
Data Disclosure Risks
Heightened risk of agents "disclosing and revealing data to people who were not authorised to have access to it," especially with personalized tasks.
Increased Attack Surface
"Agentic AI systems will also become attack surfaces," with potential for malicious actors to take control of autonomous systems.
Malicious Code Execution
Risk of "autonomously executed (and non-vetted) malicious code creeping in to production applications" when agents generate and execute code.
Key Governance Considerations
Action Permissions & Thresholds
Define precise "action permissions" and "thresholds" to control what agents can and cannot do at the system and API level.
Integrations & Data Access
Carefully control which applications agents integrate with and which data they access to enhance privacy and restrict unintended actions.
Hierarchy & Approval Matrices
Establish clear roles and responsibilities for AI agents in complex "hierarchies of agentic collaboration, supervision, and oversight."
Monitoring & Orchestration
Implement "AgentOps" to automate the monitoring and orchestration of agentic systems, tracking actions, behavior, and adherence to guardrails.
Human Oversight & Control
Essential Override Capability
"Humans must always have the ability to override or shut down an agentic AI system" to maintain ultimate control.
Targeted Review Points
Establish "clearly defined touch points for human oversight and review" that are more targeted than reviewing every AI output.
Current Industry Practice
Many companies currently restrict agents to "read-only permissions, with human approval required for significant actions."
Use Case Risk Assessment
High-Risk Domains Requiring Extreme Caution
Advising "extreme caution in using agents to autonomously perform tasks" in domains related to EU AI Act high-risk systems, such as critical infrastructure, education, employment, essential services, and law enforcement.
Compliance Challenges
Highlighting the difficulty of complying with human oversight and automated decision-making restrictions in sensitive domains while using autonomous agents.
Tasks Unsuitable for Agents (Google's Assessment)
Tasks requiring empathy and emotional intelligence
Complex human interactions
High-stakes ethical decisions
Operations in unpredictable physical environments
Central Risk Assessment Question
The key question for organizations becomes "when is it safe to use agentic AI and when is it not?" This requires continuous evaluation as the technology evolves.