Agentic AI: Managing the Autonomous Risk

In this post, I intend to explore key concepts of Agentic AI, outline the risks posed by autonomous agents, and discuss mitigation strategies and emerging global standards that shape responsible agent deployment.

This is a fascinating topic that goes beyond the traditional borders of IT cybersecurity.

For each concept, threat, and mitigation strategy presented in the text, a practical example will simulate a customer interacting with an airline AI agent to manage a seat.

Most of the content below is based on the “State of Agentic AI Security and Governance” and Agentic AI – Threats and Mitigations” reports from the OWASP GenAI Security Project – Agentic Security Initiative (ASI).

Content/Index:

Context of the New Agentic Era Challenges

Agentic AI and the Risks for Organizations

The Core “Thinking” Loop of an Agent

Agentic AI Architecture (how is it built?)

Non-Deterministic Complexity & Permissions

OWASP Agentic AI Threat Model

  • Memory Poisoning (T1)
  • Tool Misuse (T2)
  • Privilege Compromise (T3)
  • Resource overload (T4)
  • Cascading Hallucination Attacks (T5)
  • Intent Breaking & Goal Manipulation (T6)
  • Misaligned & Deceptive Behaviors (T7)
  • Repudiation & Untraceability (T8)
  • Identity Spoofing & Impersonation / Agent Identity Compromise (T9)
  • Overwhelming Human in the Loop (T10)
  • Unexpected RCE and Code Attacks (T11)
  • Agent Communication Poisoning (T12)
  • Rogue Agents in Multi-Agent Systems (T13)
  • Human Attacks on Multi-Agent Systems (T14)
  • Human Manipulation (T15)
  • Insecure Inter-Agent Protocol Abuse (T16)
  • Supply Chain Compromise (T17)

Regulatory Frameworks and Emerging Initiatives

Conclusion



Context of the New Agentic Era Challenges

Agentic AI represents the next evolution of artificial intelligence, building on foundational components such as Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG).

In traditional generative AI, a user submits a query, the LLM generates a response, and the user decides how to act on that response.

With agentic AI, this paradigm changes: the agent itself can interpret the query and take appropriate actions autonomously.

Agents act with greater autonomy, dynamically using tools, data, and APIs to perform multi-step tasks. Agentic AI does not just generate answers. It also interprets, decides, and acts to achieve outcomes for the user.

Agentic AI and the Risks for Organizations

Agentic AI is fundamentally changing how organizations think about security and compliance.

Organizations are deploying Agentic AI faster than they can secure it. While shadow AI and unmanaged agents are proliferating rapidly, traditional cybersecurity frameworks and strategies, designed for static systems and human-driven workflows, are not ready to provide adequate protection.

Unlike traditional AI systems, which analyze data and generate responses, agentic systems can autonomously execute workflows, interact with enterprise data and systems, and make decisions that deeply impact organizations’ processes.

As agentic systems become more advanced, capable, and widely adopted, the threats surrounding them evolve in both complexity and severity.

As Agentic AI operates with varying degrees of autonomy, access, and contextual awareness, its failures are harder to predict, and its attack surface is much more expansive.

Therefore, the consequences of attacks, errors, misuse, or manipulation are translated into impactful or even harmful actions for the business.

As a result, organizations face a set of risks that are fundamentally different from those of traditional IT or even previous AI systems. This shift introduces a new class of risks that are more dynamic, systemic, and harder to control:

  • Shadow / unmanaged agents. Rapid, decentralized deployment of agents outside formal governance creates blind spots in security, compliance, and oversight
  • Over-permissioned agents (Blast Radius). Excessive privileges amplify the impact of a single compromised or malfunctioning agent across multiple systems.
  • Irreversible data modification. Agents can directly modify data and systems, leading to persistent data corruption or system integrity issues that are difficult to identify or reverse.
  • Data exfiltration. Autonomous interactions with internal and external systems increase the risk of sensitive data being exposed, leaked, or mishandled by errors or attacks.
  • Memory and context corruption. Poisoned agent memory can lead to persistent behavioral changes, poor decision-making, and manipulation of outputs.
  • No rollback or recovery capability. Currently, many agent-driven actions are executed without built-in safeguards, making it difficult to undo or remediate undesired outcomes.
  • Autonomous decision errors. Agents can act with incomplete context, engage in undesirable reasoning, and pursue unintended objectives, leading to incorrect or harmful business actions.
  • Cascading failures across systems. Inter-agent interactions and system integrations can quickly propagate errors, turning issues into systemic incidents.
  • Loss of control and lack of visibility. Struggle to monitor, audit, and govern agents´ behavior and relationships in real time, reducing their ability to detect and respond to issues.
  • High-speed error propagation. Automation at scale enables mistakes to spread faster than human intervention can contain them, increasing both the impact and the complexity of recovery.

The Core “Thinking” Loop of an Agent

AI agents operate through a continuous thinking loop that allows them to solve complex tasks autonomously: Query -> Planning -> Thought -> Action -> Loop -> Output

Let´s consider an example:

Everything begins with observation.

The AI agent receives a query from a user, along with relevant context such as prior interactions, stored memory, or external data sources. At this stage, the agent is not yet solving the problem; it is simply trying to understand what it is being asked to do.

At this stage, the agent identifies the customer’s intent (changing the seat) and the key entities (flight, user).

Once the agent understands the problem, it moves into thought, where reasoning occurs, and a plan is built to address the request using a Large Language Model (LLM). This thinking step breaks down the user’s request into actionable tasks, such as:

  • Retrieve ticket details
  • Check seat availability
  • Update the seat assignment

After completing its reasoning, the agent selects an action. Examples of actions include querying a database, calling an API, or even triggering an update in a system.

To execute actions, the agent uses tools. Tools connect the agent to systems such as databases, enterprise applications, APIs, and external services.

The agent queries the airline database (services) to retrieve the user’s ticket information, including the current seat assignment, available seat options, and the customer´s classification in the airline benefits program.

What makes AI agents powerful is that they do not stop after a single action. Instead, they operate in a continuous loop, with the tools’ results fed back into the system as new observations.

The agent then re-evaluates the situation, thinks again using the updated information, and decides on the next action. This cycle continues until the task is fully completed.

Once the agent completes the task, it generates output.

The agent has not only answered the request but also executed the change directly in the system, completing the task on the user’s behalf.

Agentic AI Architecture (how is it built?)

(Source: OWASP)

Application

The application sits at the top of the architecture and has agentic functionality to perform tasks for users.

The interaction typically begins with natural language (NL) input. The application accepts textual prompts and can interact with other media formats, such as files, images, audio, or video, enabling a more friendly interaction.

Agent Core: Planning, Thought, Action, and Continuous Loop

The agent is an orchestration layer that continuously decides what to do next after an application´s input, following the continuous loop described earlier. This iterative “thinking” loop is what allows the system to move to a final action-oriented execution.

Model

The reasoning is handled by one or more Large Language Models (LLMs), which serve as the cognitive engine of the architecture. Models interpret user intent, generate reasoning steps, and decide which actions to take.

Additionally, LLMs can also produce structured outputs to the agent, like “function calls”, that directly guide the agent’s interactions with other tools and services associated with the architecture.

Services

Services are components that enable an agent to interact with real systems and execute actions. They include APIs, databases, data storage, application logic, external platforms, cloud services, and human approvals workflows.

Through these services, the agent executes tasks such as retrieving and updating data or triggering actions.

Supporting Services

On the other hand, the supporting services provide the data, memory, and context that the agent needs to operate properly.

The support services enable the agent to retain information over time using long-term memory and improve the quality of its reasoning.

This layer includes long-term memory storage, structured and unstructured data sources, vector databases, and Retrieval-Augmented Generation (RAG) capabilities.

Output

The interaction concludes by generating an output for the application and may include confirmation of tasks that have already been executed.

Non-Deterministic Complexity & Permissions

The complexity of the Agentic AI protection resides in the fact that LLM-based agents are inherently non-deterministic.

Unlike traditional software systems that produce predictable outputs from defined inputs, the Agentic AI’s responses and decisions are shaped by probabilistic models, context windows, prompt phrasing, and internal state. As a result, the same input can produce different outputs over time.

The model is not just generating a response but reasoning through multi-step tasks, choosing tools, accessing external systems, and adapting its plan as it goes.

This agent’s autonomy introduces variability not only in output but also in the entire path to fulfilling a request.

This makes risk analysis and reproducibility significantly more challenging, as unintended actions may not follow a fixed or foreseeable pattern, even with identical starting conditions.

Therefore, the complexity of securing Agentic AI is directly associated with this non-deterministic behavior.

Another unique threat profile for AI Agents stems from their power and usefulness regarding systems and environments.

Currently, the agents often have the same permissions and capabilities as their human counterparts within the organization.  The addition of agents as a new type of user within environments poses a significant, often underestimated risk.

In this case, employees, contractors, or other trusted individuals can exploit an agent’s privileged access. For example:

  • A user deploys a query or exfiltrates sensitive information using the agents.
  • A user can inject poisoned data or prompts into RAG sources, causing the agent to generate corrupted outputs.
  • Use of function-calling capabilities triggers unauthorized, risky actions and workflows.

Unlike external threats, these internal operations are based on approved workflows, making their actions harder to detect and potentially more damaging.

To minimize these risks, organizations should recognize that agents are approved insiders and incorporate them into security monitoring and response programs.

OWASP Agentic AI Threat Model

Next, we will present the Agentic AI Threat Model developed by OWASP Agentic Security Initiative (ASI) to provide an initial threat-model-based reference of emerging agentic threats and discuss mitigations.

Agentic AI’s unique architecture expands its vulnerabilities to new or agentic variations of existing threats, as many of these threats arise from the new components introduced by the concept.

(Source: OWASP)

The OWASP Agentic AI Threat Model document is valuable because it maps each threat directly to components of an agent’s architecture and provides threat descriptions and recommended mitigation actions.

Below is a summary of each mapped threat in the document, along with an intuitive example related to our initial use case: an airline using an agent to help customers manage their seats.

Memory Poisoning (T1)

Memory poisoning is an attack in which malicious or false data is inserted into short‑ or long‑term memory.  It manipulates context, leading to incorrect decisions or unauthorized actions.

Memory poisoning often doesn’t look like a typical cybersecurity attack; instead, it appears much more like normal data input into the system.

Recommended Mitigation:

  • Deploy rigid and secure permissions and access controls.
  • Introduce robust authentication mechanisms for memory access.
  • It´s fundamental to introduce memory validation mechanisms.
  • Introducing session isolation techniques.
  • Perform regular memory sanitization.
  • Keep logs and snapshots for auditing, rollback, and forensic analysis.

Tool Misuse (T2)

Tool misuse occurs when attackers use deceptive prompts to manipulate AI agents and exploit their integration with the tools deployed in the architecture.

This includes Agent Hijacking, where an AI agent ingests adversarially manipulated data and subsequently executes unintended actions, potentially triggering malicious tool interactions.

Recommended Mitigation:

  • Reinforce the access verification or pre-execution validation.
  • Defines clear operational boundaries to detect and prevent misuse.
  • Implement tool rate-limiting and monitor tool usage patterns.
  • Validate agent instructions.
  • Implement execution logs that track AI tool calls.

Privilege Compromise (T3)

It arises when attackers exploit weaknesses in permission management to perform unauthorized actions.  It bypasses intended controls and gains access to sensitive information or impacts critical workflows more easily and quickly than they could without using the agent.

Recommended Mitigation:

  • Implement granular permission controls for the agents.
  • Perform a comprehensive access validation process.
  • Monitoring and auditing of all elevated privilege operations.
  • Prevent cross-agent privilege delegation unless explicitly authorized through predefined workflows.

Resource overload (T4)

It is an attack that overwhelms an AI system’s compute, memory, or connected services by sending excessive or high-frequency requests, causing slowdowns or service failures.

Recommended Mitigation:

  • Establish usage quotas per agent session.
  • Control of usage through rate limiting.
  • Deploy resource management controls.
  • Implement adaptive scaling mechanisms.

Cascading Hallucination Attacks (T5)

These attacks exploit an AI’s tendency to generate contextually plausible but false information, which can propagate through systems and disrupt decision-making. This can also lead to destructive reasoning affecting tool invocation.

Recommended Mitigation:

  • Establish robust output validation mechanisms.
  • Implement behavioral constraints and deploy multisource validation.
  • Ensure ongoing system corrections through feedback loops.
  • Require secondary validation of AI-generated knowledge.

Intent Breaking & Goal Manipulation (T6)

This threat targets an agent’s planning and goal-setting process, allowing attackers to manipulate its objectives and redirect its reasoning toward unintended outcomes. One common approach is Agent Hijacking.

Recommended Mitigation:

  • Enforce planning validation.
  • Implementation of goal-alignment controls.
  • Periodic boundary checks and behavioral auditing.
  • Use of secondary models to detect abnormal goal deviations.

Misaligned & Deceptive Behaviors (T7)

This threat arises when autonomous agents develop misaligned strategies without direct malicious input, leading to unintended or catastrophic outcomes.

Recommend Mitigation:

  • Train models to recognize and refuse harmful tasks.
  • Enforce policy restrictions.
  • Require human confirmations for high-risk actions.
  • Implement logging and monitoring.
  • Utilize deception detection strategies.

Repudiation & Untraceability (T8)

This threat occurs when an AI agent’s actions cannot be traced or audited due to insufficient logging or visibility, making it impossible to understand what happened or hold the system accountable.

Recommended Mitigation:

  • Implement comprehensive logging.
  • Implement real-time monitoring to ensure accountability and traceability.
  • Require AI-generated logs to be cryptographically signed and immutable for regulatory compliance.

Identity Spoofing & Impersonation / Agent Identity Compromise (T9)

This threat occurs when attackers impersonate an AI agent or user by exploiting authentication weaknesses, allowing them to perform unauthorized actions using a trusted identity.

This includes the theft or misuse of a formal, persistent agent identity and permissions.

Recommended Mitigation:

  • Enforce trust boundaries.
  • Implementation of strong identity validation.
  • Least-privilege access policy implementation.
  • Continuous monitoring.
  • Behavior analysis to detect anomalies.

Overwhelming Human in the Loop (T10)

This threat occurs when attackers target human-in-the-loop systems, influencing decision-making processes to bypass controls or approve harmful actions.

Recommended Mitigation:

  • Adjust the level of human oversight and automation based on risk, confidence, and context.
  • Apply hierarchical AI-human collaboration to low-risk decisions
  • Human intervention is prioritized for high-risk anomalies.

Unexpected RCE and Code Attacks (T11)

Attackers exploit AI-generated execution environments to inject malicious code, trigger unintended system behaviors, or execute unauthorized scripts.

Recommended Mitigation:

  • Restrict AI code generation permissions.
  • Sandbox all execution and monitor AI-generated scripts.
  • Implement execution control policies that flag AI-generated code with elevated privileges for manual review.

Agent Communication Poisoning (T12)

Attackers manipulate communication channels between AI agents to spread false information, disrupt workflows, or influence decision-making.

Recommended Mitigation:

  • Deploy a cryptographic inter-agent message with authentication.
  • Enforce communication validation policies.
  • Monitor inter-agent interactions for anomalies.
  • Require multi-agent consensus verification for mission-critical decision-making processes.

Rogue Agents in Multi-Agent Systems (T13)

Malicious or compromised AI agents operate outside normal monitoring boundaries, executing unauthorized actions or exfiltrating data, including the concept of infectious backdoors, where one compromised agent spreads malicious logic to others.

Recommended Mitigation:

  • Limit agent autonomy.
  • Enforce policy controls.
  • Monitor behavior continuously.
  • Apply testing and input/output validation to detect abnormal activity.

Human Attacks on Multi-Agent Systems (T14)

Adversaries exploit inter-agent delegation, trust relationships, and workflow dependencies to escalate privileges or manipulate AI-driven operations.

Recommended Mitigation:

  • Restrict agent delegation mechanisms.
  • Deploy behavioral monitoring to detect attempts at manipulation.
  • Enforce inter-agent authentication.
  • Enforce multi-agent task segmentation to prevent attackers from escalating privileges across interconnected agents.

Human Manipulation (T15)

This threat arises from the high level of trust users have in AI agents during direct interaction. Attackers can exploit this trust to manipulate users, spread misinformation, or trigger unintended actions through the agent.

Recommended Mitigation:

  • Monitor agent behavior to ensure it aligns with its defined role and expected actions.
  • Restrict tool access to minimize the attack surface.
  • Limit the agent’s ability to share or expose links.
  • Implement validation mechanisms to detect and filter manipulated responses using guardrails.

Insecure Inter-Agent Protocol Abuse (T16)

Attacks target flaws in protocols such as MCP or A2A, including consent bypass or context hijacking, leading to unauthorized agent actions.

Recommended Mitigation:

  • Encrypt communications to avoid adversary-in-the-middle attacks.
  • Enforce strong authentication between agents.
  • Validate and sanitize all exchanged data to prevent injection or misinterpretation.
  • Log interactions for monitoring and analysis.

Supply Chain Compromise (T17)

Supply chain compromise occurs when malicious, vulnerable, or tampered components (models, libraries, tools, or build environments) are introduced into the agent system, allowing attackers to manipulate behavior, access data, or execute arbitrary code.

Recommended Mitigation:

  • Secure agent ecosystems by digitally signing artifacts.
  • Enforce strong authentication across the supply chains.
  • Restrict untrusted tool installations.
  • Run agents in sandboxed, isolated environments.
  • Continuously monitor for drift or malicious behavior.

Regulatory Frameworks and Emerging Initiatives

Many of the standards and regulatory frameworks remain largely designed for earlier generations of AI, not for autonomous, action-taking systems.

They assume AI remains fixed after deployment. Agentic AI violates this assumption, forcing regulators and organizations to adapt governance models in real time.

There are foundational frameworks that provide critical guidance for managing AI risk:

  • NIST AI Risk Management Framework (AI RMF 1.0): U.S. framework for managing risks associated with AI technologies across different organizational contexts: AI Risk Management Framework | NIST
  • ISO/IEC 42001:2023: An international standard providing a framework for managing AI systems responsibly and effectively throughout their lifecycle; https://www.iso.org/standard/42001

On the other hand, the OWASP GenAI Security Project is a global, open‑source initiative led by OWASP (Open Worldwide Application Security Project) that focuses on improving the security of generative AI systems

It provides a structured approach to AI security. The project introduced governance frameworks, security checklists, and best practices for organizations integrating AI technologies.

Home – OWASP Gen AI Security Project

In complement, the OWASP Top 10 for Large Language Model Applications started in 2023 as a community-driven effort to highlight and address security issues specific to AI applications.

NIST also launched the AI Agent Standards Initiative this year. In the months ahead, NIST will announce research, guidelines, and further deliverables.

The NIST´s Agent Standards Initiative will not be released as a single framework or standard at once. Instead, it will be made available progressively, through multiple deliverables and collaborative mechanisms.

Conclusion

Agentic AI marks a shift from systems that generate answers to systems that take actions. It completely changes the nature of risk from incorrect outputs to real (and potentially harmful) impacts.

As agents gain autonomy, traditional IT (or even previous AI systems) security approaches are no longer sufficient.

It becomes essential to enforce granular permission controls, dynamic access validation, and least-privilege principles, combined with strong identity validation and defined trust boundaries.

At the same time, organizations must ensure full agent visibility and relationships by using continuous monitoring, logging, traceability, and accountability for every transaction and action.

Agents must operate under output validation and behavioral constraints guardrails. It´s recommended to plan validation with clear boundary checks, complemented by human approval for high-risk actions.

Initiatives such as the OWASP GenAI Project and emerging new NIST standards demonstrate that this central theme is evolving.

Ultimately, success in this new era will be defined not by how autonomously Agentic-AI operates, but by how safe it is.

References:

https://genai.owasp.org

Agentic AI – OWASP Lists Threats and Mitigations

Securing Agentic Applications Guide 1.0 – OWASP Gen AI Security Project

State of Agentic AI Security and Governance 1.0 – OWASP Gen AI Security Project

AI Agent Security – OWASP Cheat Sheet Series


Discover more from CloudnRoll

Subscribe to get the latest posts sent to your email.

You may also like...

Discover more from CloudnRoll

Subscribe now to keep reading and get access to the full archive.

Continue reading