AI agents are increasingly being deployed in finance, government and medicine, but the software frameworks that run them do not yet reliably produce the records needed to monitor their actions. A study by the Partnership on AI and its colleagues found major observability gaps, including missing persistent agent identities, incomplete records of human approvals and limited visibility into changes to an agent’s permissions or “memories.” Without these records, organizations may struggle to detect policy violations, fraud or human oversight errors.
Why AI agents require continuous monitoring
Imagine a bank hiring 1,000 new employees who can access customer accounts and confidential information. Their identification badges change every day, it is unclear whether managers have reviewed or approved their work, and no one records changes they make to the approval process. This lack of visibility reflects the current state of many AI agents.
As agents move into critical sectors, from finance to government to medicine, they must be monitored in real time to reduce the risk of widespread and potentially catastrophic consequences.
That safety depends on observability: the ability to capture what a system has done, process those records into a signal and present the result to a human or automated monitoring system. In practical terms, organizations need to know what an agent did and why it did it.
However, the study found that current agent frameworks—the software used to run agents—do not reliably generate the critical observability records required for secure deployment.
Why agent behavior is harder to predict
AI agents create risks that generative AI systems alone do not necessarily present. They inherit the unpredictability of large language models, but agents also perform longer, multistep tasks. A conventional generative AI system typically produces a single output for a human to assess and act on. An agent can plan, select tools, create subagents and perform actions on its own.
As a result, an agent’s path through a task can vary. Pre-launch testing cannot fully predict every risk, making it both more difficult and more important to observe agents while they are operating.
Key gaps in agent observability
Working with colleagues in academia, civil society and industry, the researchers examined four of the most widely used agent frameworks to determine which signals they actually emit. They found significant gaps that made it difficult to achieve important safety objectives, including identifying policy violations, fraudulent activity and human oversight errors.
Among the missing requirements were persistent agent IDs and records showing which humans had approved agent tasks. Without that information, it can be difficult to establish which agent performed an action or determine whether it acted without authorization.
The researchers said these gaps can be addressed by capturing two complementary types of records: one documenting what happened during execution and another recording how the agent decided.
A “what happened” record should identify the agent while preserving privacy, document human intervention and show whether the agent’s permissions or “memories” changed. A “decision” record should describe the agent’s reasoning and indicate its level of confidence in the actions it took.
Why decision records matter in banking
Consider an AI agent that processes mortgage applications for a bank. The agent reviews an application and recommends rejecting it, after which a human loan officer signs off on the decision.
During a later audit—or in real time—banks and regulators need to reconstruct who made the recommendation, who approved it and how long the process took. They also need to examine how the agent reached its decision to ensure it did not rely on prohibited information, such as an applicant’s region of origin.
Recording what happened and how an agent reached its decision requires different technical approaches, but both are considered achievable.
OpenTelemetry could help standardize agent tracking
The absence of a reliable “what happened” record is primarily a software plumbing problem that can be addressed through OpenTelemetry. The open observability standard is maintained by engineers across the industry and by the agent frameworks that adopt it.
OpenTelemetry has launched a working group to draft a shared format for tracking agents, although the format is not yet finalized. The group needs to incorporate standards for maintaining persistent agent identities and recording human intervention, as well as changes to agent permissions and memories.
Before these systems can scale, developers and organizations must also consider the privacy implications of agent IDs that can be linked to real people.
Source: www.technologyreview.com