For the last year, the conversation around AI agents in engineering orgs was mostly about pilots. Could an agent triage a support ticket. Could an agent draft a first-pass code review. Could an agent summarize an incident. Interesting demos, low stakes, easy to walk back if they did not work.
That conversation is over. What I am seeing now, across the peer firms and infrastructure teams I talk to regularly, is agents running continuously against production systems, with real authorization, real API access, and real consequences if they misbehave. The industry data backs this up: multiple analyst firms now put the share of enterprise applications shipping with at least one embedded agent well above where it was even a year ago, and production deployment rates in banking and insurance specifically are ahead of most other sectors. This is not experimentation anymore. It is infrastructure.
I want to write about this from the seat I actually sit in, which is not "AI strategy" but "the person who gets paged when something a system does goes wrong." Because the operational reality of agentic AI is a much less discussed topic than the productivity story, and it is the part that is going to determine which of these deployments are still running in two years and which ones get quietly shut off after an incident.
The gap between "embedded" and "trusted"
There is a real split forming between how many applications technically have an agent in them and how many organizations actually trust those agents with meaningful autonomy in production. The pattern I see is that most teams cross the first threshold quickly and then stall hard at the second one, because the two problems require completely different skills.
Getting an agent embedded is mostly a model and prompting problem. Getting an agent trusted is an authorization, auditability, and failure-mode problem, and that is squarely infrastructure and security territory, not model territory. I think this is why so many agent projects stall out after the initial rollout: the team that built the demo is rarely the team equipped to answer "what happens when this agent does something wrong at 2am and nobody is watching."
What actually breaks in practice
The incidents I have seen or heard about secondhand are rarely dramatic. They are boring in the way that most infrastructure incidents are boring, which is exactly why they are dangerous.
An agent with slightly too broad a scope of API access performs a bulk action that was technically within its permissions but well outside what anyone actually intended when they granted them. An agent retries a failed action in a way that produces duplicate side effects, because nobody built idempotency into the workflow it was calling. An agent operating continuously accumulates cost in a way nobody notices until the monthly bill arrives, because the team monitored whether the agent worked and not what it cost to keep running.
None of these are novel problems. They are the same categories of failure we have always dealt with in distributed systems: overbroad permissions, non-idempotent retries, unmonitored resource consumption. What is new is the volume and the autonomy. A human operator making a mistake inside their permission scope makes it once, at human speed. An agent making the same category of mistake can make it continuously, at machine speed, before anyone notices the pattern.
Governance is lagging adoption, and everyone in the industry knows it
This is the part that should concern anyone building on this stuff right now: the surveys I have seen this year are consistent in finding that most organizations deploying agents do not yet have a mature governance model for them. Oversight, audit trails, and clear ownership of agent behavior are still catching up to how fast the agents themselves are being deployed. A meaningful share of agentic AI projects are expected to be scrapped over the next couple of years, and the honest read on that is not that the technology failed. It is that the projects were built without the operational discipline that would have let them survive contact with a real incident.
I have started thinking about this the same way I think about any other production system a firm did not build with change management or audit logging from day one: it is not that it cannot be retrofitted, but retrofitting governance onto something already running is always more expensive and more painful than building it in from the start.
What I am actually doing about this on my own systems
A few concrete things, none of which are exotic, all of which I think are underused relative to how much they matter.
Treat agent permissions the same way you would treat a service account, not the same way you would treat a helpful assistant. If you would not grant a human contractor standing write access to a system, do not grant it to an agent by default. Scope access to the narrowest set of actions the workflow actually requires, and revisit that scope on a schedule, not just when something goes wrong.
Build an audit trail before you build the agent, not after. Every action an agent takes against a production system should be logged in a way that lets you reconstruct, after the fact, exactly what happened and why. This sounds obvious. It is skipped constantly, because it is not the interesting part of the build and it does not show up in the demo.
Assume retries will happen and design for idempotency deliberately. Agents retry. That is part of what makes them useful. It is also exactly the behavior that turns a transient failure into a duplicated side effect if the underlying action was not designed to be safely repeated.
Put a human in the loop at the boundary that actually matters, not everywhere. I do not think every agent action needs sign-off; that defeats the point. But I do think every workflow needs an explicit, deliberate answer to the question of which specific actions require human confirmation before they execute, rather than that boundary being decided implicitly by whoever built the first version.
Where I think this goes
The firms that get real value out of agentic AI over the next couple of years are, I think, going to be the ones that treat it as infrastructure from the start rather than as a productivity tool that happens to touch production systems. That means the people best positioned to make agentic AI work well are not necessarily the people who are best at prompting. They are the people who already know how to think about permission boundaries, audit trails, and failure modes, because that discipline transfers directly.
If you have spent years thinking about least privilege, idempotent retries, and what a clean audit trail actually looks like, you are closer to being ready for this than you might think. The models keep improving. The operational discipline around them is the part that is still being figured out in real time, and it is the part where infrastructure engineers, not model builders, have the most to contribute.
