Human at the Helm: Governing AI Cost, Control, and Accountability
Dylan McCarthy
Principal Engineer
Dylan McCarthy
Principal Engineer
AI adoption has moved quickly from experimentation into everyday delivery.
Development teams are using coding assistants. Business teams are using generative AI platforms. Products are embedding AI capabilities into existing workflows. More recently, agentic systems have started to shift AI from a passive assistant into something closer to an active participant: planning steps, calling tools, retrieving information, taking actions, and retrying when something fails.
For many organisations, this creates a new governance challenge.
The question is no longer simply whether AI tools should be allowed. The harder question is how organisations can enable AI safely, productively, and economically as usage scales.
Teams want to move faster. Security and risk teams need guardrails. Finance teams want predictability. Technology leaders want to understand whether AI adoption is creating measurable value or simply adding a new and fast-growing cost centre.
A common answer is to keep a “human in the loop.” That still matters, especially for high-risk decisions. But for agentic systems, it is not enough. When a single task can involve many model calls, tool invocations, retrieval steps, and retries, putting a human into every loop can quickly become a bottleneck. In some cases, it can become a form of governance theatre: a manual approval step that slows the system down without meaningfully improving control.
We need a better model.
We need the human at the helm.
Human-in-the-loop is necessary, but not sufficient
Human-in-the-loop patterns are useful. They provide explicit checkpoints, review points, and approval gates. For regulated decisions, financial approvals, customer-impacting actions, security-sensitive operations, and other high-risk workflows, they remain important.
But human-in-the-loop is not the right default pattern for every AI interaction.
As AI systems become more agentic, the “loop” gets smaller, faster, and more frequent. A single user request may involve dozens of internal cycles. An agent may inspect information, call APIs, search documentation, generate content, run commands, interpret errors, and try again.
If every one of those loops requires manual approval, the organisation either loses the value of automation or creates an approval process that people eventually learn to bypass.
Human at the Helm is a different control model.
It does not remove accountability. It places accountability at the right level.
The human sets the direction. The human defines the boundaries. The human monitors the instruments. The human intervenes when the system drifts.
A human at the helm does not need to approve every turn of the propeller. But they do need to know whether the ship is still on course, whether it is burning fuel faster than expected, and whether it is heading somewhere it should not go.
For AI systems, this means having visibility into what the system is trying to achieve, what data it is using, which tools it can call, what actions it is taking, how often it is retrying, how much each cycle is costing, and when it should stop, escalate, or ask for help.
This is governance as steering, not governance as constant interruption.
Why this matters now
The first wave of generative AI adoption was often centred on individual productivity: drafting documents, summarising meetings, generating code, searching knowledge bases, or assisting with analysis. Those use cases still matter, but they are relatively easy to understand compared with agentic workflows.
Agents introduce a different challenge.
They may act over multiple steps. They may interact with systems. They may have delegated permissions. They may be built by different teams, purchased from vendors, embedded in SaaS products, or run locally by developers. Some will be officially approved. Others may appear as shadow AI before the organisation has a chance to assess them.
This is why the industry is moving towards the idea of an agent control plane.
Microsoft’s Agent 365 is one example of this direction. It positions agent governance as something that needs to be managed centrally across an organisation, not left to individual teams or isolated tools. The platform focuses on giving organisations a way to register, manage, secure, and observe agents across their environment, including agents built in Microsoft platforms, third-party agents, and unmanaged or shadow agents. [1]
That direction is important because it recognises a practical reality: organisations will not have one agent. They will have many.
They will have agents created by development teams. Agents embedded in productivity tools. Agents provided by SaaS vendors. Agents built into business workflows. Agents used by individuals. Agents running in local environments. Agents connected to enterprise data and systems.
Without a control plane, this becomes difficult to govern. With a control plane, organisations can start to answer practical questions: which agents exist, who owns them, what systems they can access, what data they can use, what actions they can take, how they are monitored, and how they are updated or retired.
These are not abstract governance questions. They are operational questions.
They are the difference between AI adoption that scales safely and AI adoption that becomes fragmented, expensive, and difficult to control.
Token economics is part of the instrument panel
Tokens are often treated as a billing detail. They are the unit that appears on invoices, dashboards, and rate cards.
But tokens are also an operational signal.
Token usage tells us how much work an AI system is doing. It shows how much information is being sent to a model, how much output is being generated, how often agents are looping, and whether the cost of a workflow is aligned to the value it creates.
This matters because AI cost does not always scale in the same way as traditional software cost.
A conventional application might have relatively predictable infrastructure usage once traffic patterns are understood. AI systems can be more variable. The same workflow may consume very different numbers of tokens depending on the user request, the model selected, the amount of information included, the number of retries, the volume of generated output, and the way the agent is designed.
For organisations moving from pilot usage to broad adoption, this creates a practical challenge. AI usage can grow across teams before the organisation has a clear understanding of where the cost is coming from or whether that cost is aligned to business value.
The goal is not to minimise tokens.
The goal is to maximise useful tokens.
That distinction matters.
Sometimes spending more tokens is the right choice. Better grounding can improve an answer, reduce rework, prevent unsafe actions, or avoid a series of failed attempts. A well-designed AI workflow may cost more per request but less per successful outcome.
The problem is not token spend.
The problem is unobserved token spend.
Cost control is not the same as cost intelligence
As AI usage grows, many organisations will naturally reach for budgets, limits, and approval gates. Those controls have a place. They can prevent runaway consumption and provide a basic level of financial control.
But cost control is not the same as cost intelligence.
A budget cap can stop the bleeding, but it does not explain why the bleeding started.
If an AI workflow is expensive, the organisation needs to understand why. Is the wrong model being used? Is the agent retrying too often? Is the task too broad? Is a high-cost model being used for low-value work? Is the AI system producing outcomes that justify the spend?
Without that visibility, organisations risk making blunt decisions.
They may restrict AI usage in places where it is creating real value. Or they may continue funding workflows that consume large volumes of tokens without improving outcomes.
Useful AI governance needs to connect cost to behaviour and behaviour to value.
This is where token economics becomes part of governance. Not because tokens are bad, but because tokens help us understand how AI systems are behaving.
Observability turns governance into control
You cannot govern what you cannot observe.
For AI systems, observability needs to go beyond uptime, latency, and error rates. Those signals still matter, but they are not enough.
A mature AI operating model needs visibility into the behaviour of AI-enabled workflows themselves. That includes usage by team, application, user, and workflow; input, output, and cached token usage; cost by model and task; model selection and routing decisions; tool calls and retry loops; data sources used in responses; permissions and actions taken by agents; escalation and policy events; quality and acceptance signals; and cost per successful outcome.
This reflects the same broader shift described in Microsoft’s Cloud Adoption Framework guidance for AI agents, where agents are treated as organisational resources that need consistent governance, security, ownership, and operational controls across the environment. [2]
Without these signals, governance becomes reactive.
Teams discover cost problems after the invoice arrives. Quality problems emerge after users lose trust. Risk problems surface after controls have already been bypassed. Shadow AI usage grows before anyone knows where it is happening.
Observability changes the conversation.
Instead of asking, “How do we stop people using AI too much?”, organisations can ask better questions. Which workflows are creating value? Which workflows are consuming cost without improving outcomes? Where are agents looping or retrying? Which models are being used for which tasks? Where should we route to cheaper models? Where is more capable model usage justified? Which tools and systems are agents accessing? Where do we need stronger controls?
This is the difference between policy and operational governance.
Policy defines what should happen.
Observability shows what is happening.
Guardrails should shape the road
As organisations adopt AI at scale, guardrails are essential. But guardrails should not simply become a set of manual gates that stand in front of every vehicle.
Good guardrails shape the road.
They define the lanes, the speed limits, the warning signs, and the points where a human needs to take direct control.
This means thinking about AI governance across ownership, identity, access, model usage, data boundaries, cost controls, and operational support.
An agent should have a clear owner, a clear purpose, and a defined lifecycle. It should have appropriate identity and permissions. It should operate within agreed data boundaries. It should use models that are appropriate for the task. It should produce telemetry that allows the organisation to understand usage, cost, quality, and risk.
For some workflows, that may mean strict approval gates. For others, it may mean automated policy enforcement, budget alerts, model routing, evaluation checks, or escalation rules. The right control depends on the risk of the workflow and the value of the automation.
The important point is that guardrails should enable safe movement, not prevent movement altogether.
These controls are not intended to slow adoption down. Done well, they do the opposite. They give teams the confidence to use AI because the organisation has visibility into what is happening and clear boundaries for how far systems can go.
Good governance does not block AI adoption.
It makes AI adoption sustainable.
What organisations should do next
For technology leaders, the next step is not simply to approve another AI tool or publish another policy. The next step is to build an operating model for AI usage.
That starts with visibility.
Organisations need to understand where AI is already being used, which teams are adopting agentic workflows, which tools and platforms are involved, and where AI systems are being given access to data, systems, or business processes.
From there, the focus should shift to control and value. Which agents need formal ownership? Which workflows require explicit approval? Which can operate safely within policy-driven guardrails? Where should model usage be routed based on cost, risk, and capability? How will token usage be attributed? How will success be measured?
This does not need to be solved all at once.
A practical starting point is to bring together architecture, security, engineering, risk, and finance stakeholders to define a small number of shared controls: approved model providers, data boundaries, agent ownership, cost attribution, observability requirements, and escalation paths.
The important thing is to treat AI governance as an operating model, not a one-time policy exercise.
At Versent, we see this as part of the next phase of cloud and platform maturity: helping organisations move from AI experimentation to governed, observable, and economically sustainable AI adoption.
Human at the Helm
Human at the Helm is not a rejection of Human-in-the-Loop.
There will always be workflows where explicit human approval is required. There will always be decisions where accountability demands a person make the final call.
But as AI systems become more agentic, organisations need a broader model.
Humans should set direction, define constraints, monitor the instruments, and intervene when systems drift. They should not be forced into every internal cycle of a system that is designed to operate through many fast, iterative steps.
In this model, governance is steering. Observability is the instrument panel. Token economics is the fuel gauge. Guardrails define the safe operating boundaries. Agent control planes help manage the fleet.
The organisations that succeed with AI will not simply be the ones that consume the most tokens or the ones that lock usage down the hardest.
They will be the ones that understand where AI creates value, where it creates waste, and how to give people the visibility and control needed to steer.
The goal is not to minimise tokens.
The goal is to maximise useful tokens.
References
[1] Microsoft Agent 365 — “The control plane for AI agents”, Microsoft.
https://www.microsoft.com/en-us/microsoft-agent-365
[2] Governance and security for AI agents across the organization, Microsoft Cloud Adoption Framework.