We keep saying that we need a human in the loop.
Which human?
That is the question I have been thinking about. If I have people working in different time zones, and agents working around the clock, they all need to understand what the company is trying to do. They need to talk to one another, follow the rules and respond when something changes.
Then somebody arrives and says, "Stop. Don't do that."
Do they stop? What stops? Who had the authority to give that instruction? And does the person intervening actually understand the job?
I am not arguing against human oversight. Quite the opposite. I am wondering whether we have made the phrase sound much simpler than the work involved.
There is more than one sort of loop
I find it helpful to separate three shapes of work. This is my way of thinking about them, not an official classification.
First, a sequence: do this, then that, then stop. There is a reasonably clear end.
Then a goal-seeking loop: keep trying until you reach the outcome. Fine, but within what time, cost and authority limits? What happens if the goal is not achievable?
Then an adaptive loop: keep working, look for new information and adjust the approach when circumstances change.
That last one is particularly interesting. Changing the approach is not necessarily permission to change the goal, spend more money or rewrite the company's rules. We need to decide where adaptation ends and a new decision begins.
Now connect several of those loops to one another. That is where my neat little picture starts to become rather less neat.
Two teams, several agents and one instruction to stop
Imagine a procurement team finishing its day in Britain. Another team is starting work elsewhere. Between them, agents are checking suppliers, comparing quotations and preparing an order.
This is an invented example, but it is the sort of arrangement I mean.
A colleague notices that something looks wrong and says, "Stop the order." Another person says, "No, carry on. We have already approved it." Meanwhile, one agent is still checking the supplier and another has queued the next step.
Which instruction wins? The latest? The most senior person's? The person who owns procurement? The person who spotted the problem?
And are they even disagreeing about the same thing? One may want to pause the payment. Another may think they are being told to abandon the research.
We have not solved that by putting a human in the loop. We have introduced several humans, several loops and a disagreement about what is happening.
Raising a concern is not the same as changing the rules
My instinct is that people should have an easy way to raise a concern. You should not need to be the chief executive to notice that the account number looks wrong.
But raising an alarm, pausing a defined activity, changing its instructions and authorising a restart are different permissions.
I would want those distinctions agreed in advance, rather than improvised by an agent from somebody's confident tone. A name or job title typed into a message is not much of an authority check.
That does not mean letting the agent decide that a valid stop instruction is inconvenient and ignore it. A properly authorised stop should work. The organisation needs to define who can trigger which intervention and how others can get an urgent concern heard.
There is a useful starting point in NIST's AI Risk Management Framework. It calls for clear responsibilities, trained people, defined oversight and mechanisms to disengage or deactivate systems whose performance or outcomes do not match their intended use. It is voluntary guidance, not a certificate that our arrangement is safe.
The difficult part is turning those principles into something that works at three in the morning.
The person with the authority may not have the context
I may own the business. That does not mean I have followed every conversation between the agents since breakfast.
If I interrupt, I need a useful explanation: what are you trying to achieve, what have you done, what evidence are you using, what happens next, and what would my intervention change?
Not pages of reassuring prose. A short, inspectable handover, with links to the actual work. An agent's explanation still needs checking against the records; confidence is not evidence.
This is not an entirely new problem. In her 1983 paper, Ironies of Automation, Lisanne Bainbridge discussed the difficulty of leaving people to monitor automation and take over when something goes wrong, while giving them fewer opportunities to maintain the skills and understanding they need.
That paper concerns industrial control and flight-deck examples, not today's agent teams. But the question travels rather well: how do you remain capable of taking over something you no longer regularly do?
There is an uncomfortable possibility here. We could automate the ordinary work, then give the human only the unusual, difficult decisions, with very little preparation.
That is quite a job description.
What does stop actually mean?
For ordinary business administration, I would want the answer to be more precise than "the chat has stopped responding".
Has the next action been blocked? Have queued jobs been held? Have the other agents been told? Is a retry about to start the same work again? Has an external service already accepted the instruction?
In our procurement example, pausing the research is not the same as preventing the order. Preventing the order is not the same as undoing one already placed.
I would want a status message along these lines:
Pause requested. New orders are blocked. Supplier research is paused. One earlier order may already have been submitted; confirmation is pending. The workflow owner must review the position before restart.
That is an example of the information I would want, not a claim that a particular product provides it.
The stopping mechanism should not depend solely on the agent remembering a sentence in its conversation. I would want shared controls around the work, checked before consequential actions, and a record of which workers have actually acknowledged the pause.
Nor would I assume that stopping immediately is always the safe action. Equipment, clinical processes and other safety-critical systems need their own properly designed safe-state procedures. An office workflow is not a model for everything.
The point is to define the behaviour, test it and tell the human what remains uncertain.
We need to prepare both sides
I keep coming back to two inductions.
The human needs to understand the workflow well enough to challenge it: what normal looks like, what the limits are, when to intervene and what they are allowed to change.
The agent system needs clear roles, permissions, current instructions and a way to handle incomplete or conflicting messages. By that I mean configuration, context and tested behaviour, not necessarily retraining the underlying model.
If I give an ambiguous instruction, a useful system should help me clarify it. If I have valid authority to stop the work, it should not require me to win a debate first.
There is relevant thinking in Filippo Santoni de Sio and Jeroen van den Hoven's account of meaningful human control. Their philosophical framework connects control with responsiveness to relevant human moral reasons and circumstances, and with people who understand the system and their responsibility for its effects.
It is not a tested recipe for running a company with agents. My practical reading is simpler: having a human's name on the process is not enough.
I would rather rehearse an awkward handover before a real one. Give people conflicting instructions in a test. Make an agent unavailable. Ask the next shift to explain what is paused and who can restart it. See whether the arrangement works when somebody is tired or missing a piece of the story.
The human should not just be there to carry the blame
Madeleine Clare Elish calls attention to a related danger in Moral Crumple Zones: a person with limited control over an automated system can end up absorbing responsibility when it fails. Her analysis examines accidents and the narratives around them; it is not a legal verdict on our businesses.
Still, it is a useful challenge. Are we giving somebody a genuine ability to understand and intervene, or just leaving their name beside an approval button?
And if the company wants work to continue overnight, who is actually available to make the decisions that still require a person? My instinct is that silence should not become permission simply because the authorised person is asleep.
Some work may be able to continue within agreed limits. Other work may need to wait. We should decide which is which before the night shift starts.
I do not think we have a neat answer yet
I want agents that can get on with useful work. I also want humans who can question them, redirect them and stop them. I do not think those ambitions are incompatible.
But I suspect the human in the loop needs to be a properly designed role, with enough context, time, authority and support to do it.
The questions I would start with are quite ordinary. Who owns this work? Who can pause it? What happens while they are unavailable? How do we know the pause reached everyone? Who can restart it?
I do not have a complete answer. I am finding the questions more interesting as the agents become more capable.
Have you run into this? When someone in your organisation says "stop", does everyone involved know what that means?
Sources and related reading
The linked papers and NIST framework inform the questions above. They do not establish that any particular multi-agent product or the proposed controls are safe. Sources checked on 20 October 2026.
For the wider working relationship, see Don't Just Delegate to AI. Work Beside It. and Go Slower Than You Think You Need To.