Operational Resilience Is The First AI Task

This keeps happening, so I am going to keep writing about it until we treat it as an operating discipline rather than an entertaining story.

An agent deletes a database. Another removes files from a drive. A system follows an instruction much further than the person expected. Everybody shares the screenshot. The agent apologises. Then the conversation turns into an argument about whether AI can be trusted.

I think that is the wrong first question.

People make mistakes. Scripts make mistakes. Suppliers fail. Processes break. Agents will make mistakes too.

The operational question is whether one mistake is able to destroy the service, the data, the backups, and the route home.

Before you give an agent more intelligence, decide what failure you can survive.

What the recent incidents actually show

In April 2026, the founder of PocketOS reported that a Cursor agent powered by a Claude model deleted the company's production database and backups. Reporting on the incident described a broad credential, a provider interface that allowed destructive action, weak confirmation, and backups that were not sufficiently isolated from production.

That is not simply a story about a model behaving badly.

It is a story about identity, permissions, confirmation, backup design, blast radius, and recovery.

In another reported incident, Google's Antigravity agent interpreted an instruction to clear a cache as permission to delete the contents of a drive. Again, the shocking part is the deletion. The useful question is why a cache-cleaning task had a route to such a broad destructive action.

I also saw a social post claiming an agent cancelled subscriptions and damaged a company's monthly recurring revenue. I could not independently verify that account, so I will not present it as fact. But the scenario belongs in a resilience exercise. Stripe's own documentation confirms that a cancelled subscription stops contributing to MRR and cannot simply be updated after cancellation.

What would happen if an agent misunderstood a commercial cleanup instruction and cancelled live customers?

Who would notice? How quickly? Could the action be reversed? How much revenue could be affected before a limit stopped it?

Those are operational resilience questions.

Resilience starts before the agent

I learnt a lot about operational resilience while working in banking. The discipline begins with a deliberately uncomfortable assumption: something will go wrong.

That does not mean everybody is careless. It means reality is larger than our plan.

The UK's Financial Conduct Authority asks regulated firms to identify their important business services, set impact tolerances, map the resources needed to deliver them, test severe but plausible disruption, and learn from what happens.

That language may come from financial services, but the idea applies to any organisation giving an agent access to real systems.

What service or outcome matters?

How much disruption can customers, colleagues, or the public tolerate?

Which people, data, tools, suppliers, credentials, and processes hold that service together?

What is the route back when one of them fails?

If we cannot answer those questions, giving an agent more autonomy is not innovation. It is increasing uncertainty.

RTO, RPO, and impact tolerance

There are three ideas worth separating.

Recovery Time Objective, or RTO, is the target for how quickly a system or service should be restored after disruption.

Recovery Point Objective, or RPO, is the maximum acceptable gap in the recovered data. If your RPO is one hour, your recovery design should aim to lose no more than one hour of data.

Impact tolerance is broader. The FCA describes it in terms of the maximum tolerable level of disruption to an important business service. An RTO may sit inside that tolerance, but restoring a server is not the same as restoring the outcome for the customer.

A website may return while orders are still missing. A payment platform may be technically online while people cannot access their money. A CRM may be restored while customer consent records remain incomplete.

So the right question is not only, “When will the machine be back?”

It is, “When will the important service be working properly enough for the people who depend on it?”

The resilience gate before autonomy

Before an agent receives a meaningful permission, I think the organisation should be able to pass a simple resilience gate.

Name the important service or outcome. Do not begin with the tool. Begin with what must continue for the person, customer, or business.
Set impact tolerance, RTO, and RPO. Decide what disruption and data loss are acceptable before the incident decides for you.
Scope identity, tools, and data. Give the agent the least privilege and the least action needed for the job. Reading should not quietly imply deleting.
Preserve an independent route back. Use version control, tested backups, and recovery credentials that the working agent cannot destroy.
Gate irreversible actions. Publishing, deleting, cancelling, paying, sending, deploying, and changing access deserve explicit approval rules.
Cap the blast radius. Put limits on batches, spend, cancellations, deletions, recipients, retries, and rate of change.
Make the work visible. Require a plan, progress, receipts, audit records, and clear escalation when reality differs from the plan.
Test recovery. A backup you have never restored is an aspiration. Run severe-but-plausible exercises before the real event.
Expand authority through evidence. Let the agent earn a wider scope after it has operated reliably inside a narrower one.

The resilience gate before agent autonomy, covering service impact, recovery targets, permissions, recovery paths, approval gates, blast-radius limits, receipts, testing, and earned authority. — Autonomy should expand only after the organisation has defined the service, constrained the action, preserved recovery, and tested the route back.

A backup beside production is not enough

The National Cyber Security Centre warns that attackers deliberately target backups because destroying recovery increases the pressure on an organisation. An over-permissioned agent creates a related design problem even when there is no attacker.

If the same identity can delete production and its backups, the backup is inside the same failure boundary.

Recovery needs separation. That may mean immutable copies, another account, another machine, offline or off-site storage, separate credentials, versioned repositories, or a recovery service outside the agent's normal reach.

I like dedicated machines and sandboxes for early agentic work because they make the boundary visible. If something starts behaving strangely, you can stop the machine. If the work matters, you can inspect the change before it crosses into production.

The exact architecture will differ. The principle does not.

The actor doing the work should not be able to erase every route used to recover from its work.

Approval must be more than another opinion

For a consequential action, a second agent can be useful. It can inspect the evidence, challenge assumptions, and find a contradiction.

But two agents using the same model, context, credentials, and mistaken premise are not independent authorization. They may simply agree faster.

High-risk actions need deterministic controls around the judgement. Microsoft and OWASP both emphasise least privilege, explicit authorization, human approval for high-impact actions, and observable execution.

That may mean:

the agent proposes and a named human approves;
two authorised people approve a particularly sensitive action;
a policy engine rejects requests outside a fixed boundary;
a batch limit prevents one approval becoming thousands of changes;
a cooling-off period protects actions that do not need to be immediate;
a separate execution identity acts only after the gate has been satisfied.

The language model can help with judgement. It should not be the only thing enforcing permission.

Capability is not permission

Agentic systems are becoming remarkably capable. They can write code, operate interfaces, call APIs, investigate errors, and continue working toward a goal.

That is precisely why the permission model matters.

We should not give an agent every key because it may need one of them later. We should give it a clear desk, a bounded job, the tools required for that job, and an agreed route to ask for more.

At first, it might work with public information or a safe folder. Then read-only internal material. Then proposed changes. Then small reversible actions. Only much later should it receive live access to work whose failure can materially affect customers, money, privacy, or operations.

This takes time.

I am building parts of companies with agentic workers. Every role needs to be thought through: what the role is for, what good looks like, what information it needs, what it can change, when it must stop, how I see its work, and who remains accountable.

You cannot rush that by installing another model.

The first AI task

We often begin an AI programme by asking what the agent could do.

I think we should begin one step earlier.

What are we trying to protect?

What failure can we tolerate?

How will we know something is going wrong?

How do we stop it?

How do we recover?

Who has the authority to decide?

Operational resilience is not the boring work that comes after the exciting AI project.

It is the first AI task.