This morning, I went to the dentist.

I had been rather pleased with the dental practice's voice AI. It had answered the phone, understood what I wanted, moved the appointment around, and found me a new time. It felt like exactly the sort of practical use of AI that should make life easier for patients and reception teams.

I arrived and said I was there for my appointment.

The receptionist looked at the booking.

"You have been booked in with the implant dentist."

I do not need an implant.

There was a short pause.

"Well, the AI screwed up."

And that was where the learning stopped.

I booked another appointment and left.

Perhaps somebody investigated it later. I do not know. The receptionist may not have had access to the conversation, the supplier, or any way to report the event. But from where I was standing, a potentially valuable piece of operational evidence had just disappeared.

The interesting question was not simply: why did the AI system make a mistake?

It was: does this organisation have a way to learn when its AI makes one?

A useful piece of evidence comes from a four-year study of production generative-AI cloud incidents at Microsoft. The researchers found that the root cause was distributed across the system: infrastructure, configuration, code, external usage, and operational error.

Bar chart showing root causes in a four-year study of Microsoft production generative AI cloud incidents: infrastructure 27.2%, configuration 24.5%, code bugs 21.5%, external usage 14.1%, and operational error 12.7%.
Published root-cause percentages from a study of Microsoft production GenAI cloud-service incidents. This is not dental-booking data and does not diagnose my case. It shows why “the AI failed” is too blunt to be useful.

In that study, infrastructure, configuration, and code together accounted for 73.2% of incidents. The visible AI failure was often the symptom of a wider system problem.

"The AI screwed up" is the start of the investigation

We would not train a new member of staff by saying, "You did that wrong," and then walking away.

We would ask what they understood, what information they had, what rule they followed, where the handoff failed, and what they need to do differently next time.

Deployed AI needs the same operating discipline.

NIST's AI Risk Management Framework says roles and responsibilities for human-AI configurations should be defined, incidents should be identifiable and shareable, feedback should reach the people who developed or deployed the system, and post-deployment monitoring should include user input, overrides, incident response, recovery, and change management.

That is a long way from: "The AI screwed up."

In this case, I can see three plausible explanations. They are not mutually exclusive, and I do not know which one was true.

Three possible explanations for an AI booking the wrong dentist: an ownership gap, a genuine edge case or system failure, and a workforce incentive gap.
Do not guess which explanation is true. Capture the event, test the possibilities, and find the evidence.

Possibility one: nobody really owns the AI

The dental practice may use a system supplied by somebody else. The receptionist may only see the resulting appointment. She may not know what the AI heard, what intent it selected, which routing rule it applied, or how to send useful feedback to the supplier.

This is a common ownership gap.

The technology has gone live, but the operating model has not.

Nobody at the organisation can answer:

  • Who owns the outcome?
  • Who receives frontline feedback?
  • Can we inspect the relevant transcript or event record lawfully?
  • Can we reproduce the booking?
  • Who can change the routing rule?
  • Who tests the correction?
  • Who tells the team what changed?

NIST published a report in 2026 on the challenges of monitoring deployed AI systems. It highlighted fragmented logging, weak incident-sharing mechanisms, the overhead of collecting user feedback, and insufficient understanding of human-AI feedback loops.

In other words, buying an AI product is not the same as building the organisational ability to operate it.

If the person who sees the failure has no route to the person who can improve the system, the organisation has installed automation without installing learning.

Possibility two: I was a genuine edge case

The system may normally work very well.

Perhaps it misheard one word. Perhaps I described the treatment in an unusual way. Perhaps the voice recognition was accurate but the intent model chose the wrong appointment type. Perhaps the intent was right but the practice-management integration mapped it to the wrong clinician. Perhaps a calendar rule or specialist label was simply configured incorrectly.

Research into conversational agents in healthcare separates failures from their contributory errors. The error can arise from the user interaction, the environment, speech recognition, language understanding, the model, or another connected component. A systematic review also found that poor understanding, limited vocabulary, voice-recognition accuracy, and weak error management were recurring user concerns.

That matters because "the AI got it wrong" is not a useful diagnosis.

Was this:

  • a speech-recognition error;
  • an intent-classification error;
  • a missing piece of context;
  • a bad routing rule;
  • an integration or calendar error;
  • a confirmation failure;
  • or a human configuration mistake?

A real edge case is not an excuse to stop. It is a gift from the live environment.

If the event is captured, the organisation can reproduce it, add it to an evaluation set, correct the right layer, test the normal journeys again, and watch for recurrence. If it is not captured, the same edge case remains waiting for the next patient.

Because this happened in a healthcare setting, the evidence also needs to be handled proportionately. "Capture the event" does not mean keeping every recording forever. It means preserving only what is necessary, under the organisation's privacy, consent, retention, and access rules.

Possibility three: the workforce has no reason to help

There is a more uncomfortable possibility.

What if the reception team believes the AI is there to remove their jobs?

Why would somebody patiently train the system that management has presented as their replacement?

I am not saying that happened here. I have no evidence that anybody deliberately withheld feedback. The receptionist may have been busy, unable to access the system, or completely unaware of any reporting route.

But the incentive problem is real.

OECD surveys of employers and workers found that training and worker consultation were associated with better outcomes from workplace AI. That makes intuitive sense. People are more likely to help improve a system when they understand why it exists, have some influence over how it changes their work, and can see a safe and worthwhile role for themselves in the future.

If management says, "This AI is here because you are expensive," it should not be surprised when the team does not volunteer its hard-won operational knowledge.

The frontline knows the strange appointment types, anxious callers, accents, shortcuts, workarounds, exceptions, clinician preferences, and quiet rules that are not in the process manual. That knowledge is exactly what the AI needs.

A failed adoption can therefore look like a model problem when it is really an organisational design problem.

The answer is not to blame the workforce for resisting change. It is to involve them in the change, give them a visible correction mechanism, show what happens to their feedback, train them, and make the future job more meaningful than the old one.

Be careful of algorithm aversion

There is another trap.

A human receptionist can make a booking mistake and we treat it as an ordinary human error. An AI makes one visible mistake and somebody says, "See, it cannot be trusted. Turn it off."

Research by Berkeley Dietvorst, Joseph Simmons, and Cade Massey found that people can lose confidence in algorithms more quickly than in human forecasters after seeing them make comparable errors. Their later work found that people were more willing to use an imperfect algorithm when they were allowed even a small amount of control over its output.

That does not mean we should excuse AI errors. It means we should avoid two lazy extremes:

  • "The AI is clever, so leave it alone."
  • "The AI made a mistake, so remove it."

The better response is calibrated trust: know what the system is good at, know where it fails, let people correct it, and monitor whether it is improving.

Healthcare already knows how to ask the better question

The NHS Patient Safety Incident Response Framework is built around learning and improvement. It asks organisations to use systems-based approaches, involve the people affected, respond proportionately, and focus oversight on strengthening the response system.

Its underlying move is important: understand how an incident happened rather than leaping straight to blame.

My dental booking was an administrative inconvenience, not a clinical incident. I am not suggesting that every wrong appointment needs a formal investigation. But the learning principle travels rather well.

A proportional five-minute response could have started with:

  1. What did Tony ask for?
  2. What appointment did the system create?
  3. What did the relevant event record say the system understood?
  4. Which routing or integration rule produced the result?
  5. Has this happened before?
  6. Who owns the follow-up?

That is not bureaucracy. It is how a deployed system becomes a better one.

A six-step learning loop for deployed AI: capture, classify, reproduce, correct, test, and close the loop, with a named owner at the centre.
The mistake becomes useful only when a named owner can carry it through correction, testing, and closure.

The deployed-AI learning loop

This is the simple operating loop I would now expect around any customer-facing AI.

  1. Capture. Record the intended outcome, the actual outcome, time, system path, and relevant evidence. Keep only what is necessary and permitted.
  2. Classify. Was it recognition, intent, context, routing, integration, handoff, confirmation, or human configuration?
  3. Reproduce. Can the team trigger the same failure safely? If not, what evidence is missing?
  4. Correct. Change the right layer: vocabulary, prompt, rule, model, integration, training, or escalation path.
  5. Test. Test the failed journey, nearby edge cases, and ordinary journeys that the correction might accidentally break.
  6. Close the loop. Tell the frontline what changed, monitor recurrence, and record the result in the audit trail.

At the centre sits one thing that cannot be automated away:

A named human owner.

That owner does not need to inspect every conversation. They need to make sure the feedback route exists, incidents are proportionately triaged, corrections are tested, privacy is respected, and somebody closes the loop.

For high-risk AI systems, the EU AI Act now makes the underlying principle explicit through post-market monitoring requirements: providers must actively and systematically collect and analyse relevant performance data throughout the system's lifetime. A dental booking assistant is not automatically a high-risk system under that law, but the operating lesson is useful far beyond regulated high-risk AI.

The real test is whether the organisation gets better

The voice AI had already helped me. It answered the phone. It moved the appointment. It made a task easier.

Then it sent me to the wrong dentist.

Both things can be true.

The system does not need to be perfect before it can be useful. But the organisation around it must be capable of learning.

If nobody owns the AI, it will repeat mistakes.

If edge cases disappear, it will never understand the edges.

If the workforce feels threatened, the organisation may lose the very knowledge the system needs.

And if one visible error causes everybody to abandon the technology, the organisation may lose the benefits as well.

So the next time somebody says, "The AI screwed up," do not let that be the end of the conversation.

Ask:

What happened, who owns it, what did we learn, and what will be different next time?

Sources and notes