Our huisarts knows which of our children worries and which one shrugs everything off. When she says “this is nothing to be concerned about,” the sentence does its work because of years of small visits, because she remembers what we tend to overreact to. Nobody in our family has ever asked whether she followed the right protocol that morning. The trust sits in her.
Compare that with navigation. Early GPS routing sent drivers into lakes and down boat ramps; the stories were real and the lakes were real. Fifteen years later I follow the blue line through cities I half know without once checking the route. Nobody decided to trust it. The system proved itself through millions of uneventful journeys, and the checking faded. Trust in systems forms through demonstrated reliability, partly while our attention quietly went elsewhere.
Which was fine, when the worst case was a wet car. The same pattern is now arriving in healthcare and mobility, where it cannot be allowed to form by accident.
The scene that will define the next decade of service design: an AI system reads an MRI and flags a structure that looks like cancer. With AI in the room everything changes. Under what circumstances may the radiologist override it? If she dismisses the finding and the system was right, she is answerable in a way she never was for an ordinary miss. Radiologists know it: in recent studies most say liability is what worries them, because the report still carries their signature. Here the patient's view: some want the machine's tirelessness, some want the doctor's judgment. As patients, we were never asked which error we would rather live with. The insurer wants to know which reading it reimburses and which mistake it forgives. None of these are questions about the model. They are questions about the system around it, and they rewrite every professional role inside it.
So who do we ultimately trust — the person reading the results, or the output? It depends on the kind of question, and a well-designed system says so out loud. Some questions the record can answer: pattern recognition across millions of prior cases, measured and audited. There the system has earned the lead. Where the case stops resembling the data, where context that never made it into a scan changes what the finding means, the person has to be able to take it back. The failure mode is leaving that boundary to be discovered case by case, under pressure, by whoever happens to be on shift.
How does a system prove it deserves trust, to a patient, a professional, a regulator? Relational trust proves itself over years of small visits. A system has to earn it in designed form: a visible track record on cases like the one on the table, in this population, on these machines, and stated limits: what it has not seen, where it is known to be weak. But the most revealing artefact is what happens at disagreement. When the doctor and the model point in different directions, is there a real path, documented, named, survivable for the person who takes it? If overriding is informal, the trust is fake. If overriding is impossible, the human is decoration.
The same logic runs upstream into agentic systems, where software now carries out whole pieces of work on its own. Someone writes the brief, decides what the system may touch, when it must stop, and who signs the result. Accountability is built at that commissioning moment or not at all. Dan Davies has a name for the alternative: organisations build accountability sinks, structures that exist so that nobody can be asked “what did you think, and why?” An ungoverned system inside a critical process is the most efficient accountability sink ever built.
The regulation is catching up fast. From August 2026, organisations deploying high-risk AI in Europe must place oversight with people who have the competence, training, and authority to actually intervene. The law now distinguishes between a human who is present and a human who can act. I read this as Europe's actual bet on AI. The race for the biggest models runs elsewhere, on deeper capital and compute; Europe is treating trust as infrastructure, and the prize goes to whoever deploys best inside real services, under real rules. The override question is precisely what a regulator will ask to see.
Something is lost here. The huisarts kind of trust — slow, personal, carried in one person across years — does not scale into these systems, and institutions that pretend otherwise. The deliberate move is to choose where a personal relationship still carries the trust, e.g. where the outcome needs to be delivered with human empathy, and where building a legible structure as foundation is enough. Most organisations are making that choice right now by default, without noticing.
To see where your organisation stands, try this once this week: pick one AI-assisted decision that matters, and ask how your system is earning your trust, and what happens when a person disagrees with it. If someone can name the path, the person who takes it, and what it costs them, you have structure. If the answer is a shrug, you have found where to start.
I am collecting what holds up and what doesn't, across sectors. If you are designing for this inside your own organisation, I would like to compare notes.
Sources: NEJM AI — randomized study, AI and perceived legal liability for radiologists, Radiology Business — ECRI 2026 patient safety report, Five reasons radiology AI has a trust problem, EU AI Act timeline, Davies — The Unaccountability Machine