Our huisarts knows which of our children worries and which one shrugs everything off. When she says “this is nothing to be concerned about,” the sentence does its work because of years of small visits, because she remembers what we tend to overreact to. Nobody in our family has ever asked whether she followed the right protocol that morning. The trust sits in her.
Compare that with navigation. Early GPS routing sent drivers into lakes and down boat ramps; the stories were real and the lakes were real. Fifteen years later I follow the blue line through cities I half know without once checking the route. Nobody decided to trust it. The system proved itself through millions of uneventful journeys, and the checking faded. That's how systems earn trust: a track record that stacks up while we're looking elsewhere.
Which was fine, when the worst case was a wet car. The same pattern is now arriving in healthcare and mobility, where it cannot be allowed to form by accident.
The scene that will define the next decade of service design: an AI system reads an MRI and flags a structure that looks like cancer. Now: under what circumstances may the radiologist override it? If she dismisses the finding and the system was right, she is answerable in a way she never was for an ordinary miss. Radiologists know it: in recent studies most say liability is what worries them, because the report still carries their signature. The patient sees it differently: some want the machine's tirelessness, some want the doctor's judgment. As patients, we were never asked which error we would rather live with. The insurer wants to know which reading it reimburses and which mistake it forgives. Every one of these is a question about the system around the model, and each one rewrites a professional role inside it.
So who do we trust — the person reading the results, or the output? It depends on the kind of question, and a well-designed system says so out loud. Some questions the record can answer: pattern recognition across millions of prior cases, measured and audited. There the system has earned the lead. Where the case stops resembling the data, where context that never made it into a scan changes what the finding means, the person has to be able to take it back. The trap is leaving that boundary to be discovered case by case, under pressure, by whoever's on shift.
How does a system prove it deserves trust, to a patient, a professional, a regulator? Relational trust proves itself over years of small visits. A system has to earn it in designed form: a visible track record on cases like the one on the table, in this population, on these machines, and stated limits: what it has not seen, where it is known to be weak. But the real tell is what happens when they disagree. When the doctor and the model point in different directions, is there a real path, documented, named, survivable for the person who takes it? If overriding is informal, the trust is fake. If overriding is impossible, the human is decoration.
The same logic runs upstream into agentic systems, where software now carries out whole pieces of work on its own. Someone writes the brief, decides what the system may touch, when it must stop, and who signs the result. Accountability is built at that commissioning moment or not at all. Dan Davies has a name for the alternative: organisations build accountability sinks, structures that exist so that nobody can be asked “what did you think, and why?” An ungoverned system inside a critical process is the most efficient accountability sink ever built.
The regulation is catching up fast. The AI Act’s answer, when its high-risk rules bite from December 2027, is the same: oversight has to sit with people who have the competence, training, and authority to intervene. The law now distinguishes between a human who is present and a human who can act. I read this as Europe's bet on AI. The race for the biggest models runs elsewhere, on deeper capital and compute; Europe is treating trust as infrastructure, and the prize goes to whoever deploys best inside real services, under real rules. The override question is precisely what a regulator will ask to see.
Something is lost here. The huisarts kind of trust — slow, personal, carried in one person across years — doesn't scale into these systems. Institutions that pretend it does end up with neither kind. The deliberate move is to choose: where a personal relationship still has to carry the trust — the news that needs a human voice — and where a legible structure is enough on its own. Most organisations are making that choice right now by default, without noticing.
Try this this week: pick one AI-assisted decision that matters, and ask how your system is earning your trust, and what happens when a person disagrees with it. If someone can name the path, the person who takes it, and what it costs them, you have structure. If the answer is a shrug, you've found where to start.
Sources: NEJM AI — randomized study, AI and perceived legal liability for radiologists, Radiology Business — ECRI 2026 patient safety report, Five reasons radiology AI has a trust problem, EU AI Act timeline, Davies — The Unaccountability Machine