Evidence cuts both ways

Every AI guardrail I have read treats the human as ground truth. Approval is the terminal state: a person said yes, therefore it is so. The machine is the thing under suspicion; the human is the instrument that resolves the suspicion.

That is backwards in one specific way. The human is the least verifiable component in the loop. When I type "I tested the hardware", that sentence has the same epistemic status as anything the model produces — a claim, with no evidence attached. We built an elaborate apparatus for the model's claims and left the human's entirely alone.

Three things happened while building ours that changed how I think about this. The third one cost me something.

The AI asked me for a photo

I was working through a hardware signing flow and told the agent I had already tested the device. It went looking for that. No test document, no recorded run, nothing in the audit chain — my claim resolved to nothing at all. So it declined to continue, and asked me to photograph the device showing the state I said I had verified.

I took the photo. Then it carried on.

The uncomfortable part is that I was not lying. I had tested it. But "I tested it" and "there is something a second person could check" are different objects, and only one of them survives contact with an auditor. The system did not accuse me of anything. It observed that a load-bearing premise had no resolvable basis, and asked for one.

There is a line in our paper — a basis that cannot be checked is a checkbox — written as a constraint on declarations. I did not expect to be on the receiving end of it within a week of publishing.

The six-second review

The second one is less dramatic and matters more.

I had to manually review a code change and the quality of its test documentation. I looked at it and said it was checked. Six seconds, start to finish. The agent did not accept that as a review. It raised what it wanted: run the thing end to end from the business scenario, and come back with the result.

No policy of mine said that. I had never written "a review must take longer than N seconds", and I would not know how to write a useful version of that rule. The requirement came out of the interaction — the claim I made was not commensurate with the time it took to make it, and the agent asked for something that would be.

I want to be careful about the mechanism, because overstating it is exactly the failure this whole system is about. I am not claiming the software learned a policy and stored it. What I can say is what was observable: the bar went up without me writing a rule, and it went up in the right direction.

My delivery got slower. That is the honest cost and I am not going to launder it into a productivity number. What I got back was different in kind: I stopped shipping things I merely believed were fine.

The contact I said I didn't have

The third one is the one I would rather not write down.

I wanted to work with a particular company. Good fit, the kind you do not get many of. My marketing agent asked whether I knew anyone there. In New Zealand that is not a pleasantry — without a warm introduction the odds of anything happening at all are close to zero.

I said no. I had gone through their people on LinkedIn some weeks before and had not seen a familiar name.

So the agent did the only thing my answer left available and planned a cold approach: the strategy with the lowest expected value in the market I actually operate in. It went nowhere, exactly as it should have.

Later I gave the marketing agent the same evidence gate the development agent already had. I expected it to be an inconvenience — a governance tax on the side of the business that is supposed to move fast. The first thing it did was decline to accept my answer, and ask me to screenshot the search. It did not treat "no contacts there" as a finished fact. It treated it as a claim someone had asserted, about a search it could not see.

I ran the search again. A former colleague was there, working on AI. He is the person who pointed me at generative models in the first place, and the reason I stayed with them when it was not obvious that I should. I have not found a good way to describe what it felt like to learn that from a screenshot I had been made to take.

Here is the part worth extracting, because it is not about carelessness. "I checked and there is nobody" is two claims wearing one sentence: that I ran a search, and that the search I ran is the search you are asking about. The first was true. The second was not — I had read those same profiles weeks earlier while thinking about org structure and open roles, so I had looked at the right page with the wrong question in my head. Memory stored the conclusion and discarded the query.

And from the inside, those two feel identical. No amount of honesty on my part could have surfaced the difference. The agent had no way to see it either; it received a fact and reasoned correctly from it.

The gate I resented would have cost me about ninety seconds. Not having it cost me the introduction. A screenshot is not a formality — it is the search re-run with the current question in mind, which is the only version of it that was ever worth anything.

Why constraining the human makes the model better

This is the part I did not anticipate, and it is why all three belong in the same piece.

An agent's context is mostly assertions, and a large share of them come from the human. When I hand-wave, that hand-wave enters the context as a fact, and everything downstream is computed on top of it. The model then reasons impeccably from a premise nobody checked, and the output is confidently wrong in a way no output-side hallucination filter can catch — because nothing in the output is unfaithful to its input. The falsehood was upstream, and it was mine.

So the discipline has to be symmetric, or it does not work in either direction. Hold the human to resolvable evidence and the model's premises get cleaner. Hold the model to resolvable evidence and the human stops being able to launder a guess through it. In an enterprise, the moment a person is in the loop at all, the loop is only as grounded as its least grounded participant.

The harder question: does the decision reach reality

Grounding is not the whole problem. A perfectly grounded decision still has to cross from a conversation into the world, and that crossing is where the assurance usually evaporates.

The common failure is not that nobody approved. It is that approval was collected in a form that establishes nothing — a click on the same machine the action came from, on a card that looks exactly like the forty-nine harmless ones before it. Ask an operator to decide how much scrutiny something deserves and you have handed the decision to the person being interrupted, who has neither the evidence to make it nor any incentive to choose the slower path.

So the weight is decided from the action itself, before the card is drawn:

The last row is the one that matters. If an agent's next move is to edit the policy that governs it, no amount of approval collected on the compromised machine means anything. It has to be read on a screen that machine does not control. That is the whole argument for hardware in this stack, and it is also why a click cannot settle the top three rows no matter who clicks it.

Watching that sequence run — four commands, four different keys, decided by the commands and not by me — is our demo. It is the shortest honest answer to "what does this actually do".

One take, on a phone. The commands really execute; the targets are throwaway files. · Watch in higher definition

The model underneath is published

The Chain of Custody: a decision-level model for AI governance (CC BY 4.0), a companion to The Proofpane Architecture.

It decomposes a governed decision into the successive re-statements it passes through — declaration, request, proposal, display, comprehension evidence, authorising response, disposition — and asks which hand-off each control attaches to. Two claims follow. Placement bounds what a control can establish, which turns several common assurance over-claims into locatable category errors rather than matters of degree. And display integrity and recorded comprehension are consecutive, non-substitutable positions: a system can prove the right thing was shown while establishing nothing about whether it was taken in.

It is a conceptual model, not a validated theory, and it says so in its own abstract. Its challenge conditions have not been run.