A technique ahead of its theory

Alchemy that works

In December 2017, accepting an award for a paper that had stood the test of time, Ali Rahimi told the field’s main conference that machine learning had become alchemy. He was careful about it: “Alchemy worked. Alchemists invented metallurgy, ways to make medication, dying techniques for textiles, and our modern glass-making processes.”

Joel Mokyr, who shared the 2025 Nobel prize in economics, explains why that is not enough. A technique sits on top of an understanding of why it works, which he calls its epistemic base. A technique with a narrow base improves by trial and error, and cannot tell a dead end from a temporary obstacle.

By Mokyr’s test, today’s language models are a narrow-base technology, and the tell is the thing the industry is proudest of. Scaling laws predict how a model’s error falls as it grows, and the predictions hold. But a fitted curve is not an account. It tells you what the error will be and not why, which is the standing of Boyle’s law before anyone knew what a gas was.

For a firm deploying agents, that is the practical problem. The vendors cannot tell you why their systems work, so the firm has to know what to check, and the theory for that already exists. It was worked out by the Cyberneticians, for machines that adapt and the organisations that run them.

The argument in full is in Call it entropy, on rsorourke.com.

A recursive intelligence-amplification prosthesis

Ashby defines intelligence: appropriate selection against the firm's goal. Beer identified a diagnostic model of the firm: five systems in a recursion, so the regulator’s rule, the board’s decision, a manager’s brief, a member of staff’s check, an agent’s task and a sub-agent’s sub-task are one problem at every level, coordinated downward through as many levels of agents as the work needs and upward to the regulator, whether a statutory one or the industry’s own. Pask says how it is used: as a conversation between people and agents at every level, in which the people can teach back what the agent proposes. That capacity to teach back is the evaluative capacity the practice builds, and it has to exist at every level, because the prosthesis fails at whichever level it is missing.

IA, not AI: the firm’s own intelligence amplified, and measured in the outcomes it is already judged on. The three turn the firm’s own context into the thing its agents are selected against, and that is where the return on an AI deployment comes from.

The research underpinning this practice lives at rsorourke.com.

The shape of an engagement

Whichever shape the work takes — a diagnostic audit, a build sprint, or an embedded retainer (see About for the shapes) — the engagement moves through four phases.

1. Frame

Pin down what problem we're actually solving and whether AI is the right answer. Usually one to three calls plus a short written diagnostic. Half the time this is where we stop — because the right answer isn't AI.

2. Prove

Build the smallest version that's still real. Working code, against real data, on a real workflow. Two to four weeks. If it doesn't survive contact with reality, we kill it here — before anyone's signed off a roadmap based on a demo.

3. Build

Take what worked from Prove and harden it into something your team can run. Add the evals, the monitoring, the human-in-the-loop, the rollback path. Four to eight weeks.

4. Run

Stay close while it's in production, until it's stable enough that we're not needed. Usually a thinning retainer over three to six months. You shouldn't need us forever.

What you'll actually have at the end

Different consultancies leave very different things behind. Here's what's in the box at the end of an engagement with us:

A written diagnostic. Your problem, the options considered, why we chose what we chose, and what we explicitly didn't do.

Working code in your repos. Owned by your team. No black boxes. No "consultant tech" you have to license.

An evaluation suite. So when the model changes, the prompt changes, or the data drifts, you'll know before your clients do. The reasoning is in How much should you check the AI’s work?.

Monitoring and alerting. Hooked into whatever you already use. Production AI is mostly observability.

A handover document. Written so the engineer who picks this up after us can actually pick it up.

Optional team sessions. If you want your engineers or partners to be able to extend the work.

Why a boutique

Three common alternatives. Why none of them, for most of the work we do:

A Big-4 consultancy. You won't get the partner you met in the pitch — you'll get whichever associate has the cheapest billable rate. The strategy will be solid and the execution will go to an offshore team. Fine for some kinds of work. Not this kind.

Hiring a senior ML engineer. If you need one for the next year, hire one. If you need four to eight weeks once or twice a year, a permanent hire is the wrong shape — and a junior hire is worse.

A generic AI agency. Most agencies will take your brief and route it to whoever's free. We won't take the work if we're not the right people for it, and you won't be on a roster.

Most often, though, the thing we're replacing is doing nothing — because you didn't know who to call.