Nothing Goes Live Because It Exists

Here's a failure that's easy to fall into. You save a new version of a prompt to try it out. The lookup grabs the newest version. From that moment, every run uses it, and nobody actually decided that. If it's a little worse, it can take a while to notice.

That's the failure mode of any system with many swappable parts. Agent setups have a lot of them:

• Prompts for each role
• Model routes, meaning which model handles which kind of work
• Harnesses, meaning which tool or runtime drives the model
• Skill packs and reference material
• Evaluators that score the work
• Fine-tuned models, once you get that far

Each one can be changed independently. Each change can make things better or worse. And by default, most systems treat "the newest version" or "the one that exists" as the one to use.

The rule I wrote at the top of the plan to fix this was short: nothing moves to active because it exists.

Five states#

Every asset in the rig now has a lifecycle state:

• Shadow. The default for anything new. It exists, it can be tested, it can run side by side with the real thing for comparison. Production never uses it.
• Review ready. It has bench evidence attached. Someone can look at the numbers and decide.
• Active. Production uses it. Getting here requires an explicit promotion by a person, plus evidence.
• Paused. Stops all new use immediately, without deleting anything. Old runs that used it still make sense.
• Deprecated. Retired, with a reason. Old traces that reference it stay readable.

And the transitions have rules:

new asset            -> shadow
shadow -> review_ready   requires: evidence links (bench runs)
review_ready -> active   requires: evidence + a human operator
active -> paused         allowed any time, audited
paused -> active         not allowed; go back through review_ready
any -> deprecated        requires: a reason

Two of those rules carry most of the weight. Only a person can make something active. And you can't un-pause straight back to active, because the reason you paused it might still be true. You go back through review with fresh evidence.

Where the state lives#

There were two reasonable places for this, and I ended up using both. Static definitions, like what a prompt says or what a route points at, live in code and config, where they're versioned and reviewed like everything else. The lifecycle state, meaning which version is actually active right now, lives in a database registry.

Each registry entry records the asset type and ID, its version, its state, an owner, the scope (a specific product or role, or global), links to the evidence, and when it was created, promoted or deprecated. Every transition writes an audit row.

That split matters. Code review answers "is this change well written?". The registry answers "has this change earned production?". They're different questions, and mixing them is how an unreviewed experiment ends up live.

Runtime reads active only#

The registry is only useful if the runtime respects it. So every lookup that picks a prompt, route or skill filters to active assets by default.

Shadow assets are only included when a run explicitly asks for a shadow comparison. That's how I test something new against the real thing: same task, both versions, compare the results. The new version gets a fair trial without ever being the one that matters.

When a run is queued, its payload carries the resolution rules it used: which states were allowed, whether shadow comparison was on and which labels were protected. If I'm looking at a strange result later, I can see exactly which versions of everything produced it.

Separate the role, the harness and the model#

A related lesson came from model routes. I used to think of a route as "the model we use for coding". That bundles three different things:

• The role. "Local coder" is a job description.
• The harness. The tool or runtime that drives the model for that role.
• The model. The specific weights behind it.

Any of the three can change without the others. A new model version can slot into the same role. A different harness can drive the same model. Keeping them as separate fields, each with its own lifecycle state, means I can promote a new model without pretending the role changed, and roll it back without touching anything else.

Check before you spend#

Lifecycle states stop experimental assets from going live. A second set of checks stops broken ones from running at all.

Before a run is queued, a preflight checks the route it's about to use:

• Does the route exist in the registry?
• Is the endpoint reachable and fresh?
• Is the model behind it actually the one the registry says?
• Does it support tool calling, if the task needs it?
• Is there a timeout set that fits the workload?

If any check fails, the run doesn't get queued. It fails right there with a clear error, instead of half an hour later with a confusing one.

Tool calling gets extra attention, because it's where models fail in the most creative ways. A small torture suite feeds each route the awkward cases: duplicate tool names, required tools, arguments that arrive in pieces, malformed JSON, very large arguments and a model that answers in plain prose instead of calling a tool at all. A route that can't handle them doesn't get used.

For routes that cost money, there's one more rule: cloud routes fail before spend. A cloud route can't even be registered without a per-task token cap, a per-project spending cap and a defined behavior when the cap is hit, which is escalate to me, not keep going. Local routes can fail closed cheaply. Cloud routes have to fail before the meter starts.

The proof that it doesn't promote itself#

The best test of this system was fine-tuning.

I built a path to train small adapter models from good agent trajectories. It exports clean training examples (only from redacted, approved runs), picks a training stack, produces an adapter and registers it as a new model route, in shadow. Then it benches the shadow adapter against the active local route and the other active lanes on pass rate, reviewer pass rate, tool-call validity, cost and speed.

On the first end-to-end proof, the decision came back: keep shadow.

That was the result I wanted. The adapter was trained on a tiny proof dataset and had no business being promoted. The important thing was that the path produced a clear, recorded "no", and that nothing about training a new model made it live. Promotion stays a separate step, taken by a person, with evidence, after a real bench and a live canary.

How it fits the rest of the rig#

This lifecycle is one stage in the same polishing loop I described in The Art of the Rock Tumbler: rough cut, shape, polish, prove, then promote only through explicit gates. It's also what makes running my own model fleet manageable. With several models, harnesses and prompts in play, I need to know at any moment exactly which combination is live, and that it got there on purpose.

The takeaway#

Systems with lots of swappable parts drift. The newest version wins by default, and nobody notices until quality slips. Give every part a lifecycle, start everything in shadow, require evidence and a human for promotion, and make the runtime respect it. Experiments become safe to run, rollbacks become boring and "what's actually live right now?" always has an answer.