The Art of the Rock Tumbler: Polishing Loops for Code and Agents

If you've ever owned a rock tumbler, you know it's a patient machine. You load rough stones, water and coarse grit into a barrel and let it turn for days. Then you wash everything, check the stones, and move to a finer grit. Then finer again. Then polish. Skip a stage, or let grit from one stage leak into the next, and you scratch everything you've done.

I keep coming back to that machine because it's the best model I've found for building systems that improve themselves. Inside Starship, my agentic infrastructure, the retry and quality-gate layer is literally called tumble. The planning documents open with the same principle: every run polishes the system, mistakes are only OK once, learnings persist, and the next run should be better than the last.

This post explains the metaphor and the machinery behind it, and why I think most agent loops get retries wrong.

One pass is never enough#

The first instinct with a coding agent is to treat it like a vending machine. You put a task in and a pull request comes out. When the output is wrong, you either accept it or throw it away.

That works for small tasks. It falls apart on anything real, for the same reason one long tumble with coarse grit never gives you a polished stone. Different defects need different treatments, in a specific order:

• Coarse grit removes the big problems: the change doesn't compile, it edits the wrong files, or it doesn't change anything at all.
• Medium grit handles correctness: the tests fail, the types are wrong, or the behavior doesn't match the task.
• Fine grit handles quality: standards, naming, documentation and the reviewer's notes.
• Polish is proof: the real CI suite is green against the latest integrated code, and the change is safe to merge.

If you ask a reviewer model for style feedback on code that doesn't compile, you're polishing a rough stone. You waste the expensive stage on problems a cheaper stage should have caught.

The formation: rough cut, shape, polish, prove, promote#

The tumbler isn't only a runtime loop in Starship. It's how I plan the work. Every action plan in the latest phase has to name which stage it serves and what evidence lets the next stage begin. There are five stages:

1. Rough cut. Build the minimum infrastructure that makes the idea exist. One adapter, one probe, one job.
2. Shape. Wire it into the real workflow, so it runs where the work actually happens.
3. Polish. Add tests, observability, clear failure messages and safety bounds.
4. Prove. Run it against a benchmark and live evidence, not just a demo.
5. Promote. Make it the default only after the gates pass.

Here's how that reads in practice. Fine-tuning goes like this: rough cut the dataset export, shape the training scripts, polish the evaluation and registry, prove it in shadow, and promote only if it beats the active route on the benchmark and in a live canary. Prompt management goes the same way: rough cut a registry, shape labels and rollback, polish few-shot selection, prove it with experiments, and promote only through lifecycle gates.

The value of that formation isn't the vocabulary. It's that "promote" is a separate step with its own evidence. A lot of engineering work skips straight from "it works on my machine" to "it's the default." The tumbler won't let you.

Inside tumble: a bounded ladder#

At runtime, tumble is the deterministic threshold layer between an agent's output and anything that matters. Its rules are deliberately boring.

Every attempt runs through gates in order. The checks are deterministic: lint, types, tests and a secret scan. A standards agent checks the change against the codebase's rules. A reviewer gate and, where it applies, a documentation validator run after that. Each gate emits a named pass or fail event, so a run's history reads like a checklist rather than a story.

When a gate fails, the run doesn't loop forever. It climbs a short, hard-bounded ladder:

attempt   -> gates pass?  -> persist, move on
          -> gates fail   -> retry once, with the failure attached
retry     -> gates fail   -> escalate once, to a stronger tier
escalation-> gates fail   -> park as needs_human, with the full trail

A few details make that ladder trustworthy:

• It's bounded in code, not by convention. The ladder is built as a fixed list: initial, retry, and an optional escalation. There's no way for a clever prompt to add a fourth attempt.
• Infrastructure failures don't use up a retry. If a request dies at the edge with a gateway timeout, that's the network, not the agent. Counting it as a failed attempt would punish the model for my plumbing, so those failures are retried without using up the ladder.
• Escalation means a different grit, not the same grit harder. A retry goes back to the same tier with the failure attached. An escalation moves to a stronger model on dedicated hardware.
• Parking is a result, not a crash. A parked task lands as needs_human with every attempt, every gate result and every tool call attached. The person who picks it up starts with the whole story.
• Review cycles are capped too. A reviewer can send work back, but only a fixed number of times before it escalates.

The policy for a routine reads like a single line of config: escalate after two failures. It's simple enough to reason about at two in the morning, which is the real test of any retry policy.

Wash the barrel between stages#

The detail that ruins the most rock tumbler batches is contamination. One grain of coarse grit carried into the polish stage leaves scratches all over a stone that was nearly finished.

Agent loops have the same problem, and it's sneaky. If a retry inherits a bloated context full of failed reasoning, the second attempt often repeats the first one's mistake. When I traced one of Starship's looping runs, it carried 48 messages and about 60,000 tokens. The model had silently seen only the last few thousand. The task contract at the top was gone. That's grit in the polish barrel.

So the retry discipline is:

• Carry forward the failure, not the whole conversation. A retry gets the task, the contract, the current diff and the specific gate failure. It doesn't get every wandering thought from the first attempt.
• Name the failure. "Tests failed" is coarse. "This test failed with this assertion after this edit" is useful.
• Run the cheap gates first. No model gets called before the deterministic tiers pass. A stronger model shouldn't spend its time on a missing import.

Mistakes are only OK once#

A rock tumbler doesn't learn. Starship has to.

The tumbler principle in the planning docs is that every run, every completion and every failure produces structured learning that feeds back into the system. Concretely:

• Each failure gets a code, not just a log line.
• Lessons from runs are stored as learning records and become candidates for the skill library. They go through the same lifecycle as any other skill: candidate, reviewed, promoted.
• Postmortems turn into tasks. When the system finds rot, such as a broken baseline CI, fixing it goes on the queue as work for the system, not as a quiet hand patch from me.

"Mistakes are only OK once" sounds strict, but it's actually kind. It means nobody, human or agent, should have to rediscover the same failure. The second time something breaks the same way, the problem isn't the agent. It's the loop.

What the tumbler is not#

The metaphor can be stretched too far, so here are the limits I keep in mind.

It's not infinite polishing. A real tumbler has a schedule. Mine has a bounded ladder, cost caps per endeavour, and a hard stop. Unlimited retries aren't diligence. They burn money and hide a broken task.

It's not a substitute for a good stone. If a task is too broad, no amount of tumbling saves it. The planner is supposed to reject or split broad tasks before they ever reach the coder. The tumbler polishes, but it doesn't change what the stone is.

It's not a model-quality fix. Some tasks fail because a local model has reached its limit. The ladder makes that visible and routes it to a stronger tier or a human. It doesn't pretend otherwise.

Using the tumbler outside agents#

None of this is specific to AI. The same shape works with human teams:

• Coarse grit first in code review. Does it build, does it do the thing, is it in the right place? Only then talk about naming.
• Bounded rework. If a pull request has gone back and forth three times, stop commenting and pair on it.
• Separate "works" from "promoted." Feature flags, shadow modes and canaries are tumbler stages for production.
• Write the lesson down once. A short rule in the team's standards beats the same review comment fifty times.

The takeaway#

Polish comes from stages, not from effort. A good loop knows which defects to remove first, washes the barrel between stages, stops after a fixed number of turns, and hands a person the whole story when it can't finish.

When I built tumble, I wasn't trying to make the agents perfect. I was trying to make every imperfection small, named and only allowed to happen once. Over enough runs, that's what makes the stone smooth.