Why I Run My Own Model Fleet

Most of the agent work in my studio runs on machines I can reach out and touch. Starship, the system I've been building to plan, write, review and ship code with agents, sends its model calls to a small fleet of Apple Silicon Macs on a shelf instead of a cloud API.

That wasn't nostalgia, and it wasn't a stance against the cloud. I wanted to understand the whole stack well enough to make it run on its own. Owning the inference layer was the fastest way to learn where it breaks.

The fleet, in plain terms#

The hardware is modest and mostly already owned:

• A coordinator. An M4 Mac mini with 24 GB of memory. It runs the job ledger, the model gateway and the run engine, and deliberately serves no models.
• Two workers. M4 Pro Mac minis with 64 GB each. These carry the bulk of the coding work on a 35B mixture-of-experts Qwen model.
• A heavy runner. An M2 Ultra Mac Studio, also 64 GB. It's reserved for escalations and the hardest tasks, one slot at a time.
• A few 8 GB Mac minis on standby for overflow. They aren't on the critical path today.

Workspaces follow the same instinct. Early plans leaned on a hosted sandbox service. I replaced it entirely with a container design on my own hardware: pre-baked images per product, a warm pool, and a target of a workspace ready in under three seconds warm and under five cold. In practice the small minis are offline right now, and previews run in containers on one of the workers.

A gateway, not a zoo of clients#

Every agent asks for a role, not a model. A coder asks for something like "coder with tools", and the gateway decides which endpoint serves it. I built the gateway directly on the Vercel AI SDK's provider adapters instead of adding another proxy layer. The registry pattern is borrowed from an open-source project I studied: a JSON manifest plus a getModel(slug) lookup.

Two rules came out of real failures:

1. An endpoint has to prove it can do the job before it can take traffic. Each model host runs a small registrar. It only advertises a role after the model shows up in the model list and passes a native tool-call probe. A broken model can't quietly take the lane.
2. A route name is not a model. In the early version, the gateway simply took the first registered endpoint for a role. A weaker local model also advertised the coder role, so it kept winning. It emitted tool calls as plain text, and the runs failed in ways that looked like the agent's fault. The fix was removing it from that role. Later, the gateway learned to pick the least-loaded endpoint instead of the first one.

When local actually beats cloud#

I keep four reasons in mind, and I try to be honest about each.

Cost. For the cloud surface around Starship, my definition of done was no more than $30 a month on top of what I already paid. The local hardware is sunk cost, and I count power against it.

Control. When something goes wrong, I can see all of it: the context window, the memory limit, the idle timeout, the parser. Inference that runs on my hardware is a system I can fix rather than a ticket I file.

Latency, once the network is fixed. Switching one tunnel from QUIC to HTTP/2 took public requests from 8–30 seconds down to about 0.2. Co-locating the model with the API removed the network from the hot path entirely. Long generations also ran into a tunnel provider's edge timeout of around 100 seconds, so long-running model calls now travel over a local tunnel instead.

Privacy of the work itself. Inference stays on hardware I own. Tracing is a different story. My observability moved from a self-hosted instance to a hosted service, and the spans are deliberately sanitized: no tokens, no secrets, no raw prompt bodies.

The lessons the hardware taught me#

Most of what I learned wasn't about models. It was about the boring layer underneath them.

Defaults will quietly sabotage you. The hard coding tasks kept looping. The agent would read a large file, decide it was truncated, and read it again. When I inspected one request, it carried 48 messages and roughly 60,000 tokens. The model had seen 4,806 of them. Ollama had silently cut the prompt down to its default context window and kept only the tail. That meant the system prompt, the task contract and the code snippet I'd carefully placed at the top were all gone. The OpenAI-compatible endpoint ignored the per-request setting, so the fix lived in the model definition itself:

FROM qwen3.6:35b
PARAMETER num_ctx 32768

That also explained why simple tasks succeeded. They finished in few steps, so their history fit inside the small window.

Memory is a policy decision. Upgrading to the newer 35B model meant upgrading Ollama, raising the GPU's wired-memory limit on the worker (and making that survive reboots), and turning on flash attention with a quantized KV cache. After that, the model still evicted itself after about five idle minutes. Reloading 24 GB on the next request produced timeouts that looked like model failures. Now the model is pinned resident.

More endpoints is not more capacity. I spent a day trying to push a batch of enhancement work faster.

• Decode was already fast, around 56 tokens per second on the 35B model.
• Ollama runs this model family with a single parallel slot, so "three in flight" was cosmetic.
• Matching concurrency to the real slot count took a 10-item batch from 983 seconds to 744, about 0.81 items a minute.
• Spreading both workloads across every box made it worse, at 854 seconds, because large calls blocked small ones.
• A separate fix turned capacity errors into waits instead of failures. That took one batch from 8 failures in 10 to none.

The honest ceiling on four dense slots is about two items a minute. My stretch goal of 500 items an hour needs more machines or a smaller, faster model. It doesn't need cleverer routing.

Where I still use the cloud#

Local-first is a direction, not a purity test.

• A hosted model stays in the registry as an off-by-default break-glass route.
• When I audited the planning side, I found that endeavour briefs and task breakdowns still defaulted to a hosted model. Some of those calls bypassed the gateway, so they weren't even attributed. That's on the list to fix, and I'd rather name it than pretend the system is fully offline.
• Model capability is real. A subtle security fix once defeated the 35B model even with perfect context. It finally went green only after I fixed how it read large files, broke its re-read loops, and handed it the scanner's own remediation guidance.

Fine-tuning is the last lever, not the first. There's a LoRA path that exports redacted, training-eligible runs, trains an adapter, and registers it in shadow. Its proof run decided to keep the adapter in shadow, which is exactly what it should do when it doesn't beat the active route.

The takeaway#

Running my own fleet didn't make the agents smarter. It made me more precise. Every failure had a name I could find: a context window, a memory cap, an idle timer, a route that pointed at the wrong model. I'd take that over a faster model I can't see inside.

The point was never to avoid the cloud. It was to understand the system well enough that it can run without me watching it.