Find the Ceiling Before You Tune

There's a particular kind of performance work that feels productive and isn't. You change a setting, run a benchmark, see a number move, change another setting and keep going. Every change teaches you something. Very few of them move you toward the goal, because you don't yet know where the ceiling is.

I did a version of this recently on the cloud worker behind the clinical data pipeline I've been building. The goal was clear: sustain about 1,000 files a minute through the full cleaning and conversion path. The worker was a containerized Python service on a managed serverless platform, fed by a task queue.

The most important thing I learned wasn't in the code. It was a single quota number I should have looked up on day one.

The arithmetic I should have done first#

Here's the calculation, in full:

regional CPU quota      = 200 vCPU
CPU work per file       ≈ 2 seconds
theoretical max         = 200 ÷ 2 = 100 files/second
                        ≈ 6,000 files/minute

That's the ceiling. Not a target, a ceiling. You will never beat it without either making each file cheaper or getting more quota. Everything else (overhead, ramp-up time, I/O, database pressure) only takes you further below it.

Against that ceiling, 1,000 files a minute is a reasonable target with room to spare. It also tells you something useful: if you're stuck well below 1,000 and the CPU isn't saturated, your problem is somewhere else, and adding instances won't fix it.

I found the quota the hard way. I tried to deploy a configuration that asked for more instances, and the platform rejected it. The error message was clear: requested 320 vCPU, allowed 200. Fifty instances at four vCPUs each is exactly 200, and that was as high as it would go.

Five minutes with the quotas page would have saved me a lot of guessing.

What to know before tuning anything#

Since then, I start every throughput effort by writing down a short list.

1. The hard ceilings. CPU quota, memory quota and instance limits per region. Rate limits on the queue. Connection limits on the database.
2. The per-unit cost. How much CPU time one unit of work takes, warm, measured end to end and broken down by phase.
3. The theoretical max. Ceiling divided by per-unit cost.
4. The target. A sustained number, well under the theoretical max, that would make the product work.
5. The current number. Measured the same way the target is defined.

Only then does tuning make sense, because now every experiment can be judged against a known gap.

Measure the per-file cost honestly#

Per-file timing turned out to be more interesting than I expected. For a typical warm file, end to end took about 2.6 seconds:

• Downloading the input from storage: a fraction of a second
• The pipeline itself: about 2 seconds of CPU
• Uploading the outputs: about 200 milliseconds

Inside those 2 seconds, parsing and validation took the biggest share. Converting to the analytics format and writing to the warehouse came next. Building the FHIR bundle and cleaning were smaller.

The first file on a fresh instance took several times longer, because it paid for warming up large caches. That number matters a lot for burst tests, and hardly at all for sustained load.

Python's GIL is a capacity constraint#

The next lesson was about concurrency.

The pipeline is CPU-bound, and it's written in Python. In standard Python, threads inside one process don't run CPU-heavy work in parallel. They take turns. Threads help you overlap waiting, like network and disk, but they don't multiply CPU throughput.

Early configurations ran several files at once as threads inside one process. That didn't make anything faster. It made each file take several times longer in wall-clock time, which was enough to push some requests past their timeouts. The logs showed long gaps nobody could explain, until it was clear they were threads waiting their turn.

Real parallelism comes from separate processes or separate instances. So the obvious next move was to run several worker processes per instance.

That caused out-of-memory crashes.

The cache that got copied four times#

The pipeline keeps a large in-memory cache of vocabulary concepts, millions of rows, because mapping codes to standard terms depends on it. One copy per instance fits comfortably. Four worker processes per instance meant four copies, and that blew straight through the memory limit.

At first I suspected a different cache: a set of HL7 reference tables. It turned out to be tiny on disk and already cached per process. The vocabulary cache was the real cost, and only measuring live memory made that obvious.

The configuration that worked was almost counterintuitive:

• Four vCPUs per instance, for headroom
• One worker process per instance, so the big cache is loaded once
• One request at a time per instance, letting the queue handle fan-out
• A small number of warm instances kept running, to reduce cold starts
• A readiness check that keeps traffic away from an instance until its caches are warm

I also tried the opposite: one vCPU per instance with one request each. It was worse, with lower throughput and more rejected requests.

Burst tests lie#

Another trap: short burst tests under-report what the system can do.

In 30-second bursts, throughput plateaued at around ten instances. It looked like a hidden limit. It wasn't. The autoscaler simply couldn't add instances fast enough through cold starts in half a minute. Requests that arrived before new instances were ready got rejected.

The real production path is a task queue that keeps sending work for minutes and retries rejected requests. Under sustained load, the platform scales much higher than a burst suggests.

The rule I use now: measure with sustained runs, and treat burst numbers as a floor, not the ceiling. My benchmark script retries rejections the same way the queue does, so it measures what production will see.

The quiet second bottleneck#

Once the worker is scaled out, other parts of the system start to feel it.

Every file made a handful of small reads from the main application database for values that don't change during a run, like organization settings and rule definitions. At fifty instances, that's a lot of identical queries hitting a small database instance. It was a likely secondary bottleneck and a real risk of running out of connections.

The fix is caching those values per process, keyed by organization and file type, so each run reads them once. If that's not enough, the next steps are a larger database tier or a connection pooler. Either way, the lesson is the same: when you scale one part of a system, look at what it talks to.

Faster deploys make faster learning#

A big share of the progress came from something that isn't throughput at all. Deploys were slow, five to ten minutes, because every build reinstalled every dependency.

Splitting the container image into two layers fixed that. A base image holds dependencies and is rebuilt only when they change. A thin code image sits on top and builds in about a minute. Routine changes went from edit to live in about a minute and a half instead of up to ten.

That changes how you work. When each experiment costs ten minutes, you run a few a day and guess between them. When it costs ninety seconds, you can measure your way forward.

What I'd do first next time#

If I started this work again tomorrow, the order would be:

1. Look up every quota and limit in the path, and write them down.
2. Measure warm per-unit cost, broken down by phase.
3. Calculate the theoretical max and compare it to the target.
4. Make deploys fast before tuning anything.
5. Benchmark with sustained load that retries like production.
6. Check memory per process before adding processes.
7. Look at the database the moment the worker scales out.

And at every step, make sure faster doesn't mean different. The speed work on this pipeline ran behind a strict output-equivalence gate, which I wrote about in Faster Runtime, Identical Verdict.

The takeaway#

Most performance work starts with tuning because tuning is fun. It should start with arithmetic. Find the ceiling, measure the cost of one unit of work and do the division. You'll know immediately whether your problem is code, configuration or a number on a quotas page, and you'll save yourself a lot of turning knobs.