Faster Runtime, Identical Verdict

There's a moment in every performance project when you find something that looks like waste. A validator that runs on every record but doesn't seem to block anything. A second parse of the same message. A check whose output nobody reads.

Delete it, and the pipeline gets faster. The benchmark looks great. Everyone's happy.

Then, weeks later, someone notices that the quality scores shifted. A batch of records that used to fail onboarding now passes. Nothing crashed. The pipeline simply started giving different answers, and nobody noticed because the only thing anyone was measuring was speed.

In early June I spent a focused stretch making the clinical data pipeline I've been building faster. The rule I wrote at the top of the plan, before any code changed, was four words long:

Faster runtime, identical verdict.

What "verdict" means here#

Every record that goes through the pipeline comes out with more than converted data. It comes out with a judgment:

• Validation results: which rules passed, which failed and how badly
• Conformance findings: how well the source matched the expected message structure
• Mapping coverage: how many coded facts mapped to standard vocabularies
• Scores per quality dimension, rolled into an overall score
• The converted rows themselves

That judgment drives real decisions. It decides whether an organization's data is good enough to turn on automation. It decides what gets flagged for review. It's what people trust when they decide to use the data.

So "the pipeline got faster" is only good news if all of that stayed the same.

Golden files#

The mechanism is old and simple: a golden set.

A golden set is a fixed collection of input files with known, saved outputs. You run the current version of the pipeline over the set once and save everything it produces: records, validation results, scores and converted rows. That's the baseline.

Then for every change:

1. Run the candidate version over the same inputs.
2. Normalize the outputs, stripping fields that are supposed to change, like timestamps and run IDs.
3. Diff candidate against baseline, file by file.
4. Green means every file is identical. Anything else is red.

My golden set for this work was 64 records, chosen to cover the formats and edge cases that matter rather than to be large. Each speed change ended with the same line in its notes: golden diff 64 of 64, green.

In pseudo-code, the gate is almost boring:

for f in golden_inputs:
  old = load(baseline, f)
  new = run(candidate, f)
  if normalize(old) != normalize(new):
    fail(f, diff(old, new))
pass()

The power isn't in the code. It's in the habit of running it every time, without exceptions, and treating red as a stop.

Two kinds of speed work#

Once the rule was in place, the work split cleanly into two categories.

Infrastructure churn. Work that changes how something is computed, but not what is computed. Examples:

• Caching a large table of HL7 reference definitions per process, instead of reloading hundreds of small files for every record. This was the single biggest win.
• Reusing one pipeline object across a batch, instead of rebuilding it for every file.
• Caching cloud credentials and clients per process, instead of creating them per file.
• Moving a slow, non-critical outbound call off the hot path into a background thread with a timeout.
• Batching per-record database and warehouse writes.

These changes are usually safe, and the golden diff confirms it quickly.

Signal changes. Work that might change what's computed. Examples:

• Skipping a validator because it "only produces warnings".
• Removing a parse pass that looks redundant.
• Shrinking a cache of vocabulary concepts to save memory.

These are where performance work turns into a regression. They aren't forbidden. They have to prove they're equivalent first.

The validator that looked like cruft#

The clearest example was a profile validator.

On every HL7 message, it re-parsed the raw text and checked the message against an expected profile. It was labeled advisory. It didn't block anything on its own. Profiling suggested the parse step walked each message several times, and this validator was one of the walks.

The easy fix was to turn it off.

Before touching it, I traced where its findings went. They landed in the validation results, and the validation results fed the scoring. Specifically, the profile findings counted toward the format dimension of the quality score. Turning off the validator would have quietly raised format scores across the board, which would have moved onboarding decisions.

So the change went the other way. Instead of skipping the validator, I kept it always on and removed its redundant parse. The parser now tokenizes the message once, and the validator receives the already-split tokens instead of re-reading the raw text. The old entry point stayed as a compatibility wrapper.

The golden diff came back identical. Same findings. Same scores. Less work.

That's the pattern I want people to take from this: the waste is often the redundant work, not the check itself.

When "off" is OK#

Some things can be turned off, once proven.

There was a hierarchy-building step that produced a nested view of each message. It turned out nothing downstream needed it for scoring or output. The proof was a pair of golden runs, one with the step on and one with it off. Both diffs came back green, 64 of 64. Only then did the default change, behind a documented flag.

The rule for flags was strict:

• Default to today's behavior. A new flag starts in the position that changes nothing.
• Flip it only with evidence. Equivalence on the golden set, written down where the next person will find it.
• Make it a decision, not a side effect. Changing what gets validated is a product decision about quality, not a performance tweak.

Things that stay authoritative#

A few things were declared off-limits for "optimization" up front:

• The vocabulary cache keeps full coverage. It's big, and it's tempting to load less of it. But mapping correctness depends on it. The work was to make it cheaper to hold and faster to warm up, never smaller, unless a mapping-parity test proved equivalence.
• Validators that feed scoring are load-bearing. If a check's output reaches a score, it isn't cruft.
• Required verification stays required. FHIR validation on sampled records is part of the onboarding gate, so it had to run. It just had to run off the throughput-critical path.

Writing these down before starting saved a lot of debate later. When a tempting shortcut came up, the answer was already on the page.

Live checks after the diff#

The golden set proves equivalence on known inputs. It doesn't prove the deployed system is healthy. So every change that shipped also got a short live check:

• Out-of-memory events in the last hour
• Crash-loop or version errors
• Rate-limit rejections, and whether the queue recovered them
• A real end-to-end run through the product, with an output-equivalence diff

All zeros, or a known small number that recovered, before moving on.

What it cost#

This discipline isn't free. Each change took longer than "just make it faster" would have. Some promising ideas got set aside because proving them equivalent would have taken more time than they saved.

It was worth it. At the end of a long run of performance changes, I could say something I rarely get to say about speed work: every one of them is proven not to have changed the answers. The scores people rely on mean the same thing they meant before.

Beyond clinical data#

The same pattern works well outside clinical data:

• Refactors. A large cleanup is a performance change for humans. A golden set of API responses or rendered pages catches the moment a refactor changes behavior.
• Agent-written code. When an agent proposes an optimization, a golden diff is a natural gate to add alongside tests. It fits naturally with the staged loops I described in The Art of the Rock Tumbler.
• Model changes. Swapping a model or prompt behind a feature deserves the same treatment, with a fixed evaluation set and a diff that a human reviews.

The takeaway#

Speed is easy to measure and easy to celebrate. Correctness is quieter, and it's the part people actually depend on. A small golden set, a strict diff and the rule "faster runtime, identical verdict" let me make a pipeline meaningfully faster without asking anyone to trust it again from scratch.