The Stealth Build: Cleaning HL7 and C-CDA Before Anyone Trusts It

For several months I've been working on a project I haven't talked about much. I've been calling it my stealth build: a pipeline that cleans HL7 and C-CDA, the two formats a huge share of clinical data still travels in.

It isn't glamorous. Nobody demos a cleaner HL7 feed at a conference keynote. But almost every promise people make about AI in healthcare runs into the same problem: the data underneath is messy, inconsistent and locally customized, and it's rarely checked before someone builds on it. This post is about that problem, and the approach I've taken to it.

Two formats, two kinds of mess#

Clinical data moves around in a few standard shapes. The pipeline focuses on two older ones and outputs two modern ones.

HL7 v2 is the workhorse. It's a pipe-delimited messaging format that hospitals, labs and clinics have used for decades for admissions, discharges, transfers, orders and results. A message is a stack of segments, each with a three-letter name, and fields are split by delimiters. A made-up example looks something like this:

MSH|^~\&|SENDER|FACILITY|RECEIVER|DEST|202601011200||ADT^A01|MSG0001|P|2.5.1
PID|1||000000^^^HOSP^MR||DOE^JANE||19800101|F
PV1|1|I|WARD^101^A

It's compact and fast, and it's everywhere. It's also loose. Many fields are optional, sites interpret fields differently, and custom "Z-segments" carry whatever a local interface team needed years ago.

C-CDA (Consolidated Clinical Document Architecture) is the document side. It's XML: a whole clinical summary with sections for problems, medications, allergies, results and vitals, each built from templates. It's richer than a v2 message and far more verbose. It has its own failure modes: free text where codes should be, local codes where standard ones should be, and sections that are technically valid but nearly empty.

On the way out, the pipeline produces:

• FHIR R4 resources (Patient, Observation, Condition, MedicationRequest, AllergyIntolerance and others) for application workflows.
• OMOP CDM 5.4 tables (person, measurement, condition occurrence, drug exposure, visit occurrence) for population-level analytics and research.

Why converting isn't enough#

The most important thing I've learned is simple: a direct HL7-to-FHIR conversion often just moves the mess into a newer format. If a lab result arrives with a local code, a unit in a free-text field and a timestamp in the wrong zone, converting it faithfully gives you a nicely formatted FHIR Observation that's still wrong.

So the pipeline's job isn't just conversion. It's assessment, repair and transparency, in that order:

1. Parse and validate structure. Identify the message type, segments, fields, components and repetitions. Flag malformed segments, missing required fields and unexpected usage.
2. Assess quality. Check identifiers, codes, units, timestamps and conflicting values. Note local conventions and custom segments rather than silently dropping them.
3. Clean and normalize. Apply repeatable rules to fix what can be fixed with confidence.
4. Map terminology. Resolve local and proprietary codes to standard vocabularies.
5. Transform. Produce FHIR and OMOP from the cleaned data, not the raw data.
6. Report. Show what came in, what was wrong, what was changed, how confident the pipeline is, and what's still unresolved.

The last step is the one people underestimate. Healthcare teams need to see what's wrong with their data, not just receive a cleaned file. A before-and-after report is how they decide whether to trust the output, and which upstream interface to fix.

Terminology is where the work is#

If I had to pick one hard problem, it's codes.

A single lab test might arrive coded in a hospital's local system, in a vendor's proprietary system, or as free text. To be useful downstream, it needs to land on a standard concept: LOINC for lab tests and observations, SNOMED CT for clinical findings, ICD-10-CM for diagnoses, and RxNorm for medications.

The pipeline resolves codes against the OMOP Athena vocabulary, a standard vocabulary set with millions of concepts and the relationships between them. Concretely, it looks up the source code, follows "maps to" relationships to the standard concept, and falls back to crosswalks for code systems that Athena doesn't map directly.

The first evaluation run taught me how much this matters. One proprietary code system was being mapped through a bad path, which produced thousands of garbage mappings. Measurement mapping, which covers labs and vitals, sat at roughly 20%. After adding a proper terminology resolver and a crosswalk for that code system, a second run a few days later reached:

Domain	Mapped to a standard concept
Measurements	91.3%
Conditions	99.2%
Drugs	90.8%
Overall	92.5% of about 101,000 concepts

That was on a 2,000-record evaluation set: 1,000 C-CDA documents and 1,000 HL7 v2 messages.

One finding from that set is a good reminder that not every gap is a bug. Every HL7 message in it was an admit, discharge or transfer message, which is demographic only, with no clinical observations. The pipeline correctly produced person records and nothing else. Knowing what your source data can contain is part of judging the output.

AI where it helps, rules where it counts#

People ask whether this is an AI product. The honest answer is that it uses AI carefully, in specific places, and rules everywhere it matters.

AI is good at interpretation. It can read a local field label, a free-text value or an odd custom segment and suggest what it probably means and where it should map. It can explain a transformation in plain language and spot patterns across many messages.

What it shouldn't do is silently decide. So the approach is:

• AI suggests and explains. Mapping suggestions come with confidence and a reason.
• Low confidence goes to a person. Uncertain mappings are routed for review, not applied.
• Approved mappings become rules. Once a mapping is validated, deterministic rules apply it every time, the same way, with an audit trail.

That balance is the point. AI makes the first mapping of a messy feed much faster. Rules make the thousandth message boring. In healthcare, boring is a feature.

The architecture, in plain terms#

The shape is deliberately simple:

• A web app where organizations connect sources, review quality, and see results. It handles auth, organizations, roles and scheduling.
• Two separate databases. One holds platform data such as users, organizations, billing and pipeline runs, and contains no protected health information. The other holds clinical data and the vocabulary. Keeping them physically separate makes it much easier to reason about who can touch what.
• A heartbeat, not a job framework. A scheduler pings the app every couple of minutes. The app claims any due imports, syncs new files into object storage, and enqueues one task per file. Real-time storage notifications feed the same path.
• A stateless worker that processes each file through parse, clean, FHIR and OMOP, then calls back with results.
• A dedicated intake service for MLLP, the minimal framing protocol that HL7 v2 interfaces traditionally use to send messages over a network connection.

Files are deduplicated by source and content before processing, so re-sending the same feed doesn't create duplicate records. Because each file is an independent task, the system scales out by adding workers, not by rewriting the pipeline.

The carwash: making it fast without changing the answers#

Internally, I call the throughput work "the carwash": dirty data goes in one end and clean records come out the other, quickly.

The goal is about 1,000 files a minute, sustained. The first real constraint turned out to be compute quota, not code. With roughly two seconds of CPU per file and a regional cap of 200 vCPUs, the theoretical ceiling is around 6,000 files a minute. Overhead, ramp-up, I/O and database pressure eat a lot of that, which is why the sustained target is lower.

The rule for all of that work is that speed changes must not change verdicts. Every optimization runs against a set of golden C-CDA and HL7 baselines, and each step's measurements are saved before and after. A change that makes the pipeline faster but alters a single output isn't an optimization. It's a regression.

It's the same idea as the rock-tumbler approach I use in my agent loops: polish in stages, and prove each stage before you promote it.

Safety and privacy#

This is clinical data, so the guardrails aren't optional:

• Protected health information lives only in the clinical database and storage, never in the platform database.
• Secrets come from a managed secret store, not from files or code.
• Development and pilots use synthetic, de-identified or tightly controlled data.
• The original message is always preserved, so every transformation can be audited against its source.
• Observability is meant to carry identifiers and timings, not clinical content.

What I'd tell anyone starting in interop#

• Read the real data before you design anything. The standard tells you what's allowed. The feed tells you what's true.
• Treat local variation as a feature to model, not noise to drop. Custom segments exist because someone needed them.
• Report quality as a first-class output. The report is often worth more than the converted file.
• Keep AI on a leash. Suggest, explain, review, then make it a rule.
• Measure everything against golden files. Healthcare data has too many edge cases to trust intuition.

Where it stands#

The pipeline runs end to end, from intake through cleaning, terminology mapping, FHIR and OMOP, with reports at each step. It's positioned for pilots with design partners, starting with controlled data and a narrow set of message types, then widening as the rules prove out.

The takeaway#

Healthcare doesn't need another converter. It needs a trustworthy layer between old interfaces and new tools: one that tells you what's wrong with your data, fixes what it can with confidence, explains every change, and refuses to guess when it shouldn't.

That's what this stealth build is for. It's quiet work, but if AI is going to be useful in healthcare, this is the kind of work it has to stand on.