Terminology Is the Hard Part

When people hear that I'm working on clinical data, they usually picture the hard part as parsing. HL7 v2 messages look like line noise. C-CDA documents are deeply nested XML with optional sections everywhere. Surely getting those apart is the real work?

It isn't. Parsing is hard the way plumbing is hard: tedious, full of edge cases, but finite. Once you can read a segment or a section reliably, you can read it reliably forever.

The part that keeps going is terminology. Every lab result, diagnosis, medication and allergy in a clinical document carries a code. The question is always the same: what does that code actually mean, in a vocabulary someone else can use? Getting that answer right, at scale, is where most of my time went in the first serious evaluation of the cleaning pipeline I've been building.

Why codes are the product#

A clinical record is useful downstream only if its facts are comparable. A hemoglobin A1c from one hospital should line up with a hemoglobin A1c from another. A diagnosis of type 2 diabetes should be the same concept whether it came from a primary care note or a discharge summary.

The industry's answer is a set of standard vocabularies:

• LOINC for lab tests and clinical observations
• SNOMED CT for clinical concepts like conditions and findings
• ICD-10-CM for diagnoses, mostly in billing contexts
• RxNorm for medications
• Smaller ones for vaccines, drug packages and procedures

The target format for analytics in my pipeline is the OMOP Common Data Model, and OMOP is strict about this. Every row in a measurement, condition or drug table wants a standard concept ID. If you can't map a code, the row still exists, but for most research purposes it might as well not.

So the success of the whole pipeline comes down to one number: of all the coded facts in the source documents, how many did we map to a standard concept? Everything else is support.

What the source data actually looks like#

In theory, a C-CDA lab result arrives with a LOINC code and you're done. In practice, the codes in real documents come in at least four flavors:

1. A standard code, used correctly. The happy path. A LOINC code in the LOINC slot.
2. A standard code with a local code alongside it. Many documents carry a primary code plus one or more "translation" codes. Sometimes the standard one is the primary. Sometimes it's buried in a translation.
3. A local or proprietary code only. The sending system used its own internal code, or a commercial interface terminology, and never mapped it to a standard.
4. Something that only looks like a code. Free text in a code field, a placeholder, or a code system identifier that doesn't match anything.

The vocabulary itself is identified by an OID or a FHIR system URI, and those need mapping too. Before you can look a code up, you have to know which dictionary to look it up in.

The evaluation that told the truth#

The first full evaluation ran on 2,000 records: 1,000 C-CDA documents and 1,000 HL7 v2 messages. The C-CDA side was the rich one, with conditions, medications, labs and vitals across many encounters. The HL7 side turned out to be admit/discharge/transfer messages, which are almost entirely demographic.

The first run came back with zero errors, which felt good for about a minute. Then I looked at the mapping rates. Measurement mapping, which is mostly labs and vitals, was sitting at about 20%.

Zero errors and 20% mapping is a dangerous combination. Nothing crashed. Nothing in the logs looked wrong. If I'd only been watching for failures, I would have shipped it.

When I dug in, the cause was a single bad assumption. Thousands of rows (5,537 of them) carried codes from a proprietary interface terminology, and the pipeline was treating that code system as if it could be resolved directly. It couldn't. Every one of those rows either failed to map or mapped to garbage.

That's the thing about terminology bugs. They rarely announce themselves. They quietly degrade the value of everything downstream.

What fixed it#

The fix wasn't one clever trick. It was a resolution order, written down and applied the same way every time.

For each coded fact, the resolver now tries, in order:

1. Translation codes inside the document. If the source already carries a standard code next to the local one, use it. The sender did the work. Trust it, but verify it exists.
2. The standard vocabulary itself. Look the code up in the OMOP vocabulary, which is a large set of concepts and relationships (about 6.3 million concepts and 39 million relationships). If the code is non-standard but has a "maps to" relationship to a standard concept, follow it.
3. A crosswalk. For local and proprietary codes with no path in the vocabulary, fall back to a maintained crosswalk file that maps them to standard concepts.
4. Record the miss. If nothing works, keep the row, keep the original code, and count it as unmapped.

In rough pseudo-code, the shape is:

resolve(code, system):
  for t in translations(code):
    if standard(t): return t
  vocab = vocabulary_for(system)
  concept = lookup(vocab, code)
  if concept and concept.is_standard: return concept
  if concept and concept.maps_to: return concept.maps_to
  if crosswalk.has(system, code): return crosswalk.get(system, code)
  record_unmapped(system, code)
  return none

None of that is novel. What mattered was making it the only path. Before, different converters had slightly different ideas about where to look. After, there was one resolver, and it tracked its own statistics: how many codes were resolved at each step, and how many fell through.

The next run, on the same 2,000 records, came back at:

• Measurements: 91.3%
• Conditions: 99.2%
• Drugs: 90.8%
• Overall: 92.5% of about 101,000 coded concepts

Same data. Same parsers. Same converters. One coherent resolution path.

Names matter too#

There's a second, quieter terminology problem: display names.

A mapped concept ID is great for a machine. A human looking at a record wants to see "Hemoglobin A1c" and "Type 2 diabetes mellitus", not a nine-digit number. Source documents often have display text, but it's inconsistent, misspelled or missing.

The third evaluation run added a name resolver that looks up the official display label for each mapped concept. After that, condition labels were at 99.2% coverage and drug labels at about 91%. That sounds cosmetic. It isn't. The first time a clinician or analyst opens a record, the labels are what tell them whether to trust it.

The fourth run was a small fix with big visible effect. Birth dates were missing on a lot of records because the parser wasn't reading the specific CDA element where they live. After the fix, 1,995 of 2,000 records had a birth date. The other five genuinely didn't have one in the source.

Four runs between February 20 and 23. Each one fixed the biggest problem the previous one had revealed.

Coverage reports, per record#

One practice I'd carry into any data pipeline: every processed record gets its own mapping coverage report.

It's a small, readable file that lists what was in the record, what was mapped, at which step of the resolution order, and what wasn't. It sits right next to the outputs.

That report does three jobs:

• Debugging. When a mapping rate drops, I can go straight to the records that dropped and see which codes fell through.
• Trust. Anyone receiving the data can see how complete it is, record by record, instead of taking an aggregate number on faith.
• Prioritization. Rolling the unmapped codes up across a run gives a ranked list of which crosswalk entries would help the most. The long tail is long, but the head is usually short.

What I'd tell someone starting#

If you're about to build anything that turns clinical documents into analytics-ready data, here's what I'd want you to know on day one.

Measure mapping, not errors. A pipeline that never crashes and maps 20% of codes is worse than one that crashes loudly. Put mapping rate per domain on your dashboard before anything else.

Write the resolution order down. One resolver, one order, one place that tracks how codes were resolved. Don't let each converter improvise.

Respect the translations. Senders often include the standard code you need. Look before you guess.

Keep the original. Never throw away the source code and system, even after mapping. Mappings get better over time, and you'll want to re-resolve old records.

Treat the crosswalk as a product. It needs an owner, a review process and tests. It's small compared to the vocabulary, but it's the part you control.

Expect your evaluation set to surprise you. Half of my "rich" data turned out to be demographic-only admit/discharge messages. That's a fact about the source, not a pipeline bug, but I only learned it by looking at real records.

The takeaway#

Parsing gets you the data. Terminology gets you the meaning.

The most valuable stretch of this project wasn't the one where the parsers came together. It was the one where a clean-looking run turned out to be mapping a fifth of its labs, and I had to slow down and build one honest path for every code. The numbers went from embarrassing to solid on the same input, and nothing about the documents changed. Only our understanding of them did.