Four Years of One Product's Data Pipelines

ZoomProp runs on property data. Listings, parcels, demographics, schools, flood risk, zoning, permits. Some of it comes from bulk files, some from APIs and a lot of it from public websites that were never designed to be read by a machine.

The pipelines behind that data have been evolving since 2022. Going back through the repos in order, oldest first, shows how the shape of the system changed. It's a useful exercise. Each generation solved the last one's biggest problem and quietly introduced the next one's.

Generation one: a scheduler on a box (2022)#

The first version was a workflow scheduler running in Docker on a single cloud server. A handful of scheduled jobs pulled listings, points of interest and a grid-based search over the map.

It worked. It also had all the classic marks of a first version:

• Setup was a numbered list of console steps. Launch the instance, attach a role, assign an address, install a deploy agent, install Docker, bring the stack up.
• Deploys were half automated. Scripts went out through a deploy tool. At least one job definition was still copied over by hand, with a note in the README to fix that later.
• Dependencies were installed inside a running container. If the container was rebuilt, someone had to remember to do it again.

None of this is a criticism. It was the right speed for the time. But you can see the problem: the system's real configuration lived partly in the repo and partly in someone's memory.

Generation two: assets and a pile of scripts (2023)#

In 2023 a new pipelines repo started on Dagster, deployed through a hosted service with a staging environment and a CI workflow on every push to main.

The big shift was thinking in assets instead of tasks. A task is "run this script at 2am". An asset is "this table of school locations exists, here's how it's built, and here's what it depends on". The scheduler then works out what needs rebuilding.

The repo grew quickly, and it shows two different ways of working side by side:

• Assets for the things that run on a schedule and feed the product, like availability calculations and content.
• A large scripts folder for everything else: census boundaries at several levels, demographics by zip code and block group, schools and amenities, a house price index, property tax rates, bulk property loads, short-term rental market data and search index maintenance.

The scripts folder was where ideas went first. That's healthy. It's also where things went to stay. Many of those scripts ran once, or a few times, and never became assets. Each had its own config, its own database helper and its own idea of what a property record looked like.

That last point turned out to be the real cost.

The shared model (2024)#

By 2024 there was an API service as well, built on FastAPI, with a separate WebSocket app for live updates. Now the same property data was being written by pipelines and read by an API, in different repos.

The fix was to pull the data model out into its own repo. The API includes the shared ORM models as a Git submodule, and the pipeline repos pull in shared code the same way. A property gets defined in one place instead of once per script.

Submodules aren't anyone's favourite tool. They add a step to cloning, and you have to remember to update them. But for this problem they did the job: one definition of the schema, pinned to a specific version in each consumer, changed in one place.

Across all four years, this is the change that made the most later work easier, because the data model stopped being reinvented per script.

Generation three: a pattern per record type (late 2024 onwards)#

In late 2024 a second Dagster project started for a newer, harder kind of data: public records from county and city websites. Building permits, business entities, judgements, liens and plat maps.

These all have the same problem. The data exists, but on thousands of different sites. So instead of one big script per source, every record type follows the same pattern:

for each record type:
  navigation schema   how to reach the data on each site
  parser schema       how to read a result into fields
  extraction          run the schemas, collect raw records
  postprocessing      clean, normalise, validate against the shared model

Each step is an asset. Each record type is a folder with the same four parts, plus its own cleaning rules and schema. There's even an example record type that exists purely as a template for the next one.

Alongside those sit a few unglamorous maintenance assets: reapply a schema to data you've already extracted, clean extracted data by hand when needed and drop partitions that need rebuilding. Those are the kind of tools you only write after you've needed them.

The same project also holds national datasets that every property needs context from: broadband and wireless coverage, cell towers, crime, energy providers, schools, transit stops, walkability, weather risk and zoning.

Models joined the pipeline, behind an interface#

Some of these sources need a model to make sense of them. Zoning codes are a good example: the rules are written as prose and every jurisdiction writes them differently.

Rather than call one provider directly from each asset, the pipeline has a small assistant layer: an abstract client, an implementation per provider and a factory that picks one. The assets ask for "an assistant" and don't care which model answers.

That design has two benefits. It's easy to compare providers on the same task. And when pricing or quality shifts, switching is a config change rather than a rewrite.

The schema generation tool I wrote about in Teaching a Model to Write Parsers eventually moved into this project as an asset of its own, so generating a schema for a new site is part of the same pipeline that uses it.

What stayed the same#

For all that change, a few things never moved:

• A relational database for records, with a search index alongside it for the product's property search.
• Python for the pipelines. Every generation, every repo.
• Geography as the backbone. Almost every dataset eventually gets joined to a parcel, a census tract or a zip code.
• Public data is messy. No generation fixed that. They just got better at containing it.

What I'd do first next time#

Looking at it in order, the lessons are pretty clear:

1. Put the infrastructure in the repo from day one. The first generation's README listing console steps is the clearest sign of trouble. If setting it up needs a person to remember things, it will drift.
2. Define the data model once, before the second consumer shows up. I'd extract the shared model the moment a second repo needs it.
3. Give scripts a path to graduation. A scripts folder is great for exploring. Add a simple rule: if it runs more than twice, it becomes an asset with tests, or it gets deleted.
4. Use one pattern per kind of data. The per-record-type layout made the hard sources boring, which is exactly what you want.
5. Keep model calls behind an interface. Providers change faster than pipelines.

There's a theme here that I keep running into: code tends to spread out and then has to be pulled back together. I wrote more about that in The Consolidation of Code Over Time.

The takeaway#

Four years of pipelines went from a scheduler on one server, to assets plus a pile of scripts, to a repeatable pattern per kind of data with a shared model underneath. The tools changed. The biggest wins didn't come from the tools. They came from defining the data once, writing the infrastructure down and giving every new source the same shape.