Teaching a Model to Write Parsers

A lot of useful property data is public. Building permits, business entity records, liens, plat maps. The catch is where it lives: on thousands of separate county and city websites, each with its own search form, its own results layout and its own quirks.

For ZoomProp, that's the long tail of property data. And the long tail is exactly where hand-written scrapers fall apart. You can write one parser for one county. You can't write three thousand, and you definitely can't maintain them.

So this month I built a small tool to change the shape of the problem. Instead of a person writing each parser, the tool visits the site, figures out how its search really works, decides whether it's worth automating at all and then asks a model to draft the parser. What comes out is small enough for a person to review.

The problem isn't parsing, it's discovery#

It's tempting to assume the hard part is turning HTML into structured records. It isn't. Models are pretty good at that if you show them the page.

The hard part is everything before the page:

• Is there a search form at all, or just a PDF list?
• What fields does it want? An address, a parcel number, a name?
• Does submitting it load a new page, or does a script fetch results in the background?
• Is there a details page behind each result, with the data you actually need?
• Is the whole thing behind a login or a CAPTCHA?

A parser written without answering those is a guess. So the tool's first job is to answer them by actually using the site.

Step one: use the site like a person would#

The tool drives a real headless browser. That matters, because many of these sites are built as single-page apps or load results with background requests. A plain HTTP fetch sees an empty shell.

For each site, it walks through the same workflow:

1. Load the page and wait for it to settle.
2. Find the search forms and work out what they ask for.
3. Run a real test search by filling in the form and submitting it.
4. Look at the results to see if there's a table, a list or nothing.
5. Follow a result through to its detail page, if there is one.
6. Note what data is there that's worth extracting.

Along the way it saves evidence: a full-page screenshot, an HTML snapshot and a structured report. When something looks off later, you can see exactly what the tool saw.

Step two: score it before you spend on it#

Not every site is worth automating. Some are perfect: a clean form, a results table, maybe even a background API you can call directly. Others are hopeless: a CAPTCHA on every search, or a login wall.

So before any parser gets written, each site gets an automation score between 0 and 1. Positive signals push it up:

• a search form exists
• results come back in a table
• there's an API endpoint behind the page

Negative signals pull it down hard:

• a CAPTCHA
• a required login
• the site blocking the browser

The weights aren't subtle. A CAPTCHA costs more than a results table earns, on purpose. The result sorts every site into high, medium or low potential.

score = base
      + form_found      ? small bump
      + results_table   ? bigger bump
      + api_endpoint    ? bump
      - captcha         ? big penalty
      - login_required  ? penalty
      - blocked         ? big penalty

high   -> generate a parser
medium -> generate, flag for a person to look
low    -> skip, record why

That score does two jobs. It keeps us from spending model calls and review time on sites that will never work. And it gives a summary across a whole region: which counties are easy, which are hard, and why.

Step three: let the model draft the parser#

For the sites worth it, the tool hands the model everything it learned: the form fields, the result structure, the detail page and the HTML it captured. The model's job is narrow. Write a parser for this site, in this shape, for these fields.

Narrow is the key. I'm not asking the model to explore the web or decide what to collect. The browser already did the exploring. The schema already says what to collect. The model fills in the site-specific part, which is the tedious bit a person would otherwise write by hand.

The tool also recognises when a site runs on one of the common government software platforms. A lot of counties buy the same permitting or records system, so the pages look alike. Knowing that up front gives the model a head start and makes the output more consistent across sites that share a platform.

Generated code still has to compile#

Model-written code has a familiar failure: it's almost right. A missing bracket, a bad indent, a stray quote.

So the tool treats the generated parser as untrusted until it at least parses as valid code. Common syntax slips get fixed automatically. If a parser still won't load, the analysis report sits right next to it, so a person can see what the tool saw and what went wrong.

That's the minimum. A parser that compiles isn't a parser that's right. The real check is running it against the site and comparing what it extracts with what the page actually shows. That's why the evidence from step one matters: the screenshot and the snapshot are what you check the output against.

Schemas, not scripts#

The output that matters isn't really the parser code. It's the schema behind it: how to navigate a site, and how to read what you find.

So the schema has two parts:

• A navigation schema. How to get from the landing page to a result. Which form, which fields, what to click.
• A parser schema. Given a result or detail page, which values to pull and what they mean.

Schemas get saved to a database, keyed by the site. That turns a pile of one-off scripts into data. You can query it. You can see which sites share a platform. You can regenerate a parser when a site changes without starting from scratch.

It also means the thing a person reviews is small and readable. "This county's permit search wants a parcel number, and the results table has these six columns" is much easier to check than a few hundred lines of scraper code.

Where the model stops#

It's worth being clear about what I don't let the model do:

• It doesn't decide what data to collect. That's in the schema, written by a person.
• It doesn't decide whether a site is worth automating. The score does, from observed signals.
• It doesn't mark its own work as correct. A parser has to load, and the real test is whether its output matches the captured page.
• It doesn't get around access controls. A login wall or CAPTCHA is a reason to skip a site, not a challenge to beat.

That last one is a deliberate line. These are public records on public sites. If a county puts a search behind a login, that's their call.

The same idea shows up in a lot of my work: the model does the tedious, site-specific drafting, and deterministic checks decide whether the draft is good enough. I wrote more about that pattern in The Art of the Rock Tumbler.

What I'd tell someone building one#

• Explore before you generate. Use a real browser and record what you see. Everything downstream depends on it.
• Score sites before spending on them. Cheap signals like "is there a CAPTCHA" save a lot of wasted effort.
• Keep the model's job narrow. Give it the evidence and a target schema, and ask for one specific thing.
• Store schemas, not just code. Data is easier to review, query and regenerate.
• Keep the evidence. Screenshots and HTML snapshots turn "the parser is wrong" into something you can actually debug.

The takeaway#

The long tail of public data is a discovery problem more than a parsing problem. A browser that really uses each site, a blunt score for whether it's worth automating, and a model limited to drafting the site-specific part turn thousands of hand-written scrapers into a reviewable set of schemas. The model writes the first draft. The checks, and a person, decide whether it ships.