A Heartbeat Instead of a Job Framework

Every data product eventually needs a way to move work through the system on its own. Files arrive. Schedules come due. Something has to notice, pick up the work, hand it to a worker and record what happened.

The usual reach is a job framework: a broker, a pool of long-running workers, a scheduler process, maybe a dashboard. I started there on the healthcare data pipeline I've been building, with a classic queue-and-worker setup. It worked on one machine. It got harder to reason about the moment it had to run in the cloud, scale out and stay correct when things died halfway through.

So I replaced it with something smaller. A heartbeat, a sweep and a stateless worker. This post is about that design and why I like it.

The shape#

Here's the whole thing in one picture:

scheduler (every 2 min)
   │
   ▼
sweep route ──► claim due schedules
   │            list source, dedupe, copy new files
   ▼
task queue  ──► one task per batch of files
   │
   ▼
stateless worker ──► parse, clean, convert
   │
   ▼
callback ──► record results, update status

There are four moving parts:

1. A managed scheduler that makes one HTTP call every two minutes. That's all it does.
2. A sweep route in the web app that receives that call, claims whatever is due and enqueues work.
3. A managed task queue that dispatches each unit of work to the worker, with rate limits and retries.
4. A stateless worker that processes files and calls back with results.

No long-lived scheduler process to babysit. No broker to run. No worker that holds state between requests.

The heartbeat#

The heartbeat is the least interesting part, which is the point. It's a cron expression on a managed scheduler that hits one internal route every two minutes, with an authenticated request.

Two minutes is a deliberate choice. It's fast enough that new files feel close to real time for batch-style clinical feeds. It's slow enough that each tick has room to finish its work before the next one arrives. And it's frequent enough that a missed tick doesn't matter much. The next one is right behind it.

The heartbeat doesn't know what's due. It just says "now". All the intelligence lives in the sweep.

The sweep: claim, list, dedupe, enqueue#

When the sweep runs, it does the same few steps every time.

Claim due schedules with an optimistic lock. Each schedule has a "next run" time. The sweep tries to claim the ones that are due by updating them conditionally, so if two ticks overlap, only one of them wins each schedule. No double-firing, without a separate locking service.

List the source. Each organization connects a source, like a storage bucket or an SFTP drop. The sweep lists what's there.

Dedupe by path and content hash. A file is new only if its path and content hash haven't been seen before. Same name with new content counts as new, and a file that has already been processed is skipped.

Copy new files into our own storage. Before processing, files are copied into storage we control. The worker never reads from a customer's source directly.

Enqueue. New files are grouped into batches and each batch becomes one task on the queue.

In rough pseudo-code:

on_tick():
  for schedule in claim_due_schedules():
    files = list_source(schedule.org)
    new   = [f for f in files if not seen(f.path, f.hash)]
    copy_to_storage(new)
    for batch in chunk(new):
      enqueue(batch)
    advance(schedule)

One core, three triggers#

There are three ways work can start:

• An import, which is a one-time backfill of everything already in a source.
• A scheduled pull, which runs on a recurring schedule.
• A real-time event, where the storage provider calls us when a single new object lands.

All three funnel into the same core function that pulls and enqueues. That's the most important design decision in the whole flow. There aren't three ingestion paths that drift apart. There's one, with three doors.

The import case has a nice trick. An import is just a schedule that knows how to end. It creates a special kind of schedule that the sweep drains across ticks, a chunk at a time, until nothing new is left. Then it disables itself. No separate backfill system. No special worker. The same heartbeat does the work.

Why this holds up#

It's idempotent. Running the same import twice is a safe no-op for files already seen. Re-enqueuing a batch that already finished doesn't create duplicates. That means retries are cheap and recovery is boring.

It's resumable. If a tick dies halfway through, the next tick picks up where it left off. Nothing important lives in memory between ticks. The state is in the database: what's been seen, what's due, what's running.

It scales horizontally. The worker is stateless, so the platform can run one copy or fifty. The queue decides how fast to hand out work.

It's observable. Every schedule, run and file has a row with a status. "What happened to this file?" is a query, not a log search.

Backpressure belongs to the queue#

One lesson I learned the hard way: the queue's limits need to match what the worker can actually handle.

Early on, the queue was configured to allow far more concurrent dispatches than the worker could ever run at once. The worker's real ceiling, set by instance counts and per-instance concurrency, was a small fraction of that. The result was a pile of rejected requests during bursts, retries chewing through the retry budget, and files that failed for no reason other than impatience.

The fix was simple. Cap the queue's concurrency and dispatch rate to match the worker's real ceiling. Let the queue hold the backlog. That's what it's for.

Two related rules made a big difference:

• Don't retry client errors. If the callback rejects a payload as malformed, sending it again won't help. Retry server errors with a short, flat backoff instead.
• Make failures re-queueable. A "retry failed" action that puts failed files back on the queue turns a bad afternoon into a button press.

A gate before automation#

There's one more piece that isn't technical so much as product judgment.

An organization can't turn on automation, or run a large import, until it's proven that its data processes well. It has to run a sample first, and that sample has to reach a minimum size with a good pass rate. Until then, the sweep skips that organization's schedules and records why.

This saves everyone from the worst version of onboarding: connecting a source with a hundred thousand files, letting them all run and discovering an hour later that a mapping problem broke every one of them. Prove it on a small batch first. Then open the tap.

It does create one recurring confusion. When someone says "the import does nothing", the answer is usually "the organization isn't ready yet". That's a documentation and UI problem, and I'd rather have it than the alternative.

Only one import can run per organization at a time, for similar reasons. It keeps the queue fair and the status easy to understand.

What I gave up#

This design isn't free.

• Latency has a floor. Scheduled work can wait up to a couple of minutes for the next tick. Real-time events skip that wait, but scheduled pulls don't.
• The sweep has to stay fast. If one tick takes longer than two minutes, ticks start to overlap. The optimistic lock keeps that safe, but it's a signal to split work into smaller chunks.
• Managed services decide some things for you. Retry limits, dispatch rates and timeouts are configuration on someone else's platform. That's mostly a gift, but it means reading their docs carefully.

For a clinical batch pipeline, all three are fine trades.

The takeaway#

The most reliable systems I've built have the fewest long-running parts. A heartbeat, a sweep that only does work it can claim, a queue that owns backpressure and a worker that forgets everything between requests. Each piece is boring. Together, they're the part of the platform I worry about least, and that's the highest compliment I can give infrastructure.