The Job That Died at 89%: Checkpoint on Every Flush, Not on Completion

A data pull ran for hours and died at 89%. It did not produce 89% of the output. It produced nothing.

Two separate design decisions caused that, both of them the kind that look reasonable when you write them and only reveal themselves the first time a long job is interrupted. Which it will be.

Failure one: the job was tethered to the thing that started it

A process launched in the background from an interactive session is a child of that session. When the session ends, is reaped, times out, or is cleaned up by the environment, the child goes with it.

The lifetime you are relying on is not the job's. It is the session's, and on a managed environment the session's lifetime is not yours to control. Some of these were reaped inside a minute.

A multi-hour job needs to be genuinely detached: started as its own process, disowned by its parent, with its output going to a file and its process id written to disk. That last part matters more than it sounds. A detached process you cannot identify later is a job you can neither monitor nor stop, and you will need to do both.

Failure two: the checkpoint was written on completion

This is the one worth internalising, because it is almost universal in first drafts.

The job tracked its progress and wrote a checkpoint file so it could resume. The checkpoint was written when the run finished.

A checkpoint written at completion only ever helps a job that completed — which is the exact case that needed no help. For every run that dies partway, and partway is where runs die, it does nothing at all.

Eighty-nine percent of the work was done. All of it was thrown away, and the restart began at zero.

Checkpoint on every flush, not on completion. A checkpoint that only exists after success is a comment, not a recovery mechanism.

The pattern that works

Any long-running job should be built so that killing it at a random moment costs you one batch, not the run.

1. Batch the work. Process a fixed number of records — a few hundred to a few thousand — then flush.
2. Write the checkpoint at every flush, in the same step. Cursor position, records processed, timestamp. Data and checkpoint move together or the checkpoint will drift out of sync with reality.
3. Write it atomically. Temporary file, then rename. A checkpoint half-written during a kill is worse than no checkpoint, because the resume logic will trust it.
4. Make resume idempotent. Upsert rather than insert. A resumed job will reprocess part of the last batch, and that must be harmless. If re-running a batch corrupts the result, you do not have resumability, you have a coin flip.
5. Log progress with an absolute count, not a percentage. "412,000 of 2,700,000" tells you the rate and the remaining time. "15%" tells you nothing you can act on.

Do the arithmetic before choosing the architecture

The other thing that had not been done was the simplest: working out how long the job would take.

The API in question capped at 100 records per page and sustained roughly 104 records per second. For 2.7 million records, that is about seven hours.

Seven hours is a completely different engineering problem from seven minutes. It needs detachment, checkpointing, resumability, and a monitoring path. Seven minutes needs none of those. The same code shape serves both badly, and the only way to know which one you are in is to multiply two numbers before you start.

Two limits to check first, because both change the answer by an order of magnitude:

• Page size, per endpoint. These vary within the same API. One endpoint allowing 100 per page and another allowing 10 is a ten-times difference in total runtime, and nothing in the code will tell you until it is running.
• Whether a changed-since filter exists. If it does, you run the full pull once and every subsequent run is a delta of minutes. If it does not, every refresh costs you the full seven hours, and that fact belongs in the decision about whether to build this at all.

Monitoring, not watching

A seven-hour job should be answerable without a human sitting in front of it.

The checkpoint file gives you that for free, if you write it on every flush. Read it and you have the current cursor, the count, and the timestamp of the last write. A stale timestamp means the job is dead or stuck — and the process id file tells you which, and lets you end it cleanly.

That is the whole monitoring stack: two small files, written by the job, read by anything. No dashboard required.

Resumability is what makes consistency possible

The training rule is never miss twice. Miss one session and you go the next day. What makes that rule survivable is that a program can absorb a missed day without being invalidated — you resume from where you are and the block continues.

A system with no checkpoint does not have that property. It has to run perfectly or it is worth nothing, which means every interruption is a total loss and the whole thing becomes fragile in exactly the way a training block is not.

Build the recovery path first. Then the interruption is an inconvenience instead of a restart, and you get to be the operator who resumes rather than the one who begins again.