Idempotency can kill the retry

The second request found exactly what it was supposed to find.

A durable parent record already existed. The operation had been accepted once, so the retry gate refused to accept it again. No duplicate parent was created. No completed child was repeated. The request returned through the safe path the system had been designed to protect.

Farther inside the workflow, the first request was still unfinished.

Items before N had changed state. Item N had failed midway. Items after it had never been seen. The closing record and event did not exist. Yet the only retry available from outside had found the parent record and stopped.

Duplicate protection worked exactly as designed.

That is why recovery failed.

I found out the way you always find out about these things. Not from the dashboard; it was green; the dashboard is always fucking greeeen. Not from the logs, which proudly recorded a duplicate correctly rejected. From a user report, arriving sideways days later, about work that had simply never happened. The system had defended itself. Against its own recovery.

Recorded is not completed

The marker was not corrupt. The duplicate check was not inverted. The request did not slip through an obscure race. The system correctly enforced a rule against evidence that proved less than the workflow assumed.

A durable record can prove that an operation began. It cannot prove that the operation finished unless completion is modeled separately.

The workflow had collapsed two states into one operational meaning. A parent record with all required work complete and a parent record with required work still missing both looked like recorded. The retry gate could observe only that shared fact, so it treated both states as duplicates.

The distinction needs to be explicit before the recovery design can be honest.

Completion does not require every effect to succeed. Long-running workflows may end in explicit failure or cancellation. Completion requires the system to know that nothing remains pending under the workflow’s contract.

Work assigned to a recovery path remains incomplete but recoverable. Ownership prevents it from becoming stranded; it does not make it complete.

That distinction must also survive at the duplicate boundary. Finding the parent should answer whether the operation has entered before. It should not answer whether the work is finished.

When a repeated request meets a recorded operation, the system should expose the actual state: completed successfully, still running, recovery pending, awaiting repair, or terminally failed. If work remains incomplete, it may also point toward the legitimate recovery actor. It should not silently return the parent as proof of success, and it should not turn every duplicate caller into an uncontrolled recovery worker.

Where the retry door closed

The durable sequence placed the parent before the batch it represented.

First, the operation created its parent record. Then it discovered the affected items, performed the required effect for each one, wrote the closing records, and emitted the final event. The parent was necessary identity for the work. It also became the evidence that caused every repeated outer request to exit.

The failure occurred after that identity committed. Items before N had changed state; whether they had reached their intended state was no longer certain. Item N interrupted the batch. Later items were never attempted. The closing work did not happen.

The parent record did not remove the need for recovery. It removed the parent request as the actor that could perform it.

Think of a border agent who stamps your passport on the way in. When you come back for the bag you left at the carousel, he waves you off: you have already been admitted. He is right, technically. Your bag does not care about technically.

The moment that record caused every repeated request to exit, nothing beyond it could rely on replay through the outer operation. Any required work still on the other side now needed its own transaction, checkpoint, child identity, worker, or repair path.

FIG 01 · WHERE THE SECOND ATTEMPT STOPS

The same retry never gets back to the failure.

The duplicate check protects the request boundary. Recovery must enter through the incomplete work.

Dont you get it? You have spent the retry!!!

Failure is not forbidden; recovery responsibility has moved. If the parent request can no longer re-enter, the architecture must be able to name what can. Idempotency does not eliminate retry. It decides where retry is no longer allowed to enter.

What must survive without it

The effects around the marker do not all owe the same thing. Classify them against the workflow’s completion contract before choosing a mechanism.

Decide what completion requires

Begin by deciding whether the effect is required. If the operation cannot honestly claim completion without it, the effect is part of the completion contract. Calling it best-effort does not change that obligation.

An effect may be explicitly expendable when the core state remains usable, its failure is observable, and no future state or user promise depends on it. Metrics, notifications, retrospective records, and non-critical links can fit here. Expendable must be decided before failure, not assigned afterward because no recovery path can be found. It must also be reconsidered when later workflows begin depending on work that was once merely informational.

Move atomic work inside the closure

Required work that must succeed or fail with the parent belongs inside the same atomic boundary where the storage system can provide one. A transaction is appropriate when partial persistence has no valid meaning and all participating effects can actually share the transaction.

Once the parent marker has committed and closed the retry door, that option is gone for anything left beyond it. Work cannot remain after the closure and still be described as atomic with the parent.

This is not an instruction to pull every downstream action into one enormous commit. External calls, long-running work, and independent item outcomes often cannot or should not share that boundary.

Make incomplete required work resumable

Any required work that can remain incomplete after the parent commits needs durable evidence of what is complete, what remains, and which actor may continue it. Checkpoints, child identities, child-level idempotency, durable workers, and explicit repair paths are possible implementations.

The design is resumable only when it can re-enter the unfinished state without repeating completed siblings. A checkpoint that says “something failed” but cannot identify where to continue is an incident receipt, not a recovery path.

Isolate independent siblings

In a batch of independent items, one failure should not automatically consume the retry opportunity of every later sibling. Per-item isolation can allow the batch to continue and keep completed items complete.

Continuing is damage containment, not necessarily recovery. The failed item still needs an observable state and, when its required outcome remains pending, a recovery owner. Isolation complements resumption; it does not replace it.

The production workflow that formed this convention kept its parent marker and isolated the batch items. A failure at item N no longer prevented later siblings from being attempted. The failed item remained a separate recovery obligation. The fix contained the damage without pretending that containment had completed the missing work.

Early durable identity may still be necessary to stop concurrent requests from performing the same irreversible work. Fine. You have spent the retry.

The post-commit recovery audit

The review question is not only whether duplicate work is prevented. It is what happens to required work after duplicate prevention closes the outer path.

1. Locate the closure

Write the finite completion contract before tracing the workflow. Name the state and effects this operation promises, the domains it owns, the downstream work required for completion or recovery, and the consequences delegated to another independently recoverable workflow. The audit stops at an explicit handoff only when that handoff has durable acceptance and an owner of its own.

Identify the outer entry path and the durable marker that makes repeated execution stop. Locate the exact moment that marker closes re-entry through the parent operation.

Then, within that declared scope, trace every required state transition and effect still beyond that moment. Do not stop at direct writes. Include later items in a batch, closing records, events that release future work, user-visible state, and recovery metadata whose absence would make the durable operation unusable or unknowable.

The result should be a boundary, not a general suspicion: retry is permitted on this side and becomes a no-op on the other.

2. Construct the stranded states

Introduce failure at each downstream boundary and record the durable state it leaves behind. Which items completed? Which item failed? Which later items were never attempted? Which closing state, event, or link is now absent?

For each incomplete state, name the actor, event, checkpoint, child identity, worker, or repair operation that can legitimately re-enter it. If no such path exists, the failure has been made permanent by design.

Do not hide behind a generic retry policy. Prove that the named actor can reach this exact state after the parent request has already become a duplicate.

3. Prove recovery

Exercise the re-entry path. Verify that it resumes unfinished work without duplicating completed siblings, violating operation order, or erasing the evidence needed for another attempt.

Then race two legitimate re-entry attempts against the same incomplete state. Prove that only one can own the work at a time, or that child-level idempotency makes concurrent application safe. Checkpoint and completion transitions must move forward monotonically; a losing or stale actor must not overwrite newer progress.

Only now choose the implementation. Some work may need to move inside the parent’s atomic boundary. Work that remains outside needs child idempotency, checkpoints, durable workers, or explicit repair to provide re-entry. Independent batches may also need per-item isolation so one failure does not consume later siblings. The mechanism follows the failure state; it does not replace the need to describe one.

Finally, keep completion separate from durable identity all the way to the duplicate boundary: the repeated caller should learn whether recovery is still owed and who is allowed to move it.

Recovery is still owed

Once the parent operation can no longer re-enter, where can its unfinished work re-enter the system?

The second request should still stop. Duplicate protection remains necessary. But the existence of the parent record can no longer masquerade as proof that the workflow has nothing left to do.

Once the parent request is denied re-entry, every required effect beyond it must already have its own recovery path. Independent siblings must not erase one another’s opportunity to run. Work the system permits itself to lose must have been classified as expendable before the failure arrived.

Idempotency did its job. The workflow spent the retry before it had finished needing one.

So go look at your own markers. Not the ones you wrote last week — the old ones, the load-bearing ones, the ones that have been correctly rejecting duplicates for years. Somewhere behind one of them there is an item N that never got its retry.

Where does your unfinished work re-enter? If the honest answer is “it doesn’t,” your idempotency is not protecting the system. It is keeping the receipts of the failures it made permanent.