Retries repeat intent, not time
A retry is a second attempt to produce the same intended result. It is not permission to repeat every side effect. That distinction matters when a workflow charges a card, sends a message, allocates inventory, or updates another system.
Networks fail in ambiguous ways. A caller may time out after the server has committed the work. If the caller repeats the request without a stable identity, the server cannot distinguish recovery from a new instruction. Safe retries begin by making that identity explicit.
Give each operation a durable identity
An idempotency key represents one business operation across every transport attempt. The receiving service stores the key with the operation state and final response. A duplicate request can then return the existing result instead of repeating the work.
The key must live long enough to cover realistic retry windows, and its scope must match the operation. A key reused across unrelated actions creates false duplicates. A key created again for every HTTP attempt provides no protection at all.
- Generate the key before the first network attempt.
- Store it in the same consistency boundary as the protected state change.
- Return the previous result for completed operations.
- Expose in-progress state rather than starting parallel work.
Model state transitions directly
A boolean success flag is rarely enough for durable work. Use explicit states such as received, processing, completed, retryable failure, and terminal failure. Each transition should define who may perform it and what evidence is recorded.
Explicit state prevents two workers from quietly owning the same job. A worker can claim work with a lease or compare-and-set update, renew that claim while processing, and release it through a deliberate transition. If the worker disappears, another worker can recover after the lease expires.
Retry only failures that can improve
A timeout, temporary dependency outage, or rate limit may succeed later. Invalid input, missing authorization, and a rejected business rule usually will not. Retrying permanent failures adds load while hiding the real problem.
Use bounded attempts, exponential delay, and random jitter so many workers do not retry at the same instant. When the retry budget is exhausted, move the operation into a visible recovery path. A dead-letter queue is useful only when someone owns its inspection and replay process.
Test the ambiguous moments
The valuable tests interrupt a workflow between side effects. Stop it after an external service accepts a request but before the local state records completion. Run two workers against the same item. Deliver the same message more than once. Let a lease expire during processing.
A retry-safe design makes these moments observable and recoverable. The goal is not exactly-once transport, which distributed systems rarely provide. The practical goal is one intended business result, even when delivery happens more than once.