How many times should a workflow retry?
There is no universal number. Choose attempts and delay from the downstream recovery pattern, request cost, SLA and risk of duplicated side effects. The policy should be explicit and observable.
D2 Automation Knowledge
A reliability framework for transient failures, bounded retries, terminal errors, dead-letter queues and safe replay.
Direct answer
Retry only failures that have a realistic chance of succeeding later. Bound the number of attempts, increase delay between attempts, preserve the original event and error context, and route terminal failures to a dead-letter path that can be inspected and replayed. Blind retries turn one failure into load amplification; no retries turn temporary network problems into lost work.
Engineering model
Failure policy = classify error → bounded retry → backoff/jitter → terminal routing → evidence-preserving replay
01 / Design rule
429 rate limits, temporary 5xx responses and network timeouts may be transient. Invalid credentials, schema validation failures and forbidden access usually need intervention rather than repeated attempts. Classification prevents wasted retries and hides fewer real defects.
02 / Design rule
Retrying immediately can worsen an outage or rate-limit condition. Exponential or staged backoff spaces attempts over time; jitter reduces synchronized retry spikes when many jobs fail together.
03 / Design rule
A dead-letter record should contain enough context to diagnose the failure: original event reference, workflow/version, attempt count, error category and last error. It is not simply a notification that something failed.
04 / Design rule
Replaying terminal failures is useful only when the same uniqueness and state controls remain active. Otherwise recovery can create duplicate downstream effects even though the original error is fixed.
Implementation checklist
FAQ
There is no universal number. Choose attempts and delay from the downstream recovery pattern, request cost, SLA and risk of duplicated side effects. The policy should be explicit and observable.
No for important workflows. A notification should point to durable context and a recovery action. Otherwise the operator knows a failure occurred but cannot safely resume it.
Evidence standard
D2 publishes these boundaries explicitly. The methodology page explains what evidence is required before a system is described as implemented, validated or production-backed.
Read D2 evidence methodologyRelated system evidence
Apply the framework