Design the retry before you need it.
Reliable systems do not treat every error as “try again.” They distinguish no-effect, confirmed effect, partial effect, stale state and unknown effect, then choose a bounded recovery path.
Resilience patterns we value
Idempotency with evidence
Do not call an operation idempotent merely because retries are convenient. Define which duplicates are harmless and how that property is established.
Bounded retry
Limit retry count/time and require fresh authority when the original attempt already consumed execution authority.
Crash/restart recovery
Persist enough state to resume safely after interruption without replaying ambiguous operations.
Stale-state rejection
Sequence/revision/fingerprint important decisions so older state cannot silently overwrite newer authority.
Visible degradation
If an optional provider fails, preserve core behavior and show the limitation rather than presenting stale/partial data as current truth.
Recovery as a product path
Users/operators need a clear next action when something fails; recovery should not depend on hidden expert knowledge.
Test the failure modes
Timeout-before-effect and timeout-after-effect look identical to a caller that only sees a lost response. Qualification should exercise both, plus partial effects, concurrency, revocation, expiry and stale configuration.