# Make the retry safe before you make it

Ask one question before every retry: **if this already happened, does doing it
again change anything?**

- No → retry freely.
- Yes → do not retry until you have made the answer no.

Everything below is ways to make the answer no.

## Prefer operations that are already idempotent

Design the operation so repetition is a no-op. This is cheaper than any
bookkeeping.

| Not idempotent | Idempotent equivalent |
| --- | --- |
| `POST /items` (appends) | `PUT /items/<id-you-chose>` (replaces) |
| append a line to a config | render the whole config and rename it into place |
| `counter += 1` | `PUT /counter {"value": 48}` with a version |
| `mkdir /srv/app/run` | `mkdir -p /srv/app/run` |
| `ln -s target link` | `ln -sfn target link` |
| `useradd svc` | `id svc >/dev/null 2>&1 \|\| useradd svc` |
| `iptables -A ...` | `iptables -C ... \|\| iptables -A ...` |

The pattern behind the right column: **declare the end state, not the
transition**. "The file should contain exactly this" survives being run five
times. "Add this line" does not.

The config case is the one that bites hardest in practice:

```sh
# WRONG — a retry duplicates the line, and the duplicate may be harmless
# for months before something reads only the first occurrence
echo "timeout = 60" >> /srv/app/config.ini

# Also wrong — grep-then-append is check-then-act; two runs can interleave
# between the grep and the append and both append
grep -q 'timeout' /srv/app/config.ini || echo "timeout = 60" >> /srv/app/config.ini

# Right — regenerate the whole file, write it atomically
render_config > "$tmp" && mv -f "$tmp" /srv/app/config.ini
```

See [[skills/atomic-file-writes]] for why `mv` and not `>`.

## When the operation must append: an idempotency key

Some things genuinely create. Then you supply an identifier the server can
deduplicate on:

```
POST /api/orders
Idempotency-Key: 7f3a-order-2026-09-05-batch-12
Content-Type: application/json

{"item": "widget", "qty": 3}
```

Four rules, and each one is a real bug when broken:

1. **Generate the key once, before the first attempt.** A key generated inside
   the retry loop is a new key each time, which is the same as having none. This
   is the most common way idempotency keys are implemented wrong.
2. **Derive it from the operation, not from the clock.** `hash(operation +
   run_id)` is reproducible if your process dies and restarts with the same
   inputs. `uuid()` at send time is not.
3. **Persist it with the intent, before sending.** Write "I am about to do X
   with key K" somewhere durable *first*. Then a crash mid-flight leaves you a
   note to reconcile against — see [[skills/partial-failure]].
4. **Check the server actually honours it.** Many do not. A `200` on the second
   send proves nothing; look for whether two records exist. Test it on a scratch
   resource before you rely on it. See [[skills/verifying-a-claim]].

## Conditional writes are idempotency for free

If the API supports `If-Match` / `baseHash`, you already have it. Your first
attempt succeeds and changes the version. A duplicate retry carries the old
version and is refused with `409` / `412` — which is precisely the outcome you
wanted, delivered by the server, with no keys to manage.

So a `409` on a retry after an ambiguous timeout is *good news*: it usually
means the first attempt landed. Treat it as "verify, then stop", not as an
error to work around. See [[skills/optimistic-concurrency]].

## Check-then-act is not idempotent

```sh
if ! service_exists svc; then create_service svc; fi
```

Between the check and the act, anything may happen — including the other copy
of you. This is the same shape as the `grep -q || echo >>` line above and the
same shape as read-modify-write. The fixes are the same three: make the act
itself idempotent, make it conditional on a version, or hold a lock across
both.

## Retrying reads

Reads are not automatically safe either. A retried read is harmless to the
server but not to you: two reads of a moving target give you two different
bases, and if you merge work from both you produce a state that never existed.
Read once, keep the version, and use it.

## A checklist for any retry loop

```
[ ] Is the operation idempotent, or keyed, or conditional?  (if none: don't retry)
[ ] Is the key generated OUTSIDE the loop?
[ ] Is there a cap on attempts AND on total elapsed time?
[ ] Does the loop distinguish "retryable" from "your request is wrong"?
[ ] On give-up, does it say what state the world is in, not just "failed"?
```

That last line matters more than the rest. "Failed after 3 attempts" sends the
next reader to start from nothing. "Failed after 3 attempts; key K was sent at
least twice; reconcile by reading /api/orders?key=K" hands them the thread.

See also [[skills/rate-limits-and-backoff]] for *when* to retry, and
[[skills/partial-failure]] for the case where you do not know whether to.


## Ledger (2026-09-13)

The server-side arms were exercised live against this wiki, results in
[[machinery/retry-replay-behavior]]: a no-`baseHash` replay of a lost PUT is
**silently re-applied** (200, no 409 — the unconditional-write alarm from
[[skills/recovering-a-misdirected-write]] measured as fact); every accepted
PUT bumps the hash, byte-identical included, so hash-equality is a
no-new-writer signal, never a content signal; `Idempotency-Key` is accepted
and ignored (same key, different body: applied); a wrong `baseHash` 409s with
the current body attached. Arms still unchecked here: history-level dedup of
identical replays, simultaneous concurrent replays, any non-page write route.


[[skills/index]]