The problem

A mobile app sends POST /payments to charge a customer. The server processes the charge and writes to the database, but the response is lost because the phone switched from Wi-Fi to mobile data. The client sees a timeout. It does the sensible thing and retries. Without protection, the customer is charged twice.

The client cannot know whether the first request failed before, during or after processing. In any system with timeouts, retries, message redelivery or a user double-clicking a button, the same request will sometimes arrive more than once. An operation is idempotent if executing it several times has the same effect as executing it once. Idempotency does not prevent duplicates from arriving; it makes them harmless.

Natural idempotency

Some operations are idempotent by their nature:

  • PUT /users/7/email with a full value: repeating sets the same value.
  • DELETE /orders/15: after the first call the order is gone; later calls change nothing (the response code may differ, the state does not).
  • INSERT ... ON CONFLICT DO NOTHING on a business key.
  • "Set status to SHIPPED" instead of "advance the status".

Others are not: POST /payments, "add 10 to the balance", "send an email". Where you can, redesign the operation to be naturally idempotent, for example by letting the client supply the identifier of the thing being created. When that is impossible, use an idempotency key.

Idempotency keys

The client generates a unique key (a UUID is fine) for each logical operation and sends it in a header, such as Idempotency-Key. The same key is reused on every retry of that operation, and a new key is used for a new operation. The server stores the key together with the outcome and, on a repeat, returns the stored outcome instead of executing again.

Client                Server                   Database
  | POST + key K1       |                          |
  |-------------------->| insert K1 (IN_PROGRESS)  |
  |                     |------------------------->|
  |                     | charge, store result     |
  |   (response lost)   |------------------------->|
  | POST + key K1       |                          |
  |-------------------->| K1 found: COMPLETED      |
  |<--------------------| return stored response   |

The table

CREATE TABLE idempotency_record (    key           text PRIMARY KEY,    request_hash  text        NOT NULL,    status        text        NOT NULL,  -- IN_PROGRESS, COMPLETED    response_code int,    response_body jsonb,    created_at    timestamptz NOT NULL DEFAULT now());

The PRIMARY KEY is the real guard. Two concurrent requests with the same key race on the insert, and the database lets exactly one win:

try {    repo.insertInProgress(key, hash);          // unique violation if exists} catch (DuplicateKeyException e) {    var existing = repo.find(key);    if (!existing.requestHash().equals(hash)) {        throw new IdempotencyKeyReuseException();   // 422    }    if (existing.isCompleted()) return existing.toResponse();    throw new RequestInProgressException();         // 409, client retries later}var response = paymentService.charge(request);repo.complete(key, response);return response;

Three details deserve attention.

  1. Request hash. Store a hash of the request body. If the same key arrives with a different payload, that is a client bug, not a retry; reject it instead of silently returning an unrelated response.
  2. Atomicity. The business change and the COMPLETED record should be committed in the same database transaction whenever they live in the same database. Then a crash cannot leave a charge without a record, or a record without a charge.
  3. IN_PROGRESS handling. If the process crashes after inserting IN_PROGRESS but before completing, later retries would be blocked forever. Either make the whole thing one transaction (the insert rolls back with the crash) or add a timeout after which an IN_PROGRESS record may be taken over.

External side effects

When the operation calls a payment provider, a single database transaction cannot cover it. Pass your own stable identifier to the provider as its idempotency key if it supports one, so that a retry of the call is also deduplicated on their side. Record the intent first, call the provider, then record the outcome; on recovery, ask the provider for the status of that identifier instead of charging again.

Expiry

Keys cannot be stored forever. Choose a retention period longer than the longest realistic client retry window and document it, then delete older rows with a scheduled job. A key that returns after expiry is treated as new, so clients must not reuse keys for different operations. A partial index on created_at or table partitioning keeps cleanup cheap on large tables.

Consumers of messages

The same idea applies to message handlers. At-least-once delivery means a consumer may see the same message twice. Store processed message ids (or use a natural unique constraint on the effect) and skip duplicates, ideally in the same transaction as the effect.

Common mistakes

  • Checking "does the key exist?" with a SELECT and then inserting. Two requests pass the check together. Use the unique constraint.
  • Generating a new key on every retry in the client library, which defeats the whole mechanism.
  • Caching only successes. If the first attempt failed with a business error, decide whether the retry should replay that error or run again.
  • Returning 200 with an empty body on a duplicate instead of the original response.
  • Applying idempotency keys to GET, which is already safe.

Trade-offs

Idempotency storage adds a write per request and a cleanup job. Replaying stored responses means keeping response bodies, which may contain sensitive data; store only what a retry needs and apply the same retention rules as for the data itself. For low-value operations, natural idempotency is cheaper than a key table.

The HTTP contract

Describe the idempotency key explicitly in the API documentation so clients know what to expect. Document the key format (for example a UUID), its maximum length, which endpoints require it, the retention period and the error cases. Typical responses: replay of the original response for the same key and body; 422 for the same key with a different body; 409 together with Retry-After when processing has not finished yet.

On the client side the rule is simple: create the key when the user's intent is formed (for instance when the "Pay" button is shown) and keep it in memory. On a network failure, retry with the same key. If the user edits the basket and pays again, that is a new intent and needs a new key.

A concrete scenario: two button presses

A user presses the "Pay" button twice in quick succession. On each press the client sends the same key for the same intent. Both requests reach the server at the same moment; both try to insert, and the database allows only one. The winner executes the payment; the loser gets a DuplicateKeyException, then either reads the record or returns 409, or returns the stored response once the winner has finished. The user sees one payment and one response. The same mechanism works for network retries and for message redeliveries.

Finally, test idempotency. Write a test that sends the same request with the same key twice, and also in parallel, and asserts that only one side effect was created. This test shows that the mechanism also holds under a race and protects it against future changes.

Checklist

  • Every non-idempotent endpoint that clients retry accepts an idempotency key.
  • A unique constraint, not a prior read, resolves the race.
  • The request hash is compared.
  • The effect and the record commit together.
  • Retention is defined and old records are removed.

Where to go next

Concurrency behaviour underlies all of this; revisit isolation levels and lost updates. Cached responses and idempotency records share expiry concerns, see caching strategies. For the transaction boundary details, read Spring transaction pitfalls.