Designing for Idempotency in Distributed Systems

In distributed systems, exactly-once delivery is a myth. Networks drop packets, services crash mid-request, and clients retry on timeout. If your operations aren't idempotent, you'll charge customers twice, send duplicate emails, or create phantom records. Idempotency is how you make retries safe.

What Idempotency Actually Means

An operation is idempotent if performing it multiple times produces the same result as performing it once. HTTP GET is naturally idempotent. HTTP POST is not, unless you design it to be.

The key distinction:

• Naturally idempotent: SET balance = 500: same result regardless of how many times you execute it
• Not idempotent: SET balance = balance + 100: each execution changes the result
• Made idempotent: INSERT ... ON CONFLICT DO NOTHING: first call inserts, subsequent calls are no-ops

The subtlety is in "same result." Idempotency means the system state after multiple executions is the same as after one execution. The response can differ (201 on first call, 200 on subsequent calls), but the effect on the system must not change.

The Idempotency Key Pattern

The most common pattern for making non-idempotent operations safe is the idempotency key. The client generates a unique key (typically a UUID) and sends it with the request. The server tracks processed keys and returns the cached result for duplicate requests.

@PostMapping("/payments")
public ResponseEntity<Payment> createPayment(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody PaymentRequest request
) {
    Optional<Payment> existing = paymentStore.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return ResponseEntity.ok(existing.get());
    }
 
    Payment payment = paymentService.process(request);
    payment.setIdempotencyKey(idempotencyKey);
    paymentStore.save(payment);
    return ResponseEntity.status(201).body(payment);
}

There's a subtle bug in the code above. Two concurrent requests with the same idempotency key can both pass the findByIdempotencyKey check and execute the payment twice. The fix is to use a database-level unique constraint on the idempotency key and handle the constraint violation:

@Transactional
public ResponseEntity<Payment> createPayment(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody PaymentRequest request
) {
    Optional<Payment> existing = paymentStore.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return ResponseEntity.ok(existing.get());
    }
 
    try {
        Payment payment = paymentService.process(request);
        payment.setIdempotencyKey(idempotencyKey);
        paymentStore.save(payment);  // UNIQUE constraint on idempotency_key
        return ResponseEntity.status(201).body(payment);
    } catch (DataIntegrityViolationException e) {
        // Concurrent request already processed this key
        return ResponseEntity.ok(paymentStore.findByIdempotencyKey(idempotencyKey).orElseThrow());
    }
}

Implementation Details

• Store the key atomically with the operation result. Use a database transaction to ensure the key is recorded only if the operation succeeds. If the operation fails and the key is stored anyway, the retry will return a cached error instead of retrying the operation.
• Key expiration: idempotency keys don't need to live forever. 24-48 hours is typical. After expiration, a retry with the same key creates a new operation. Use a TTL index or a scheduled cleanup job.
• Key scope: scope keys per user or per merchant to avoid collisions. Two different merchants sending the same UUID should not conflict.
• Response caching: store the full response (status code, body) alongside the key. When returning a cached result, return the exact same response the first request produced. This is what Stripe does with their idempotency implementation.

Idempotency in Event-Driven Systems

Message brokers like Kafka and SQS guarantee at-least-once delivery. That means your consumers will see duplicates. The consumer must handle this gracefully.

Deduplication Strategies

1. Message ID tracking: store processed message IDs in a set (database table, Redis). Skip messages you've already seen. This works well for low-to-medium throughput but adds a database lookup per message.

2. Natural idempotency: design operations to be naturally idempotent. UPSERT instead of INSERT. SET status = COMPLETED instead of incrementing a counter. This is the preferred approach because it requires no external state tracking.

3. Transactional outbox: when producing events, write the event to an outbox table in the same transaction as the business operation. A separate process publishes outbox entries. Even if the publisher retries, the consumer can deduplicate by event ID.

The transactional outbox pattern deserves special attention because it solves the dual-write problem: when you need to update a database and publish a message, and either one can fail independently. Writing both to the database in the same transaction guarantees atomicity, and a background poller or CDC (Change Data Capture) process handles the actual publishing.

-- Within the same transaction
INSERT INTO orders (id, customer_id, total, status) VALUES (:id, :customer, :total, 'CREATED');
INSERT INTO outbox (event_id, aggregate_type, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'Order', :id, 'OrderCreated', :payload);
COMMIT;
-- A background process polls the outbox table and publishes to Kafka/SQS

Database-Level Idempotency

Several database patterns enforce idempotency at the storage layer:

• Unique constraints: INSERT ... ON CONFLICT DO NOTHING prevents duplicate records
• Conditional updates: UPDATE ... WHERE version = :expected ensures you're operating on the expected state (optimistic locking)
• Check-and-set: read the current state, compute the new state, write only if the state hasn't changed

-- Idempotent status transition
UPDATE orders
SET status = 'shipped', updated_at = NOW()
WHERE id = :order_id AND status = 'paid';
-- Returns 0 rows affected if already shipped or not in 'paid' state

Optimistic locking with a version column is particularly effective for high-read, low-write workloads (see PostgreSQL Concurrency Control for more on locking strategies):

UPDATE orders
SET status = 'shipped', version = version + 1, updated_at = NOW()
WHERE id = :order_id AND version = :expected_version;
-- If 0 rows affected: someone else modified this row. Re-read and retry.

The version check guarantees that you're operating on the state you think you are. If another transaction modified the row between your read and your write, the update affects zero rows and your application can re-read the current state and decide how to proceed.

Common Pitfalls

• Side effects: even if the database operation is idempotent, side effects (sending emails, calling external APIs) may not be. Guard side effects with a processed flag or outbox pattern. The worst version of this bug: your payment is idempotent, but the email notification isn't, so the customer gets charged once but receives five confirmation emails.

• Race conditions: two concurrent requests with the same idempotency key can both pass the "not seen" check. Use database-level uniqueness constraints, not application-level checks. An in-memory set or Redis SETNX without a surrounding transaction still has a window for duplicates.

• Partial failures: if the operation succeeds but saving the idempotency key fails, the next retry will re-execute. Atomic transactions solve this. The key and the operation result must be written in the same transaction.

• Non-deterministic operations: if the operation involves randomness or timestamps, retries will produce different results. Use the idempotency key to store and return the original result rather than re-executing the operation.

• Forgetting cleanup: idempotency keys accumulate. Without a TTL or scheduled cleanup, the tracking table grows indefinitely and query performance degrades.

Testing Idempotency

Idempotency bugs are notoriously hard to catch in unit tests because they require concurrent execution. A testing strategy that works:

1. Deterministic tests: call the endpoint twice with the same idempotency key. Assert the second call returns the cached result and no side effects fire.
2. Concurrent tests: send N requests with the same key simultaneously. Assert exactly one execution occurred. Use a CountDownLatch or similar mechanism to synchronize the requests.
3. Failure injection: kill the server after the operation succeeds but before the response reaches the client. Restart and retry. Assert the operation was not duplicated.

Key Takeaways

• Assume every operation will be retried. Design accordingly.
• Use idempotency keys for non-idempotent API endpoints. Store the full response for cache hits.
• Store the key atomically with the operation result. Use database constraints, not application checks.
• In event-driven systems, deduplicate at the consumer level. The transactional outbox pattern solves the dual-write problem.
• Prefer naturally idempotent operations (UPSERT, conditional updates) over tracking state externally.
• Guard non-idempotent side effects separately.
• Test concurrent execution explicitly. Idempotency bugs only appear under load.