Why Customers Get Charged Twice: Idempotency in Distributed Systems

When talking about distributed systems, whenever you hear the term "exactly-once delivery," you should know that this is nothing but a myth. Many things can happen in practice. For instance, networks can drop packets, services can crash mid-request, and clients tend to retry whenever a timeout occurs.

In this text, you'll learn what happens when your operations are not idempotent. Those retries will charge your customers twice, send duplicate confirmation emails, or create phantom database records. Idempotency is the key to making these retries completely safe.

What Is Idempotency in Distributed Systems?

An operation is idempotent if executing it multiple times leaves the system in the exact same state as executing it a single time.

The first thing to understand is that an HTTP GET request is naturally idempotent, while an HTTP POST request is not, unless you explicitly design it to be.

To understand the core distinction, consider how state changes occur:

• Naturally idempotent: SET balance = 500 produces the exact same balance regardless of how many times you run it.
• Not idempotent: SET balance = balance + 100 increments the total every single time it executes.
• Made idempotent: INSERT ... ON CONFLICT DO NOTHING inserts a row on the first call, while subsequent retries result in harmless no-ops.

The most important subtlety here lies in the phrase "same result."

Idempotency guarantees that the overall system state after multiple executions stays identical to the state after a single execution. The HTTP response code itself can differ (such as returning a 201 Created on the first call and a 200 OK on subsequent retries), but the underlying side effect on your system needs to stay the same.

How Does an Idempotency Key Prevent Duplicate Requests?

An idempotency key is a unique request identifier generated by the client that allows the server to recognize duplicate retries and safely return a cached response.

The most reliable pattern for making non-idempotent operations safe is implementing an idempotency key. When initiating an action, the client generates a unique key (typically a random UUID) and includes it in the request headers. The server tracks processed keys in persistent storage and immediately returns the cached result whenever a duplicate key arrives.

What Is the Bug in Basic Idempotency Implementations?

Application-level checks for idempotency keys fail under concurrency because simultaneous requests can check for key existence before either transaction commits. A basic implementation checks for an existing key before processing a charge:

@PostMapping("/payments")
public ResponseEntity<Payment> createPayment(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody PaymentRequest request
) {
    Optional<Payment> existing = paymentStore.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return ResponseEntity.ok(existing.get());
    }
 
    Payment payment = paymentService.process(request);
    payment.setIdempotencyKey(idempotencyKey);
    paymentStore.save(payment);
    return ResponseEntity.status(201).body(payment);
}

In a concurrent environment, two requests carrying the same key can execute findByIdempotencyKey at the exact same moment. Both checks return empty because neither transaction has saved the key yet, resulting in double execution.

To eliminate this race condition, place a database-level unique constraint on the idempotency key column and catch the constraint violation explicitly:

@PostMapping("/payments")
public ResponseEntity<Payment> createPayment(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody PaymentRequest request
) {
    Optional<Payment> existing = paymentStore.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return ResponseEntity.ok(existing.get());
    }
 
    try {
        Payment payment = paymentService.processOnce(idempotencyKey, request);
        return ResponseEntity.status(201).body(payment);
    } catch (DataIntegrityViolationException e) {
        // A concurrent request with the same key committed first.
        // Its transaction is finished, so its saved result can be read.
        return ResponseEntity.ok(paymentStore.findByIdempotencyKey(idempotencyKey).orElseThrow());
    }
}
 
// PaymentService
@Transactional
public Payment processOnce(String idempotencyKey, PaymentRequest request) {
    Payment payment = process(request);  // records the charge in this same transaction
    payment.setIdempotencyKey(idempotencyKey);
    return paymentStore.saveAndFlush(payment);  // UNIQUE constraint on idempotency_key
}

Three important details make this pattern work:

• Atomic transactions: The charge and the key are written inside a single transaction. When the unique constraint rejects the duplicate request, its charge rolls back automatically.
• Forced flushing: Calling saveAndFlush() forces the database insert to execute inside processOnce(), throwing the DataIntegrityViolationException immediately rather than later at commit time.
• External error handling: The catch block sits outside the @Transactional boundary. In Postgres, a transaction that encounters a constraint violation refuses all subsequent queries, meaning the saved result from the winning request must be read inside a fresh transaction.

If process() communicates with an external API instead of writing exclusively to your database, a local rollback cannot undo an external charge. Forward the idempotency key directly to the upstream gateway.

Payment processors like Stripe, Adyen, and Square natively accept idempotency keys to safely drop duplicate network requests.

What Implementation Details Ensure Idempotency Key Safety?

Proper key management requires atomic state persistence, scoped namespaces, key expiration, and full HTTP response caching. When implementing idempotency keys across production systems, adhere to these four essential rules:

• Atomic storage: Store the key atomically with the operation result using database transactions. If an operation fails but the key is recorded anyway, future retries will return a cached error instead of retrying the operation.
• Key expiration: Idempotency keys don't need to live forever. As a general rule, 24 to 48 hours is enough. Use TTL indexes or scheduled background cleanup jobs to remove expired keys and prevent storage tables from degrading query performance.
• Key scope: Scope keys per user or per merchant. This is done to avoid collisions. Two distinct accounts generating identical UUID values should never conflict.
• Response caching: Save the complete HTTP response body and status code. Return the exact response the first request produced, following the standard set by APIs like Stripe.

How Do You Achieve Idempotent Event Processing in Distributed Systems?

In event-driven architectures, consumers achieve idempotency by handling at-least-once message delivery using message deduplication or naturally idempotent operations.

Message brokers like Kafka and SQS will guarantee at-least-once delivery. This design protects messages from being lost when a delivery fails, but it also means that your consumers will eventually encounter duplicate events. Consumers must be designed to handle these duplicates gracefully without producing unexpected side effects.

What Are the Best Deduplication Strategies for Message Consumers?

Message consumers rely on three primary deduplication strategies: message ID tracking, naturally idempotent state updates, and transactional outboxes. Take a look:

1. Message ID tracking: Store processed message IDs in a persistent set (such as a database table or Redis cache). Skip the processing for any message ID you have already recorded. While effective for low-to-medium throughput, this approach adds an extra database lookup per message.
2. Natural idempotency: Design operations to be naturally idempotent directly at the storage layer. Use UPSERT statements instead of INSERT, or execute SET status = COMPLETED rather than incrementing a counter. This is the preferred approach because it requires no external state tracking.
3. Transactional outbox: When producing events, write outgoing events directly to an outbox table inside the main database transaction, ensuring the message and the business entity are persisted atomically.

How Does the Transactional Outbox Pattern Solve the Dual-Write Problem?

The transactional outbox pattern solves the dual-write problem by wrapping database updates and outgoing event records inside a single atomic database transaction.

A common issue in event-driven systems is the dual-write problem, or rather, updating a database and publishing a message to a broker, where either action can fail independently. Writing both the business record and an outbox event in the same transaction guarantees atomicity. In short, either both exist, or neither does:

-- Within the same transaction
INSERT INTO orders (id, customer_id, total, status) VALUES (:id, :customer, :total, 'CREATED');
INSERT INTO outbox (event_id, aggregate_type, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'Order', :id, 'OrderCreated', :payload);
COMMIT;
-- A background process polls the outbox table and publishes to Kafka/SQS

A background process that polls the outbox table, or a CDC (Change Data Capture) engine that reads it from the database log, handles the actual publishing to Kafka or SQS. Even if the publisher retries sending a message, downstream consumers can easily deduplicate by reading the unique event_id.

How Do You Enforce Database-Level Idempotency?

Database-level idempotency is enforced through unique constraints, optimistic locking with version columns, and conditional updates that prevent duplicate state mutations.

Several database patterns enforce idempotency directly at the storage layer:

• Unique constraints: Using INSERT ... ON CONFLICT DO NOTHING prevents duplicate record creation at the table level.
• Conditional updates: Executing UPDATE ... WHERE version = :expected ensures mutations only occur if the record remains in the expected state.
• Check-and-set: Read the current state, compute the new state, and execute the write only if the underlying state has not changed.

For idempotent status transitions, a conditional update alters the record state only when specific preconditions are met:

-- Idempotent status transition
UPDATE orders
SET status = 'shipped', updated_at = NOW()
WHERE id = :order_id AND status = 'paid';
-- Returns 0 rows affected if already shipped or not in 'paid' state

This query returns 0 rows affected if the order is already shipped or is not currently in the paid state.

Optimistic locking with a version column is particularly effective for high-read, low-write workloads. You can check out PostgreSQL Concurrency Control to learn more about locking strategies:

UPDATE orders
SET status = 'shipped', version = version + 1, updated_at = NOW()
WHERE id = :order_id AND version = :expected_version;
-- If 0 rows affected: someone else modified this row. Re-read and retry.

If 0 rows are affected, another transaction modified the row between your read and write steps. The version check guarantees that you are operating on the state you expect. When a collision occurs, your application can re-read the updated state and decide how to proceed.

What Are the Most Common Idempotency Pitfalls?

Common idempotency pitfalls include unprotected side effects, in-memory or Redis race conditions, partial failures across non-transactional boundaries, non-deterministic operation execution, and omitting key cleanup.

Avoid these frequent mistakes when designing idempotent systems:

• Unprotected side effects: Even if your database operation is idempotent, external side effects like sending emails or calling third-party APIs might not be. You should guard side effects with a processed flag or outbox pattern. The worst variation of this bug occurs when your payment is idempotent, and yet your email notification is not. The result here is one charge but five confirmation emails sent to the customer.
• Race conditions: Two concurrent requests that have the same idempotency key can both pass an application "not seen" check. An in-memory set only sees its local server instance. A Redis SETNX, on the other hand, allows only one request through, but it is always outside your database transaction. If the charge fails after the key is set, or the process crashes between the two steps, the key and the result will no longer agree. That's why you should always enforce uniqueness at the database layer.
• Partial failures: If a particular operation succeeds but saving the idempotency key fails, then the next retry will re-execute the operation. Atomic transactions solve this issue because the key and the operation result must always be written in the same transaction.
• Non-deterministic operations: Let's assume that an operation involves randomness or timestamps. In that case, retries will produce different results. Here, use the idempotency key to store and return the original result instead of re-executing the logic.
• Forgetting cleanup: Idempotency keys tend to accumulate quickly. Without a TTL or scheduled cleanup job, tracking tables will grow indefinitely. As a result, query performance will degrade.

How Do You Test Idempotency in Production Code?

Testing idempotency requires deterministic retry assertions, concurrent execution tests using tools like CountDownLatch, and server failure injection. And since idempotency bugs typically manifest under load, unit tests calling an endpoint once will miss them. That's why you should apply a three-part testing strategy that actually works:

1. Deterministic tests: Call the endpoint twice with the exact same idempotency key. Then, check whether that second call returns the cached result and that there are no side effects.
2. Concurrent tests: Send N requests with the very same key simultaneously. Then, check whether exactly one execution occurred. Use a CountDownLatch or some similar synchronization primitive in Java to fire requests at the exact same moment.
3. Failure injection: Kill the server process after the database operation succeeds but before the response reaches the client. Restart the server, issue a retry, and assert that the operation was not executed a second time.

What Are the Key Takeaways?

Designing idempotent systems requires atomic state management, database-level uniqueness constraints, and deduplication at both the API and consumer layers.

• Always assume that every operation will be retried and design your code accordingly.
• Always use idempotency keys for non-idempotent API endpoints and store the full HTTP response for cache hits.
• Always store the key atomically with the operation result using database constraints rather than application-level checks.
• In event-driven systems, make sure to deduplicate at the consumer level. Here, the transactional outbox pattern solves the dual-write problem.
• You should always go for naturally idempotent operations like UPSERT and conditional updates over tracking state externally.
• Always guard non-idempotent side effects separately.
• Test concurrent execution explicitly since idempotency bugs tend to only appear under load.