The Saga Pattern: Managing Distributed Transactions Without Two-Phase Commit

Two-phase commit sounds elegant in textbooks. In production microservices, it's a coordination nightmare. Services go down, networks partition, and holding locks across distributed systems kills throughput. The Saga pattern offers a pragmatic alternative: break a distributed transaction into a sequence of local transactions, each with a compensating action if something goes wrong.

The Problem with Distributed Transactions

In a monolith, a single database transaction ensures atomicity. Transfer money between accounts? One transaction, one commit, one rollback if anything fails. In microservices, the data lives in different databases owned by different services. There is no shared transaction coordinator.

Two-phase commit (2PC) tries to solve this by introducing a coordinator that asks all participants to prepare, then commit. But 2PC has serious drawbacks:

• Blocking: participants hold locks while waiting for the coordinator's decision
• Single point of failure: if the coordinator crashes after prepare but before commit, participants are stuck
• Latency: the round trips add significant overhead
• Availability: 2PC reduces availability because all participants must be up simultaneously

I've seen teams attempt 2PC across microservices with XA transactions in Java. It works in development. In production, the first time a network partition hits during the prepare phase, you end up with one service committed and another stuck holding locks, and a team scrambling to manually reconcile state at 3 AM.

Sagas: Local Transactions with Compensation

A saga is a sequence of local transactions. Each step commits independently. If step N fails, the saga executes compensating transactions for steps N-1 through 1, undoing the effects of the previous steps.

T1 → T2 → T3 → T4 (success)
T1 → T2 → T3 ✗ → C2 → C1 (compensation)

Each Ti is a local transaction. Each Ci is its compensating transaction.

Example: Order Fulfillment Saga

1. Create Order (Order Service): reserve the order in PENDING state
2. Reserve Inventory (Inventory Service): decrement stock
3. Process Payment (Payment Service): charge the customer
4. Ship Order (Shipping Service): schedule delivery

If payment fails at step 3:

1. Compensate Inventory: restore the reserved stock
2. Compensate Order: mark the order as CANCELLED

Choreography vs. Orchestration

There are two ways to coordinate saga steps:

Choreography

Each service publishes events and listens for events from other services. No central coordinator: the saga emerges from the event flow.

• Pros: Simple, decoupled, no single point of failure
• Cons: Hard to reason about, difficult to debug, implicit workflow scattered across services
• Best for: Simple sagas with few steps (2-3 services)

The debugging problem with choreography is real. When an order gets stuck in PENDING, you have to trace events across multiple services and message brokers to figure out where the chain broke. There's no single place that shows you the saga's current state. For a two-step saga this is manageable; for anything beyond three services, it becomes a serious operational burden.

Orchestration

A central orchestrator service directs the saga. It tells each participant what to do and handles compensation on failure.

• Pros: Explicit workflow, easy to understand, centralized error handling
• Cons: The orchestrator is a coordination point (though not a lock holder), additional service to maintain
• Best for: Complex sagas with many steps, conditional logic, or timeout handling

The orchestrator maintains a saga execution log: a persistent record of which steps have completed, which are pending, and which need compensation. This log is what makes orchestration debuggable.

public class SagaExecution {
    private String sagaId;
    private SagaStatus status;  // RUNNING, COMPENSATING, COMPLETED, FAILED
    private List<StepExecution> steps;
    private Instant startedAt;
    private Instant completedAt;
}
 
public class StepExecution {
    private String stepName;
    private StepStatus status;  // PENDING, SUCCEEDED, FAILED, COMPENSATED
    private String request;
    private String response;
    private Instant executedAt;
}

When something goes wrong, you query the saga execution log for the saga ID and see exactly which step failed, what the response was, and whether compensation has started. This level of visibility is invaluable in production.

Designing Compensating Transactions

Compensation is the hardest part of sagas. Not every action has a clean inverse:

• Reversible actions: cancelling an order, releasing a reservation. Straightforward.
• Irreversible actions: sending an email, charging a credit card. You can't unsend an email. Instead, send a follow-up ("sorry, your order was cancelled"). For payments, issue a refund.
• Idempotent compensation: compensations may be retried. Design them to be idempotent so repeated execution produces the same result.

A pattern that works well in practice is to design saga steps as reservations rather than final actions. Reserve inventory (don't decrement it), authorize payment (don't capture it), schedule shipping (don't dispatch it). The final step confirms all reservations. If anything fails, releasing a reservation is much simpler than reversing a completed action.

-- Step: Reserve inventory (not a final decrement)
UPDATE inventory
SET reserved = reserved + 1
WHERE sku_id = :sku AND (available - reserved) > 0;
 
-- Compensation: Release reservation
UPDATE inventory
SET reserved = reserved - 1
WHERE sku_id = :sku AND reserved > 0;
 
-- Final confirmation: Convert reservation to actual decrement
UPDATE inventory
SET reserved = reserved - 1, available = available - 1
WHERE sku_id = :sku;

Failure Modes and Edge Cases

• Compensation failure: what if the compensating transaction itself fails? You need retry logic with dead letter handling. In the worst case, compensations that repeatedly fail end up in a dead letter queue for manual resolution. Build dashboards for this. It will happen.
• Concurrent sagas: two sagas modifying the same resource can conflict. Use semantic locks or resource reservations. For example, two orders trying to reserve the last item in stock: the second saga's reservation step should fail gracefully, triggering compensation for its earlier steps.
• Partial visibility: between steps, the system is in an inconsistent state. Design the UI and API to handle this gracefully (e.g., show "processing" state). Users will refresh the page and see a half-completed order. Make that experience intentional, not confusing.
• Timeouts: set deadlines for each step. If a participant doesn't respond within the deadline, the orchestrator should trigger compensation rather than waiting indefinitely. A saga stuck in RUNNING for hours is a data consistency risk.
• Ordering of compensations: compensations should run in reverse order to avoid dependencies. If you compensate inventory before cancelling the order, and the order cancellation fails, you have an active order with no inventory reserved.

Observability

Sagas introduce eventual consistency, which means you need strong observability to maintain confidence in your system:

• Saga completion rate: what percentage of sagas complete successfully vs. require compensation? A sudden drop signals a problem in one of the participant services.
• Compensation rate: track how often each step triggers compensation. If the payment step fails 10% of the time, that's a signal to investigate the payment service.
• Saga duration: measure P50 and P99 end-to-end saga completion time. A slow saga means users are waiting longer for their order to process.
• Stuck sagas: alert on sagas that have been in RUNNING or COMPENSATING state longer than your timeout threshold.

When to Use Sagas

• Business transactions spanning multiple services with separate databases
• Long-running processes where holding locks is impractical
• Workflows where eventual consistency is acceptable
• Any flow involving external services (payment gateways, shipping APIs) where you can't control the transaction boundary

When Not to Use Sagas

• Single-database transactions: just use ACID
• Strong consistency requirements: sagas provide eventual consistency, not immediate
• Simple request-response flows: the saga overhead isn't justified
• High-frequency, low-latency paths: the coordination overhead may be too high

Key Takeaways

• Sagas replace distributed transactions with a sequence of local transactions plus compensations
• Choose choreography for simple flows, orchestration for complex ones
• Maintain a saga execution log for debuggability and auditability
• Design steps as reservations rather than final actions when possible
• Design compensating transactions carefully: especially for irreversible actions
• Make compensations idempotent and retriable
• Monitor saga completion rates, compensation rates, and stuck sagas
• Accept that the system will be temporarily inconsistent between saga steps