Mastering the SNS + SQS Fan-Out Pattern in Event-Driven Systems

Most backend engineers reach for a message queue when they need asynchronous processing. But what happens when a single event needs to trigger multiple independent workflows? Queues are point-to-point. You need fan-out, and on AWS, the SNS + SQS pattern is the canonical solution.

Why Fan-Out Matters

Consider an e-commerce order placement. When a customer completes checkout, you need to:

• Send a confirmation email
• Update inventory
• Trigger fraud detection
• Push analytics events
• Notify the warehouse

If each of these is a separate service, a single queue won't cut it. You'd need the publishing service to know about every consumer and send messages to each queue individually. That's tight coupling: exactly what event-driven architecture is supposed to eliminate.

Fan-out decouples the publisher from the consumers. The publisher sends one message to a topic. The topic distributes copies to all subscribers. Each subscriber processes independently. Adding a new consumer means creating a new SQS queue and subscribing it to the topic. The publisher doesn't change at all.

How SNS + SQS Fan-Out Works

The pattern is straightforward:

1. SNS Topic: receives the event from the publisher
2. SQS Queues: each subscriber has its own queue, subscribed to the topic
3. Consumers: each service polls its own queue independently

{
  "TopicArn": "arn:aws:sns:us-east-1:123456789:order-placed",
  "Subscriptions": [
    { "Protocol": "sqs", "Endpoint": "arn:aws:sqs:us-east-1:123456789:email-queue" },
    { "Protocol": "sqs", "Endpoint": "arn:aws:sqs:us-east-1:123456789:inventory-queue" },
    { "Protocol": "sqs", "Endpoint": "arn:aws:sqs:us-east-1:123456789:analytics-queue" }
  ]
}

Each queue receives its own copy of the message. If the email service is slow or down, it doesn't affect inventory updates. Each consumer scales independently. Dead letter queues catch failures per-consumer.

Message Structure

Design your SNS messages with a clear, stable schema. Include enough context for all consumers to avoid callback queries to the publisher:

{
  "eventType": "ORDER_PLACED",
  "version": "1.0",
  "timestamp": "2026-03-05T14:30:00Z",
  "correlationId": "abc-123-def-456",
  "payload": {
    "orderId": "ord-789",
    "customerId": "cust-456",
    "items": [{"sku": "SKU-001", "quantity": 2, "price": 29.99}],
    "totalAmount": 59.98,
    "currency": "USD"
  }
}

The correlationId is essential for tracing a single business event across all consumers. When the email service processes this message, it logs the correlation ID. When the inventory service processes its copy, it logs the same ID. During an incident, you can query your log aggregator by correlation ID and see the full processing chain across all services.

Message Filtering

Not every subscriber needs every message. SNS supports filter policies that let you route messages selectively:

{
  "FilterPolicy": {
    "order_type": ["premium", "enterprise"],
    "region": ["us-east-1"]
  }
}

This keeps queues lean and avoids wasting compute on messages a service doesn't care about. Filter policies are evaluated by SNS before delivery, so filtered-out messages never reach the queue and don't incur SQS costs.

SNS supports two filter policy scopes: MessageAttributes (default, filters on metadata) and MessageBody (filters on the payload content itself). Body-based filtering is more flexible but slightly more expensive to evaluate. For most use cases, message attributes are sufficient and more performant.

Failure Isolation and Retry

Each SQS queue provides independent retry and failure isolation:

• Visibility timeout controls how long a message is hidden after a consumer picks it up. Set this to at least 6x your expected processing time to handle retries gracefully.
• Dead letter queues (DLQ) catch messages that fail after N retries. Always configure a DLQ. Without one, poison messages (messages that can never be processed successfully) will cycle through the queue forever.
• Redrive policies let you replay DLQ messages after fixing bugs.

This means a bug in the analytics consumer doesn't block order confirmations. Each failure domain is isolated.

Handling Poison Messages

A poison message is one that will never process successfully, no matter how many times you retry. Maybe the message has an unexpected schema, references a resource that doesn't exist, or triggers a bug in your consumer. Without a DLQ, this message blocks the queue (or wastes compute retrying indefinitely).

Configure your redrive policy to move messages to the DLQ after 3-5 failed attempts. Then monitor the DLQ:

• Alert when DLQ depth exceeds zero. Any message in a DLQ means something is wrong.
• Build a dashboard showing DLQ depth per queue over time. A growing DLQ indicates a systemic issue, not a transient failure.
• After fixing the bug, use the SQS DLQ redrive feature to replay messages back to the source queue.

Ordering and Deduplication

Standard SNS + SQS is at-least-once, best-effort ordering. If you need strict ordering, use SNS FIFO with SQS FIFO queues. FIFO guarantees:

• Messages within a message group are processed in order
• Exactly-once processing (within the 5-minute deduplication window)

The tradeoff is throughput: FIFO topics support 300 publishes/second (with batching, up to 3,000).

In practice, most fan-out use cases don't need strict ordering. The email service doesn't care whether it processes order A before order B. The analytics service doesn't care either. If you need ordering for a specific consumer (e.g., inventory updates for the same SKU must be ordered), use FIFO for that specific queue and standard for the rest. You can mix FIFO and standard SQS subscriptions on the same SNS topic by using separate topics for ordered vs. unordered flows.

Idempotent Consumers

Since standard SQS is at-least-once, every consumer must handle duplicates. SQS can deliver the same message more than once, and SNS can deliver to the same queue more than once. Design your consumers to be idempotent:

• Use the MessageId or a business key as a deduplication key
• Check if the message has already been processed before executing the business logic
• Use database-level constraints (unique indexes, conditional updates) as a safety net

Operational Playbook

Monitoring

Track these metrics per queue:

• ApproximateNumberOfMessagesVisible: messages waiting to be processed. If this is growing, your consumers aren't keeping up.
• ApproximateAgeOfOldestMessage: how long the oldest message has been waiting. Set an alarm if this exceeds your SLA (e.g., 60 seconds for order confirmations).
• NumberOfMessagesSent / NumberOfMessagesReceived: throughput. A sudden drop in sent messages might indicate a publisher issue.
• DLQ depth: any non-zero value needs investigation.

Scaling Consumers

SQS consumers scale horizontally. Lambda is the simplest option: configure an event source mapping with a batch size and concurrency limit. For ECS or EKS workloads, use SQS-based autoscaling based on ApproximateNumberOfMessagesVisible / numberOfConsumers.

When to Use This Pattern

• Multiple independent consumers need to react to the same event
• You want loose coupling between publisher and subscribers
• Consumers have different scaling, latency, or reliability requirements
• You need per-consumer failure isolation and retry policies

When Not to Use It

• Single consumer: just use SQS directly, no need for the SNS layer
• Complex routing logic: consider EventBridge for content-based routing with rich rule expressions
• Cross-account or cross-region: possible but adds operational complexity with IAM policies and VPC configuration
• Very high throughput with ordering: FIFO topics have throughput limits; consider Kafka for high-throughput ordered streams

Key Takeaways

• SNS + SQS fan-out decouples event publishers from consumers
• Each consumer gets independent scaling, retries, and failure isolation
• Include correlation IDs in messages for end-to-end traceability across all consumers
• Use filter policies to avoid unnecessary message processing
• Design every consumer to be idempotent; at-least-once delivery is the default
• Always configure DLQs and alert on non-zero depth
• Use FIFO variants when ordering matters, but be aware of throughput limits
• Keep messages small (under 256KB) or use S3 for large payloads with the extended client library