Cloud

Building block · 9 min

Queues and pub/sub

SQS and SNS, Pub/Sub and Cloud Tasks, Service Bus: delivery guarantees, ordering and limits.

A queue holds work until a worker is ready for it. One message goes to one worker, the worker deletes it when the job is done, and if the worker dies the message comes back for someone else. Pub/sub is the other half: one published message is copied to every subscriber. The course covers both ideas in Queues and workers and Queues and pub/sub, and the trap they share, duplicates, in Jobs that can run twice.

Every cloud sells both, and on the surface they look interchangeable. Underneath they differ in what "in order" means, how long a message can sit, how big it can be, and whether "exactly once" is a real promise or a marketing word. Those differences decide designs. If you need replay, many readers of the same history, or ordering across millions of keys, you want a log rather than a queue: see Event streams.

Side by side

AWS Google Cloud Azure Self-hosted
Work queue SQS (standard and FIFO) Pub/Sub pull subscriptions; Cloud Tasks for rate-limited HTTP jobs Service Bus queues; Queue Storage for simple, cheap ones RabbitMQ quorum queues
Fan-out SNS topic to SQS queues; EventBridge for rule-based routing Pub/Sub topic with many subscriptions; Eventarc for routing Service Bus topics and subscriptions; Event Grid for routing RabbitMQ exchanges
Ordering FIFO queues, per message group Ordering keys, per key Sessions, per session ID Per queue, with one consumer
Largest message 1 MiB 10 MB 256 KB on Standard, up to 100 MB on Premium; 64 KB on Queue Storage Configurable
Longest a message can wait 14 days 31 days Set by time-to-live; Queue Storage can keep messages indefinitely Bounded by your disks
You pay per Request: up to 10 messages per request, each 64 KB billed as one Data volume: per TiB of throughput, with a 1 KB minimum per request Operation on Standard, dedicated messaging units per hour on Premium Servers and the people running them