Message Queues & Async Work

A queue decouples the thing that requests work from the thing that does it. That buys three properties: the producer stops waiting, load spikes buffer instead of failing, and the consumer can be scaled or restarted independently.

Queues vs logs. A traditional queue (SQS, RabbitMQ) hands each message to one consumer and deletes it on acknowledgement. A log (Kafka) keeps an ordered, replayable record that many independent consumer groups read at their own offsets. If more than one system needs the same events, or you might want to reprocess history, you want a log.

Delivery guarantees. At-most-once can drop messages. At-least-once is what nearly everything gives you, and it means duplicates are guaranteed to happen eventually. Exactly-once delivery is not achievable end-to-end across a network; what systems offer is exactly-once processing, built from at-least-once delivery plus idempotent consumers.

That makes idempotency the consumer's job, not the broker's. Give each message a stable id and record processed ids, or design the operation so applying it twice is harmless.

Failure handling. A message that always fails will be redelivered forever and block progress, so a dead letter queue catches messages after N attempts for human inspection. Retries need exponential backoff and jitter — synchronised retries are how a brief blip becomes a sustained outage.

Ordering is weaker than people expect. Most brokers guarantee order only within a partition or message group, so if order matters, it must be part of the key design.

Asynchrony has a cost, and the caller pays it. The producer stops waiting, and in exchange nothing tells it whether the work succeeded. That means a status it can poll, or a notification, or an accepted-then-failed state somebody has to reconcile — none of which existed while the call was synchronous. If the caller genuinely needs the answer before it can respond, a queue does not remove the waiting, it moves it somewhere harder to see.

Watch the queue, not only the consumers. Depth and the age of the oldest message are the two numbers that matter: depth says whether you are keeping up, age says how stale the worst case already is. A consumer that is running, healthy, and falling steadily behind looks identical to a healthy one on a CPU dashboard. Alert on lag, and alert on dead letter queue depth separately, or failures pile up somewhere nobody is looking.