Understanding Message Delivery Semantics in Event-Driven Microservices

To Nha Notes | April 15, 2023, 12:54 p.m.

Detail of the interactions with the message broker

Each of these steps can fail and can originate different consistency issues:

1. The broker, or the connection between the order service and the broker, can be unavailable. The order service will fail to publish the message to the broker.

2. After receiving the message, the broker can lose the connection with the order service, or the service can crash, failing to acknowledge the message’s reception.

3. The inventory service, or the connection between the inventory service and the broker, can be unavailable. The broker will fail to deliver the message to the consumer.

4. The consumer, in this case the inventory service, can fail to acknowledge the reception and processing of the message to the broker.

Depending on the behavior of the producer and the consumer, we can have different message delivery semantics:

• At-most-once delivery: Messages might fail to be delivered but won’t be delivered more than once. For example, in Figure 7-6, if the order service publishes a message but the message broker has an internal error persisting the message and returns an error, the message might not have been processed by the broker. With at-most-once semantics, the order service when faced with this error won’t retry the operation. Meaning messages will never be duplicated, but not all messages will reach the consumers. We avoid duplicate messages, but we also accept that we can lose messages.

• At-least-once delivery: Messages will always be delivered but might be delivered more than once. In the same use case we discussed before, if the order service didn’t receive an acknowledgment from the broker, it could retry the operation. The broker already received the message and only failed to acknowledge it. By retrying, we would end up publishing a duplicate message. Thus, we guarantee messages are always delivered but might be more than once.

• Exactly-once delivery: Messages will always be delivered to the consumers without duplicates. In this case, the order service might receive an error, and even by retrying the operation, the message would arrive only once to the inventory service. Exactly-once is the most desirable delivery semantic but is also exceptionally hard to guarantee in a distributed system.

 Achieving at-least-once delivery can be straightforward, and by designing the event consumption to be idempotent, we can avoid incorrect states if duplicates are found. We might have duplicate work, but it likely won’t be significant if we process a handful of messages twice in a range of thousands or millions of events. Designing for idempotency makes the overall system more resilient and flexible (rewinding the event stream can be a valid approach, for example).