Messaging & distributed systems

#Synchronous calls, asynchronous messaging and brokers

If checkout waits for email and analytics, a secondary outage can prevent a valid purchase. Synchronous HTTP fits a decision needed immediately; messaging fits work that can follow the committed order.

A broker accepts, routes and stores messages according to its configuration. Publishing is not automatically durable, and acceptance does not mean processing finished. A command requests an action from an owner; an event announces a fact that multiple subscribers may care about.

Checkout -> database: order + outbox row in one transaction // (1)
Dispatcher -> broker -> email queue -> email worker          // (2)
                     -> analytics queue -> analytics worker

The costs include eventual consistency, duplicates and operational complexity. Expose meaningful pending states and monitor message age. See topic 12 for CAP and consistency; messaging does not automatically guarantee availability.

#RabbitMQ components and exchange types

A producer publishes to an exchange with a routing key. Bindings connect exchanges to queues using routing rules, and consumers read from queues. Connections are normally long-lived, with lighter channels inside them; virtual hosts provide logical isolation and separate permissions.

The exchange does not store the backlog. A DLX is also an exchange and needs a bound queue to retain failures. These are the core AMQP 0-9-1 exchange forms; RabbitMQ also has extensions and additional types. See the exchange reference.

exchange=orders (topic), routingKey=order.paid.eg
  binding=order.*.eg -> regional-orders
  binding=order.#    -> all-orders
  binding=order.*    -> no match (only two words)
exchange="", routingKey=email -> queue named email

#RabbitMQ.Client 7: publishing and confirms

The application sending a message is the publisher; RabbitMQ is the broker that receives it and routes it to a queue. A successful socket write means data left your application, not that the broker accepted or retained it. A publisher confirm is the broker’s response about the publish. It differs from a consumer ack, the consumer’s signal after it finishes handling a delivery. This console example requires RabbitMQ.Client 7.x and a local broker; production needs TLS, restricted permissions and external configuration.

using RabbitMQ.Client;
using System.Text;

var factory = new ConnectionFactory { HostName = "localhost" };
await using var connection = await factory.CreateConnectionAsync();
var options = new CreateChannelOptions(
    publisherConfirmationsEnabled: true,
    publisherConfirmationTrackingEnabled: true);                  // (1)
await using var channel = await connection.CreateChannelAsync(options);
await channel.QueueDeclareAsync(
    queue: "orders", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object?>
    {
        ["x-queue-type"] = "quorum"                               // (2)
    });
var properties = new BasicProperties
{
    Persistent = true,
    MessageId = Guid.NewGuid().ToString()
};
using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await channel.BasicPublishAsync(
    exchange: "", routingKey: "orders", mandatory: true,
    basicProperties: properties, body: Encoding.UTF8.GetBytes("order-created"),
    cancellationToken: timeout.Token);                           // (3)

#Acknowledgments, prefetch and dead lettering

When a broker delivers a message to a consumer, the consumer sends an ack (acknowledgment) after successful handling so the broker knows it can clear that delivery. If the consumer saves a database change and then crashes before the ack, the broker may deliver the message again; the handler must tolerate duplicates without repeating the effect. Prefetch limits delivered messages that are still unacknowledged; it is different from the number of handlers running at once.

// Inside a BackgroundService: channel and HandleAsync are application collaborators.
// Provision the queue and a bound DLQ first; configure its DLX through broker policy.
using RabbitMQ.Client;
using RabbitMQ.Client.Events;

await channel.BasicQosAsync(0, prefetchCount: 16, global: false); // (1)
var consumer = new AsyncEventingBasicConsumer(channel);
consumer.ReceivedAsync += async (_, delivery) =>
{
    byte[] body = delivery.Body.ToArray();                      // (2)
    try
    {
        await HandleAsync(body, stoppingToken);
    }
    catch (InvalidDataException)
    {
        await channel.BasicNackAsync(
            delivery.DeliveryTag, multiple: false, requeue: false); // (3)
        return;
    }
    await channel.BasicAckAsync(delivery.DeliveryTag, multiple: false); // (4)
};
await channel.BasicConsumeAsync("orders", autoAck: false, consumer: consumer);
await Task.Delay(Timeout.Infinite, stoppingToken);               // (5)

The fragment isolates permanent failures. Transient failures require explicit callback exception handling and bounded delayed retries or retry queues; an unhandled callback exception is not a retry policy. Closing a channel requeues unacknowledged deliveries, but unlimited immediate requeue creates a hot loop. Monitor the DLQ and provide controlled replay. See the client API guide.

#Kafka: logs, partitions and offsets

When several systems need to replay history, a queue that removes acknowledged messages may be the wrong model. Kafka partitions a topic into ordered logs, each with partition-local offsets. Reading does not remove records; time or size retention controls their lifetime, while compaction is a separate policy for retaining keyed state.

In a conventional consumer group, a partition has one active consumer within that group. Different groups read independently. Committing before processing can lose effects; committing afterwards can repeat them after a crash. The committed offset is a restart position, not proof that every external system was updated.

topic: orders
partition 0: offset 0 [order A v1]  offset 1 [order A v2] // (1)
partition 1: offset 0 [order B v1]
group invoicing: consumer X -> p0, consumer Y -> p1     // (2)
group analytics: consumer Z -> p0 and p1                // (3)

Producer retries, idempotent-producer settings and in-flight limits affect order preservation. Parallel handling within a partition or skipping failed records can reorder business effects. See Kafka's design documentation.

#RabbitMQ, Kafka and Azure Service Bus

RabbitMQ also supports Streams; this comparison concerns traditional queues. Choose using replay, routing, SLA, operational skills and cost rather than universal speed claims. Service Bus duplicate detection has bounded scope and time; it does not guarantee exactly-once business effects. See the Service Bus overview.

#Transactional outbox and inbox

Saving and then publishing are separate writes: a crash between them loses the event, while publishing first can announce an order that never commits. Store the business change and outbox record in one database transaction, then dispatch later. At the consumer, persist the message ID and effect in one transaction.

// EF Core fragments; entities and serializer are application code.
await using var tx = await db.Database.BeginTransactionAsync(ct);
db.Orders.Add(order);
db.Outbox.Add(new OutboxMessage(messageId, Serialize(order))); // (1)
await db.SaveChangesAsync(ct);
await tx.CommitAsync(ct);

// Dispatcher: claim a row safely before publishing; broker confirms required.
await PublishWithConfirmAsync(row.Id, row.Payload, ct);       // (2)
row.MarkSent();
await db.SaveChangesAsync(ct);

// Consumer uses a different DbContext and transaction.
await using var receiveTx = await db.Database.BeginTransactionAsync(ct);
db.Inbox.Add(new InboxMessage(consumerName, messageId));      // (3)
await db.SaveChangesAsync(ct);
await ApplyDatabaseEffectAsync(db, message, ct);             // (4)
await db.SaveChangesAsync(ct);
await receiveTx.CommitAsync(ct);
// Only now acknowledge the delivery.

#Delivery guarantees, idempotency and ordering

Assigning a fixed status may be idempotent but does not prevent a stale event from moving an order from Shipped back to Paid. Use per-aggregate versions or sequences, and define gap rejection, buffering or reconciliation.

In RabbitMQ, one queue with a single active consumer and sequential handling limits reordering; requeues, retries and multiple producers require further rules. Kafka orders within a partition, not across partitions. An ordered outbox must stop later rows overtaking a failed row for the same order. Global order has a coordination cost.

#Sagas: choreography and orchestration

Reserving stock and collecting payment across services cannot use one local transaction. A saga tracks local transactions and requests compensations such as release or refund after failure. Compensation is a new business action, rather than a magical rollback or erasure of a payment.

Pending -> StockReserved -> PaymentCaptured -> Confirmed // (1)
StockReserved + PaymentRejected -> ReleasingStock -> Cancelled
PaymentCaptured + FulfilmentFailed -> Refunding -> Refunded // (2)

If invariants require immediate enforcement, consider one transactional boundary before introducing a saga. Sagas do not supply isolation; explicit pending states and reservations help prevent races.

#Resilience and messaging frameworks

A slow dependency can consume available connections and execution capacity. Retries with jitter handle transient failures; timeouts bound waiting; circuit breakers temporarily reject calls after repeated failures; bulkheads isolate concurrency capacity. Do not retry validation failures or repeat unsafe charges without an idempotency contract.

// Package: Microsoft.Extensions.Http.Resilience; Polly v8-based integration.
using Microsoft.Extensions.Http.Resilience;

builder.Services.AddHttpClient("inventory")
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.MaxRetryAttempts = 3;
        options.Retry.UseJitter = true;                   // (1)
        options.Retry.DisableForUnsafeHttpMethods();     // (2)
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(10); // (3)
    });

MassTransit and Wolverine offer consumer pipelines, retries, durable messaging and saga facilities through their chosen integrations. Configure persistence, outbox and failure handling explicitly; an in-memory outbox is not a durable transaction. The HTTP handler protects outgoing HTTP, rather than defining consumer retries. See HTTP resilience, MassTransit outbox and Wolverine durability.