ADR 0014 — Ordered processing and convoys¶
Status: Proposed (design)
Date: 2026-08-14
Related: ADR 0006,
ADR 0007,
ADR 0013,
RabbitMQ,
issue #135
Context¶
Operators (and BizTalk migrants) need in-order processing for a business key (order id, IBAN, device id) and a story for convoys: related messages that must not overtake each other. A later message in the same convoy must not start while an earlier one is still running, retrying, or sitting on the DLQ.
Today MeshFlows does not guarantee order:
- AMQP trigger dispatch (
POST /invoke/amqp/dispatch) can start several workflows for one publish (1→N). - Orchestrator queue workers use prefetch > 1 (
ORCH_WORKER_PREFETCH_SMALLdefault 20) and multiple replicas. - Subscriber ACK is after a synchronous Orchestrator invoke (ADR 0013).
- Retry / Dashboard DLQ replay can re-offer a message after a newer message for the same key has already started.
correlation_id exists on the envelope but is used as a trace/run id, not as
a FIFO session.
BizTalk-style parallel convoys (wait until all members of a correlation set arrive) need dehydrate/resume (ADR 0007) and are out of scope for the first implementation slice.
Decision¶
1) Ordering is opt-in, per key — never global FIFO¶
Default behaviour stays parallel (throughput). Ordered processing is declared on the AMQP trigger (and forwarded onto the run envelope):
invocation:
type: amqp
ordering:
key: header:OrderId # or context:amqp.headers.order_id / CEL later
on_dlq: block # block | skip (see §4)
Messages without a resolvable key keep today’s parallel path.
There is no total order across unrelated keys, workflows, or the whole broker.
2) Sequential convoy = single-flight per order_key¶
For a given (workflow_name, order_key):
- At most one run is in-flight (
accepted/queued/running/interrupted). - Later messages for the same key are parked in arrival order until the
in-flight run reaches a terminal status (
completed,failedafter final retry policy,cancelled). - Arrival order = broker delivery order for that consumer path, recorded when
the message is accepted (monotonic
seqper key).
This is the MeshFlows mapping of a BizTalk sequential convoy.
3) Hold the convoy in Postgres, not a second broker¶
Stay on RabbitMQ (ADR 0006). Do not introduce Kafka partitions or a new broker for FIFO.
v1 store (next to run_state_store):
| Table (logical) | Role |
|---|---|
message_order_lock |
one row per (workflow, order_key) with current run_id / status |
message_order_hold |
parked envelopes FIFO (seq, payload/ref, headers) |
Worker loop for an ordered message:
- Resolve
order_key. If empty → existing parallel execute. - Try to acquire the lock. If free → start run, ACK/complete as today.
- If held → insert into
message_order_hold, ACK the RabbitMQ delivery (so the broker is not a second timeout clock), do not start the workflow. - When a run for that key becomes terminal → dequeue the next hold row and
enqueue it onto
orchestrator.run.{small|large}as today.
Parking after ACK is intentional: RabbitMQ consumer timeouts must not release a convoy member out of order. The hold table is the source of truth.
4) Redelivery inside a convoy¶
| Event | Behaviour |
|---|---|
| In-flight run retries (workflow retry / queue retry) | Key stays locked. Held messages wait. |
| In-flight run goes to DLQ | on_dlq: block (default): lock remains; hold queue does not advance until an operator retries or skips. on_dlq: skip: mark that seq abandoned, start next hold. |
| Operator retries from DLQ (#110 Dashboard) | Re-enter at the front of that key’s FIFO (original seq), never appended as a new last message. |
| Duplicate / idempotent re-offer of seq N while N is in-flight | No second run; existing idempotency key wins. |
| Duplicate of seq N after N completed | No-op (idempotency), do not unlock out of order. |
Do not start message N+1 while N is on the DLQ unless on_dlq: skip.
5) What this is not¶
- Not a peek-lock complete step for RabbitMQ (ADR 0013 stays).
- Not parallel convoys / correlation-set dehydrate (follow ADR 0007).
- Native Service Bus topic publish + subscription peek-lock/complete is a connector (#127); triggers are ADR 0015 option C.
- Not prefetch=1 on the shared work queue as the product solution (that only serializes everything on one replica and still races with DLQ replay).
First implementation slice (not this PR)¶
- Trigger YAML
ordering.key+ envelope headerX-MeshFlows-Order-Key. - Postgres lock + hold + release-on-terminal.
- Tests: two messages same key with prefetch > 1 complete in publish order; DLQ of N blocks N+1 until retry; different keys still overlap.
Dashboard UX for “blocked keys” can wait; DLQ retry already exists.
Consequences¶
- Throughput for unordered traffic is unchanged.
- Ordered workflows trade latency/throughput for correctness per key.
- Operators must pick a stable key; a missing key silently falls back to parallel (log a warning, do not fail the trigger).
- Parallel convoys remain a documented gap until checkpoint/resume exists.
See also¶
- Messaging gaps: Capability comparison
- DLQ operator slice: issue #110