Multi-tier order flows

Business constraint

Place order: reserve inventory, charge payment, enqueue fulfillment. Partial failure must not strand money or stock.

Naive v0

Distributed transactions across three systems. Or best-effort sequential calls with no saga/compensation.

Failure drill

Charged but not reserved. Reserved but not charged. Double charge on retry. Fulfillment ships twice.

Evolution path

Iteration 1

Order aggregate as source of truth + outbox events. Workers drive steps idempotently.

Iteration 2

Reservations with leases/TTL; payment with idempotency keys; fulfillment claimed once.

Iteration 3

Compensation paths documented and tested (release reserve, refund).

Implementation cut

State machine in the order service. Every external effect keyed. Chaos-test mid-step crashes.

Numbers

Duplicate charge rate ≈ 0. Orphan reservation rate under TTL. End-to-end success after retry.

Pattern tags

• outbox-idempotency
• lease-fencing

Self-check

1. Why avoid multi-system 2PC here?
2. What does the outbox guarantee?
3. How do reservation leases help?
4. What must be idempotent?
5. What compensations do you need?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.