Event ingestion

Business constraint

Devices and services emit events at high rate. Downstream analytics and product features consume them.

Naive v0

HTTP handler writes directly to many databases. Or accept without durable log and hope consumers keep up.

Failure drill

Loss on crash after 200 OK. Duplicates without keys. Head-of-line blocking on one poison event.

Evolution path

Iteration 1

Append-only intake log first; ack only after durable. Consumers checkpoint offsets.

Iteration 2

Idempotency keys for producers. Poison quarantine. Parallelism by partition key.

Iteration 3

Backpressure to producers when lag exceeds SLO.

Implementation cut

Durability before fan-out. Define ordering per key, not globally, unless required.

Numbers

Loss rate ≈ 0 after ack. Lag watermark. Dedupe hit rate. Poison rate.

Pattern tags

• append-only-log
• outbox-idempotency

Self-check

1. When do you ack to the producer?
2. What ordering can you claim?
3. How do poison events get handled?
4. Why partition keys?
5. What metric triggers backpressure?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.