Async pipelines and backpressure
Queues that protect producers from drowning consumers — and vice versa.
Async pipelines and backpressure
Business constraint
Upload → virus scan → thumbnail → notify. Product wants "fire and forget" after the HTTP 202.
Naive v0
Goroutine per upload with unbounded channels. Or drop straight into a shared queue with no consumer concurrency limit. Retries without idempotency keys.
Failure drill
A slow scanner backs up memory. Process OOM. Or the queue grows for hours and notify latency becomes days. Duplicate notifies spam users after a broker redelivery.
Evolution path
Iteration 1
Bound in-process queues; when full, apply backpressure to the accept path (429/503) or spill to durable storage with explicit lag SLOs.
Iteration 2
Stage the pipeline with idempotent steps and an outbox: accept writes a durable intent, workers claim with leases.
Iteration 3
Separate critical path (scan) from best-effort (notify). Observe lag, retry age, and poison-message rate.
Implementation cut
Design each stage as: claim → process → ack. Define poison handling. Never "infinite buffer = infinite patience."
Numbers
Queue depth alert before memory cliff. End-to-end p99 lag budget per stage. Duplicate notify rate near zero with idempotency keys.
Pattern tags
Self-check
- What is backpressure in one sentence?
- Why is unbounded buffering a silent outage?
- How does an outbox help dual-write races?
- What makes a stage idempotent?
- When should accept reject vs spill?
Walkthrough
Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.