Cache miss storms

Business constraint

A product detail page is cached. Marketing drops a campaign on one SKU. Cache TTL is 60s. Origin rebuild is 80ms under light load.

Naive v0

On miss, every request hits the origin. No lock, no soft expiry. Negative lookups also miss every time.

Failure drill

At expiry, thousands of concurrent readers stampede. Origin CPU saturates. Timeouts cascade to the edge. Hit rate looks fine until the cliff — then everything burns together.

Evolution path

Iteration 1

Add single-flight / request coalescing per key: one fill, others wait. Cap waiters; excess fail fast or serve stale.

Iteration 2

Soft TTL: keep serving stale while one refresher runs. Jitter hard TTLs so keys do not align on wall-clock seconds.

Iteration 3

Negative cache short-lived misses. Measure origin QPS per key during drills, not only global hit rate.

Implementation cut

Implement coalescing in-process first (lab cache-coalesce). Document multi-node: local singleflight alone is not enough — need shared lease or probabilistic early refresh at the fleet.

Numbers

During expiry storms, origin calls per hot key should collapse toward 1 (or O(replicas) without a shared lock). Fill latency p99 stays within origin SLO; reader p50 stays near cache hit.

Pattern tags

• single-flight
• soft-ttl

Self-check

1. What does single-flight guarantee and what does it not?
2. Why measure per-key origin QPS?
3. How does soft TTL differ from longer hard TTL?
4. What is negative caching for?
5. When do you still need a shared lock across nodes?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.

Lab package: see /systems/labs/cache-coalesce/.