Hot shards

Business constraint

A celebrity user or SKU concentrates traffic on one shard. Average utilization looks fine; one shard melts.

Naive v0

More replicas of the same shard keying. Or ignore per-shard metrics.

Failure drill

p99 latency from one shard. Retries amplify. Rebalancing moves the hotspot without changing the key.

Evolution path

Iteration 1

Per-shard metrics and alerts. Detect skewed keys.

Iteration 2

Split hot keys (salting, subshards) or cache with single-flight in front.

Iteration 3

Consistent hashing with bounded load; planned reshard drills.

Implementation cut

Average CPU is a vanity metric. Practice key salting carefully with read path fan-in.

Numbers

Max/mean shard QPS ratio. Hot key share of traffic. Reshard impact window.

Pattern tags

• hot-key-tactics
• consistent-hashing

Self-check

1. Why can fleet averages lie?
2. What is key salting?
3. How does caching help hot keys?
4. What does bounded-load hashing change?
5. What do you drill before a viral event?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.