Scalable load balancer thinking

Business constraint

You outgrow a single reverse proxy. Need horizontal LB behavior without melting backends on scale-out.

Naive v0

Equal round-robin to all targets including brand-new ones. No slow start. Health checks only TCP.

Failure drill

Cold caches and JIT warmups get slammed. Flapping unhealthy instances flap the pool. Consistent-hash peers reshuffle too much on one node loss.

Evolution path

Iteration 1

Active health checks that exercise the real dependency path. Slow-start new targets. Connection draining on remove.

Iteration 2

Consistent hashing for cache-friendly affinity where it helps; plain RR for stateless CPU-bound work.

Iteration 3

Protect origin with concurrency limits and retry budgets at the LB layer.

Implementation cut

Write the failure modes you refuse (herding onto cold nodes, retry amplification). Match LB algorithm to stateful vs stateless.

Numbers

Error rate during scale events. Cache hit rate with/without affinity. Drain time vs deploy SLO.

Pattern tags

• consistent-hashing
• soft-ttl

Self-check

1. Why is TCP health often insufficient?
2. What is slow start for?
3. When prefer consistent hashing?
4. How do retries at LB amplify outages?
5. What is connection draining?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.