Distributed leases

Business constraint

Only one worker should process a given shard of jobs. Workers crash. Clocks drift. Networks partition.

Naive v0

SETNX lock with TTL. Crashed holder expires; new holder starts. Old holder wakes and also writes.

Failure drill

Split-brain writes. Duplicate side effects. "I thought I still held the lock" bugs that only show in chaos drills.

Evolution path

Iteration 1

Lease includes a monotonically increasing fencing token. Storage rejects writes with stale tokens.

Iteration 2

Heartbeat renews lease before expiry with skew safety margin. On renew failure, stop work immediately.

Iteration 3

Store intents under the fencing token. External systems that can carry the token should validate it.

Implementation cut

Build the lease table/API first in lab. Prove zombie write rejection with a unit test before wiring real jobs.

Numbers

Lease length ≫ renew RTT, ≪ max acceptable duplicate window. Measure renew failures and fenced rejections in prod metrics.

Pattern tags

• lease-fencing
• leader-election

Self-check

1. What does fencing add over TTL locks?
2. How do you choose lease duration?
3. What should a worker do on renew failure?
4. Where must the token be checked?
5. How does this relate to leader election?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.

Lab package: see /systems/labs/lease-lock/.