Task scheduler

Business constraint

Run millions of scheduled tasks. Workers scale horizontally. Tasks must not double-fire effects carelessly.

Naive v0

One cron process. Or SELECT due tasks without claiming. Or claim without fencing.

Failure drill

Missed fire on single-node death. Double execution after timeout. Hot "due now" scans.

Evolution path

Iteration 1

Task rows with run_at, claim via lease + fencing (lab task-scheduler).

Iteration 2

Index by due time carefully; avoid thundering herd on the same second with jitter.

Iteration 3

Idempotent handlers; poison retries with backoff.

Implementation cut

At-least-once delivery + idempotent effects. Prove zombie claim rejection in tests.

Numbers

Claim conflict rate. Overdue age histogram. Duplicate effect rate ≈ 0 with idempotent keys.

Pattern tags

• lease-fencing
• outbox-idempotency

Self-check

1. What does claiming change vs naive select?
2. Why fencing on leases?
3. How do you reduce due-time herds?
4. What does at-least-once imply?
5. How are poison tasks handled?

Walkthrough

Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.

Lab package: see /systems/labs/task-scheduler/.