Log streamer and append-only feeds
Serving a growing event stream without rewriting history.
Log streamer and append-only feeds
Business constraint
Operators want a live tail of job logs. Jobs can emit MBs/minute. Multiple viewers attach mid-stream.
Naive v0
Buffer all logs in memory per job. Replay from byte 0 on every connect. Block producers when a slow viewer cannot keep up.
Failure drill
Memory grows with job lifetime. One stuck SSE client stalls the writer. Catch-up storms when a popular job finishes and everyone reconnects.
Evolution path
Iteration 1
Append-only segment files (or ring buffer) with offsets. Clients resume from last offset. Drop/disconnect slow consumers instead of blocking producers.
Iteration 2
Cap retained bytes per job; spill older segments to object storage. Index by offset for catch-up.
Iteration 3
Fan-out via an in-process hub for live subscribers; cold catch-up reads segments. Apply backpressure by closing slow sockets.
Implementation cut
Treat the log as a cursored stream. Document retention. Prefer disconnecting slow readers over stalling writers.
Numbers
Producer never blocks >N ms on fan-out. Retention bound in bytes/time. Catch-up throughput measured separately from live lag.
Pattern tags
Self-check
- Why is "buffer everything" wrong for long jobs?
- What is an offset good for?
- How do you protect producers from slow consumers?
- When do you spill to object storage?
- What breaks if clients cannot resume?
Walkthrough
Recordings will appear here when published. Until then, work the failure drill and lab locally — pause after each iteration and write the metric that moved.