Why LLM memory gets expensive (and what to cut first)
Context windows are not free RAM — every token you keep multiplies cost, latency, and failure modes.
Why LLM memory gets expensive (and what to cut first)
“Just stuff more into the prompt” feels like caching. It is closer to shipping your entire working set across a paid, lossy bus on every turn.
Cost drivers
- Prompt tokens — billed every request; history grows without bound unless you trim.
- Attention / latency — long contexts are slower even when the model “supports” them.
- Wrong recall — large bags of memories increase the chance you retrieve noise.
Fixes that are systems work
- Rank and pack memory (relevance × freshness × size), do not FIFO forever.
- Summarize cold history; keep hot facts as structured records.
- Separate agent home (durable loop state) from per-task scratch (disposable).
- Measure $/successful task, not just tokens/day.
Companion reading: Why An LLM’s Memory Gets Expensive.