Long context windows changed the default answer to 'how do we give the model this information' from 'build a retrieval pipeline' to 'paste it in'. For prototypes that is a genuine improvement. For production systems it quietly relocates the cost, because you now pay for the entire context on every single turn, and quality does not scale with the amount of text you supply.
Context engineering is the discipline of deciding what occupies the window and in what order. It is where most of the cost and a surprising amount of the quality of an LLM feature actually live.
Structure the prompt for the cache
Prompt caching is the highest-return change available in most LLM applications, and it depends entirely on prefix stability. The cache matches from the start of the prompt, so everything static must come first and everything variable last. Put a timestamp at the top of your system prompt and you have disabled caching for every request, permanently, for no benefit.
# Order matters: stable prefix first, volatile content last.
messages = [
{'role': 'system', 'content': SYSTEM_PROMPT, # never changes
'cache_control': {'type': 'ephemeral'}},
{'role': 'user', 'content': tool_catalogue}, # changes on deploy
{'role': 'user', 'content': tenant_policy_document, # changes per tenant
'cache_control': {'type': 'ephemeral'}},
{'role': 'user', 'content': conversation_summary}, # changes per session
{'role': 'user', 'content': user_question}, # changes per turn
]
# Anti-patterns that silently destroy the cache:
# - a generated timestamp or request ID in the system prompt
# - retrieved chunks inserted ABOVE the static policy text
# - reordering tool definitions between calls (sort them)On an agent loop, where the same large prefix is resent on every step, getting this right took one of our features from unviable to routine — the marginal cost of a step became the new tokens, not the whole conversation. If you do one thing to an LLM system this quarter, sort your prompt by volatility.
Compaction beats truncation
Long-running sessions eventually exceed any window. The naive fix is dropping the oldest turns, which reliably discards the constraint the user stated at the start and keeps the small talk from two minutes ago. Compaction is better: summarise the middle of the conversation into a structured brief — decisions made, constraints stated, entities in play, open questions — and keep the first turns and the last few verbatim.
- Summarise into a schema, not prose. Prose summaries lose exactly the specifics you needed to keep.
- Preserve the opening turns verbatim; that is where the task definition and the constraints live.
- Compact on a token threshold, not a turn count. Turn count is a poor proxy for context pressure.
- Keep tool results out of history once consumed — a fetched document that has been acted on is dead weight.
- Log what was dropped. 'The assistant forgot' is almost always a compaction bug, and undebuggable without it.
Retrieval did not become obsolete
The uncomfortable finding from every long-context evaluation I have run is that quality degrades well before the window fills. Information in the middle of a very long context is used less reliably than information near the edges, and irrelevant material actively distracts. Twelve well-chosen chunks consistently beat two hundred pages, and cost two percent as much.
A large context window is a bigger desk, not a better filing system. What you put on it still decides the outcome.
The reasonable synthesis: use long context to remove the fragility of aggressive chunking — retrieve sections instead of sentences, keep whole documents when they are genuinely small — and keep retrieval to decide what is worth including at all. Selection is the expensive judgement. The window size only changes how much slack you have when you get it slightly wrong.