Your agent doesn't fail because the model is weak. It fails because the context it runs on is broken. Here is the 5-layer architecture every production agent needs, and the 5 engineering rules that separate builders who ship from builders who debug forever.
7 min · The AI How
5context layers
4,000Ttools before turn 1
152turns = 200K ctx
10×relevance vs recency
The architecture
The 5 layers of agent context
Every token your agent sees falls into one of these five layers. Engineering each layer deliberately is what separates a prototype from a production agent.
01InstructionsYour system prompt. The only thing you fully control.
~2,000–8,000Tper session
Written once, read every turn — make every token earn its place
Vague instructions are soft suggestions; precise ones are enforced constraints
This is where your spec lives — or should
Engineering rule
Spec, not suggestion
"be brief" is ignored. max_tokens: 200 is enforced.
02MemoryWhat your agent remembers across sessions.
0Twithout an external store
Session context is wiped on close — nothing survives without an external store
User preferences, past decisions, long-term constraints all need explicit storage
Vector DB, key-value store, or plain file — any store beats no store
Engineering rule
Relevance, not recency
Semantic retrieval is 10× cheaper than loading the last N turns.
03StateWhat your agent knows right now, in this turn.
INFERREDif you don't write it explicitly
State is current step, pending approvals, last tool call, open constraints
Agents that infer state from history re-read 40+ turns every loop
Write state explicitly. Agents that read history to understand state will loop and lose the thread.
Engineering rule
Checkpoint every 10 turns
Capture state before drift. Recoverable at any checkpoint.
04ToolsFunction definitions loaded into context before your agent acts.
~200Tper tool definition
Every tool schema (name, description, input_schema) loads into context automatically
20 tools = ~4,000 tokens consumed before the first agent call
Audit your tool list. Every unused tool is wasted context budget.
Engineering rule
Cap at 3,000 tokens
Default tool schemas balloon to 12K+. Cap descriptions at 3K. Stable agents.
05HistoryEvery prior turn — user messages, assistant replies, tool calls.
152 turnsfills a 200K context window
History grows linearly every turn — unmanaged, it fills your context window
At 1,200 tokens/turn, 152 turns saturates 200K tokens completely
The model compacts history automatically — but it drops constraints silently
Engineering rule
Prune before the model does
Model compaction drops constraints randomly. You drop strategically.
The cheatsheet
5 engineering rules
One rule per layer. Memorise these and your agents will be materially better before you write a single line of new code.
Semantic search retrieves the right turn. Last-N retrieves noise. 10× cheaper.
03
Checkpoint every 10 turnsState
Captures explicit state at T:10, T:20, T:30. Drift is recoverable.
04
Cap at 3,000 tokensTools
Default schemas reach 12K+. A 3K cap stabilizes context budget every turn.
05
Prune before the model doesHistory
Compaction silently degrades constraints. Prune yourself — keep what matters.
The compaction problem
The model will compact your history. It will not ask permission.
When your context window fills, the model summarises prior turns automatically. Precise constraints like no_external_calls become vague suggestions like "maybe avoid external calls when possible". One rule in four gets dropped entirely. This is silent behaviour change. The fix is to prune your history before the model gets the chance.
3 of 4
constraints degraded
1 of 4
dropped silently
Coming next
Part 2 — Specs as context
The spec is the highest-leverage document you can write for an AI agent. One spec, written once, governs all 5 layers simultaneously. Part 2 shows exactly how to write one.