Context Engineering for AI Agents: How to Architect LLMs

An agent resolves the first message of a conversation cleanly. By message nine it quotes a superseded policy, forgets the identifier it verified two turns earlier, and calls a write tool with arguments it already spent. The model weights did not change between those turns. The context did.

That failure mode defines the discipline. The instruction was fine on turn one and stayed fine, so no amount of prompt editing repairs turn nine. What broke is the information state around the instruction, and engineering that state across turns, under a token budget, is context engineering.

The two systems below run the same model on the same task. Only the context pipeline differs.

Naive context stuffing

Everything the agent saw, pasted every turn

• Full transcript plus whole documents in append order

• Decisive facts land mid window where attention is weakest

• Prefill cost grows quadratically and every turn pays it again

• State lives in prose, so the model re-derives it each call

• Fails quietly through contradiction, truncation, and drift

Token spend per turn: the full history. Recall at mid window positions: lowest on the curve.

Engineered multi tier context pipeline

Selected, ranked, and verified before the call

• Hybrid retrieval picks chunks and a reranker orders them

• Stable prefix cached once, volatile evidence placed last

• A structured scratchpad carries state as typed fields

• Budget counters decide what gets pruned or summarized

• Fails loud: low confidence triggers retrieval or handoff

Token spend per turn: cached prefix plus the delta. Every claim in context carries a source id.

What Separates Context Engineering from Prompt Engineering

The card shows two builds of the same agent with opposite behavior, and naming the difference precisely matters because the two disciplines budget different resources and fail in different ways.

The term crystallized in June 2025. Dex Horthy's HumanLayer essay argued that context engineering is the core job of engineers building agents, and Tobi Lütke's post on X that month said the phrase names the real skill better than prompt engineering, because the job is providing all the context a task needs to be plausibly solvable by the model.

Harrison Chase's June 2025 essay for LangChain defined it as building dynamic systems that supply the right information and tools in the right format. Andrej Karpathy sharpened it days later. His June 25, 2025 post on X gave the field the line most engineers now quote:

"Context engineering is the delicate art and science of filling the context window with just the right information for the next step."

Karpathy's LLM OS framing came with it: the model is a new kind of CPU and the context window is its RAM, the working memory of a new class of computer. Anthropic's engineering blog gave the idea its production treatment in 2025 with Effective context engineering for AI agents, arguing for the smallest set of high signal tokens that produces the behavior you want.

The artifact difference is the cleanest separator. A prompt is a static string: you can diff it, version it, and A/B test it, and the search space is the vocabulary of instructions. Context is a per turn snapshot assembled by code: a stable prefix, a memory state, ranked evidence, compressed tool outputs, and budget counters. The search space is not vocabulary but the selection, ordering, compression, and expiration of information under a hard token limit.

Prompt engineering shapes how the model thinks. Context engineering controls what the model knows right before it acts.

The algorithmic difference changes how you debug. A prompt defect reproduces with one call and one string. A context defect needs the full trace: what retrieval returned, how the reranker ordered it, what the scratchpad held at step four, and which tokens were evicted at the watermark. You debug the pipeline, not the sentence.

Once context becomes a pipeline, it inherits pipeline failure modes. The first one sits inside the attention mechanism itself, before any of your code runs.

The Physics of Context Degradation in LLM Agents

Context fails in three physical ways: where information sits in the window, how much noise surrounds it, and what it costs to keep it there. Each has measurements and closed form math behind it, and each dictates a design rule downstream.

Lost in the Middle and the U Shaped Attention Curve

Liu et al.'s Lost in the Middle study, published in TACL 2024, measured position bias directly. In multi-document question answering they moved the gold passage through every position of the window and scored accuracy at each one. The result was a U shaped curve: accuracy peaked when the answer sat at the very start or the very end and dropped sharply in the middle, in some configurations below the closed book baseline where the model received no documents at all.

A synthetic key value retrieval task showed the same pattern, and models marketed as long context were not immune. The mechanism has a human analogue in the serial position effect from memory research, where people recall the first and last items of a list better than the middle ones. Transformers show a computational version of it: attention is learned from natural text where openings and closings carry disproportionate weight, and causal masking keeps early tokens visible to everything that follows.

Models read the edges of the window first. The middle of a long prompt is the worst place to put a fact that matters.

Three placement rules follow. Put the instruction and the current question at the edges, never in the middle. Place the highest ranked evidence last, closest to the question, where recency attention is strongest. When you must truncate history, cut from the middle and keep both ends intact.

Context Rot and Multi Turn Entropy Drift

Position bias explains where degradation happens. Context rot explains when.

Chroma's 2025 context rot study evaluated 18 frontier models and found that accuracy on simple retrieval and reasoning tasks decays as input length grows, well inside the advertised window. Longer inputs did not fail at the limit. They failed gradually, from the first thousand distractor tokens onward.

Practitioners reached the same conclusion from the other direction. The recurring complaint in r/LocalLLaMA threads on long context models is that a clean needle-in-a-haystack score predicts almost nothing about real work, because finding one unique string in a pile of distractors is the easiest retrieval a model will ever do.

NVIDIA's RULER benchmark put numbers on that gap in 2024. Across 17 long context models that scored near perfect on the vanilla needle test, only half held usable accuracy at 32K tokens once tasks required multiple needles, multi-hop tracing, or aggregation. Almost all of them fell below the threshold before reaching their claimed window.

A needle-in-a-haystack score is not an effective context length. Multi-needle and multi-hop tests put the real number well below the marketed one.

Multi-turn loops make it worse. Laban et al.'s 2025 multi-turn study tested 15 models across six tasks and found an average performance drop of 39 percent when the same task arrived as an underspecified conversation instead of a single complete instruction. The transcript itself becomes the problem: a hallucinated step written at turn three becomes context at turn four, and the model treats its own earlier mistake as evidence.

The mechanism is normalization. Softmax attention divides a fixed budget of weight across every position, so each appended tool result, retry, or dead end claims probability mass. The attention distribution over the tokens that matter flattens as the window fills, and you can write that flattening as the entropy of the attention distribution at turn t:

H(p^(t)) = - Σ_i  p_i^(t) · log p_i^(t)

As distractor mass accumulates, H rises and focus spreads from signal toward noise. Every turn is then conditioned on a slightly noisier state than the last, which is how an agent stays fluent while getting less accurate.

Every token you add to the window taxes every other token, because softmax divides a fixed weight budget across all positions.

Attention Compute and KV Cache Economics

Degradation is also a budget problem with closed form math. During prefill, self-attention scores every query against every key, so attention compute scales as O(N^2 · d) per layer for N tokens of dimension d. Double the window and prefill compute quadruples. During decode, each new token attends against cached keys and values at O(N · d) per token, but the cache itself grows linearly with N:

kv_bytes = 2 · L · H_kv · d_head · N · b

The factor of 2 covers keys and values, L is the layer count, H_kv the number of key value heads, d_head the head dimension, and b the bytes per element. Concrete shapes make the cost visceral.

DeepSeek V3's Multi-head Latent Attention compresses keys and values into a shared latent vector per layer, which is why its per token cache lands near a fifth of the 70B shape. The 70B row means a single 100K token conversation holds about 32 GB of cache before you batch a second user.

Doubling the window quadruples prefill compute and doubles the cache. Context size is a hardware bill before it is a quality choice.

Prefix Caching Architecture

Serving stacks attack the cost with reuse. vLLM's PagedAttention, published in 2023, manages the KV cache in OS style pages so sequences share blocks instead of reserving contiguous memory. SGLang's RadixAttention, described in the 2024 SGLang paper, keeps the cache in a radix tree and shares any matching prefix across calls automatically.

Providers expose the same idea as a billing feature. Anthropic's prompt caching lets you mark cache breakpoints and bills cached reads at a fraction of base input price, and OpenAI's prompt caching discounts cached prefix tokens automatically on long prompts.

The design rule follows directly: stable content first, volatile content last. System instructions, policies, and tool schemas sit in the head and stay byte identical across calls. Retrieved evidence, the scratchpad, and the current message go at the tail, which is also where recency attention is strongest.

Manus quantified the payoff in their 2025 context engineering post: their agents run about 100 input tokens for every output token, so the cache hit rate on the prefix dominates the cost of the whole system. A rewritten head invalidates the cache on every call, which is why several rules in the later pillars exist to protect prefix stability.

Keep the head of the prompt stable and the tail fresh. Prefix caching rewards the first and attention rewards the second.

Physics sets three constraints: position bias, entropy drift, and quadratic cost. The first pillar works inside all three by controlling what enters the window at all.

Pillar 1 Dynamic Hybrid Retrieval and Contextual Chunking

Static retrieval embeds the user's message, takes the top k chunks by cosine similarity, and pastes them in. That pattern breaks for agents within a day, because agent queries are generated mid task, chained across steps, and full of exact identifiers. Dynamic retrieval treats every step as a new search problem with its own query construction, fusion, and ranking.

Query construction comes first, because an agent's raw step text is rarely the right query: it is long, polite, and full of irrelevant context. HyDE generates a hypothetical answer and embeds that instead, since answers sit closer to documents than questions do. Step-back prompting asks a more general question first to fetch framing context.

Decomposition handles the compound requests, splitting one instruction into atomic sub-queries that each retrieve cleanly. The rewrite costs one small model call and routinely matters more than the index underneath.

Hybrid Retrieval with BM25 and Dense Embeddings

BM25 scores a document by exact term overlap, weighted by inverse document frequency and normalized for document length:

score(q, d) = Σ_i IDF(q_i) · f(q_i, d) · (k1 + 1) / (f(q_i, d) + k1 · (1 - b + b · |d| / avgdl))

It is unforgiving and precise. An order id, an error code, or a clause number either appears or it does not, and BM25 ranks accordingly.

Dense embeddings do the opposite job: they map text into a vector space where paraphrases sit close together, so a query about getting money back matches a refund policy with zero shared tokens. Each method fails where the other succeeds. Embeddings wash out rare exact strings because an order number is noise in a semantic space, and BM25 cannot match a paraphrase it has never seen.

BM25 finds the exact string. Embeddings find the meaning. Agent queries mix identifiers with intent, so the pipeline needs both.

Reciprocal Rank Fusion

Merging two ranked lists is its own algorithmic problem, because BM25 scores and cosine similarities live on incompatible scales. Reciprocal Rank Fusion sidesteps calibration by fusing ranks instead of scores:

RRF(d) = Σ_lists 1 / (k + rank(d))     with k = 60

The constant k = 60 comes from Cormack, Clarke, and Buettcher's 2009 fusion study, and it damps the influence of any single list's top item so a document must be broadly ranked to win. A chunk that ranks second in BM25 and third in vector search outranks a chunk that leads one list and is absent from the other. RRF needs no training, no score normalization, and about ten lines of code, which is why it survives in production stacks while learned fusion gets abandoned.

Contextual Retrieval with Chunk Headers

Hybrid fusion still assumes the chunks themselves are self-contained, and they usually are not. A chunk that reads "customers receive a 10 percent credit" is meaningless without its parent document: which company, which policy version, which conditions.

Anthropic's Contextual Retrieval method from September 2024 fixes this at index time. Before embedding and BM25 indexing, a model reads the whole document and writes a short chunk specific context, typically 50 to 100 tokens, which is prepended to the chunk. The chunk becomes: "This passage is from the Acme refund policy, version 12, effective March 2026, covering partial credits for late delivery. Customers receive a 10 percent credit on the original order total."

Anthropic reported 49 percent fewer failed retrievals with contextual prepending and 67 percent fewer when combined with reranking. The cost is one offline generation pass per chunk, about $1.02 per million document tokens with prompt caching, and both indexes benefit: the embedding encodes the situated meaning and BM25 gains the exact names and dates the header adds.

A chunk without document context is a sentence without a subject. Fifty to 100 tokens of prepended context fix both indexes at once.

Cross Encoder Reranking

Retrieval so far has used bi-encoders, where query and document are encoded separately and search is a fast approximate dot product. A cross-encoder removes the separation: query tokens and document tokens pass through the transformer together, every query token attends to every document token, and the model outputs a single relevance score for the pair. A bi-encoder compares two compressed summaries, while a cross-encoder reads the query against the document, which is why it catches fine grained relevance the shortlist misses.

The cost is structural. A cross-encoder runs one full forward pass per candidate pair, so scoring a whole corpus is out of the question. The production pattern is a funnel: BM25 and dense retrieval return about 150 candidates between them, RRF fuses the lists, and the cross-encoder reranks the fused pool down to a top 20. Expect tens to a few hundred milliseconds of added latency, spent exactly where it pays: the final ordering of the few chunks that will consume window budget.

Placement is the last step, and the U curve from the physics section applies here. The strongest chunk goes last, closest to the question, and the second strongest goes first. The middle of the evidence block gets the weakest material or nothing at all.

Retrieve for recall and rerank for precision. The context window only has room for precision.

The full retrieval stack fits in four stages, each with a distinct job and a distinct cost profile.

Retrieval decides what enters the window on one turn. An agent runs for many turns, so the harder problem is deciding what survives between them.

Pillar 2 Agent Working Memory and Structured Scratchpads

A single-turn assistant can afford to treat the transcript as memory because its job ends with each reply. An agent cannot. It verifies an order at step one and issues a refund at step four, and between those steps it needs to know what is done, what is pending, and what it already checked.

The Three Tier Memory Hierarchy

Production agent memory splits into three tiers with different lifetimes, different writers, and different read paths.

The tiers protect each other. Working memory stays small because episodes are archived out of it. Episodic memory stays honest because it stores outcomes rather than raw transcripts. Semantic memory stays stable because nothing writes to it mid conversation.

Working memory answers what is happening now. Episodic memory answers what happened before. Semantic memory answers what is always true.

The Memory Scoring Function

A memory store needs a ranking rule, and the canonical one comes from Park et al.'s Generative Agents paper in 2023. Each memory m gets a retrieval score from three components:

s(m) = α_rec · recency(m) + α_imp · importance(m) + α_rel · relevance(m)

Recency is an exponential decay over the time since the memory was last touched. Importance is a 1 to 10 rating the model assigns when the memory is written, separating a mundane event from a decisive one. Relevance is embedding similarity to the current situation. The paper set all three weights to one, and production systems tune them: support agents usually raise importance so a verified constraint outranks a recent triviality.

A memory store without a scoring function is a pile. Recency, importance, and relevance turn it into a ranking.

A Production Scratchpad Schema

Working memory needs a concrete data structure, and prose is the wrong one. A transcript forces the model to re-derive state on every call and invites it to improvise the missing pieces. A structured scratchpad makes state explicit, diffable, and prunable. Here is a working schema for a refund agent, captured mid task:

{
  "task_id": "tkt_48217",
  "created_at": "2026-09-01T10:14:02Z",
  "goal": "Decide and execute a refund for order 1042",
  "customer": { "id": "cust_881", "verified": true, "tier": "standard" },
  "plan": [
    { "step": 1, "action": "verify_order", "tool": "shopify.get_order", "status": "done", "result_ref": "ev_01" },
    { "step": 2, "action": "check_refund_policy", "tool": "kb.search", "status": "done", "result_ref": "ev_02" },
    { "step": 3, "action": "issue_refund", "tool": "payments.refund", "status": "in_progress" },
    { "step": 4, "action": "confirm_and_close", "tool": "crm.update_ticket", "status": "pending" }
  ],
  "evidence": [
    { "ref": "ev_01", "source": "shopify.get_order", "fetched_at": "2026-09-01T10:14:20Z", "ttl_s": 900, "summary": "Order 1042 delivered 2026-08-18, total 84 USD" },
    { "ref": "ev_02", "source": "kb:refund-policy#v12", "fetched_at": "2026-09-01T10:14:31Z", "ttl_s": 86400, "summary": "Delivered orders refundable within 30 days of delivery" }
  ],
  "tool_log": [
    { "call_id": "c_101", "tool": "shopify.get_order", "status": "ok", "latency_ms": 240 },
    { "call_id": "c_102", "tool": "kb.search", "status": "ok", "latency_ms": 180 }
  ],
  "open_questions": [],
  "budget": { "token_budget": 24000, "tokens_used": 9120, "retrieval_calls_left": 3 },
  "confidence": 0.86,
  "handoff": { "intent": null, "done": [], "blocker": null, "ask": null }
}

Four design decisions carry the weight. Evidence entries store a summary plus a reference instead of the raw payload, so the full order JSON enters the window once, gets compressed, and leaves. Every evidence entry carries fetched_at and ttl_s, so the runtime expires stale state instead of letting it contradict fresher data. The budget block turns pruning into a policy decision, and the handoff block stays null until confidence drops.

A scratchpad stores pointers and summaries instead of payloads. Raw tool output enters the window once, gets compressed, and leaves.

Model Proposes and Runtime Disposes

The schema is half the design. The other half is who writes to it. The safe pattern is proposal plus validation: the model proposes a state change, such as marking step three done and attaching evidence ev_03, and the runtime validates the patch against the schema and the tool log before committing it.

A claimed refund with no matching payments.refund call in the log does not commit. Every field ends up with provenance, and you can replay the whole task from the patch history when something goes wrong.

The model proposes state changes and the runtime validates them against the schema. That split is what makes agent state debuggable.

Pruning runs on written rules. Compress on completion: when a step closes, replace its raw tool output with a one line summary and a reference. Evict by TTL before you evict by relevance, because expired data is wrong data. Never prune the goal, the constraints, open blockers, or unverified writes.

Working memory gives the agent a present tense. The next pillar isolates everything the agent touches through tools, so a single bad payload cannot poison the state you just designed.

Pillar 3 Tool Schema Isolation and Dynamic Injection

Retrieval decides what the agent knows. Tools decide what the agent can do. Tool state is context too: every schema sits in the prefix and bills every call, and every result passes through the window on its way to the scratchpad.

The Token Cost of Bloated Tool Definitions

A common failure is dumping full OpenAPI specs into the system prompt. A catalog of verbose schemas can spend thousands of tokens before the conversation starts, and each added tool is one more way for the model to pick wrong. Forty tools with overlapping descriptions produce selection errors that look exactly like reasoning failures, and the distraction compounds with the attention economics from the physics section.

Schema on Demand Routers

The fix is to load schemas lazily. Anthropic's code execution with MCP post from November 2025 measured the extreme case: loading every tool definition upfront and passing intermediate results through the window cost about 150,000 tokens in their example, while presenting tools as code APIs and loading definitions on demand cut it to about 2,000 tokens, a 98.7 percent reduction.

The general pattern is a router. Expose a small search or describe tool that lists capabilities, and load the full schema only when the agent selects a tool. The prefix stays small, and the tool space can grow without growing the prompt.

Load tool schemas on demand. A router that fetches definitions when selected keeps the prefix small no matter how large the tool catalog grows.

Mask Tool Tokens Rather Than Remove Them

The Manus post adds a counter-rule that surprises people: once a tool's tokens are in the context, keep them there. Removing or rewriting tool definitions mid-run changes the prefix and invalidates the KV cache on every later call, which at a 100 to 1 input to output ratio is the most expensive mistake available.

The alternative is logit masking. Keep the tool tokens stable and constrain the action space with a state machine that masks invalid tool names during decoding. The model physically cannot call a tool that the current state forbids, the prefix never changes, and the cache keeps paying off.

Mask, do not remove. A stable prefix with constrained logits beats a rewritten prompt on cost and cache hit rate.

Compact Schemas and Idempotency Keys

The schemas you do load should be small and strict. Keep names short and namespaced, params typed, descriptions to one line, and declare side effects so the runtime knows which calls need the write gate from Pillar 5. Every write tool takes a mandatory idempotency_key so a retried call produces the same result instead of a duplicate charge, the same contract Stripe documented for payment APIs.

{
  "name": "payments.refund",
  "description": "Refund a captured charge in whole or part",
  "params": {
    "order_id": { "type": "string", "required": true },
    "amount_cents": { "type": "integer", "required": true },
    "reason": { "type": "string", "enum": ["damaged", "late", "wrong_item", "other"] },
    "idempotency_key": { "type": "string", "required": true }
  },
  "returns": { "refund_id": "string", "status": "string" },
  "side_effects": "writes_money"
}

Every tool schema bills every turn. Short names, typed params, and a mandatory idempotency key keep the prefix cheap and retries safe.

Raw Output Isolation and Restorable Compression

Tool results deserve the same discipline as tool schemas. A raw order payload or an HTML page can run thousands of tokens, and most of it is noise for the decision at hand. Isolate it: the full payload goes to an object store, the window gets a one line summary plus a reference, and the scratchpad evidence entry points at the store.

Compression here is restorable, not lossy. When the agent needs the detail again, it re-fetches by reference instead of relying on a summary of a summary. This is the same pointer discipline as the scratchpad, applied to everything tools return.

Working state is now structured and tool state is isolated. The next pillar decides what happens when the window fills anyway.

Pillar 4 Attention Budgeting Pruning and Compaction

Every context pipeline needs a budget policy, because the physics section guarantees the window degrades as it fills. Attention budgeting turns the vague instruction to keep context small into thresholds, algorithms, and an ordered ladder of interventions.

Watermark Zones

Treat context occupancy like a reservoir with three zones. The exact thresholds are starting values to tune against your model's lost-in-the-middle curve, not universal constants.

The red zone rule is the one teams skip and regret: never make a model call while over the red watermark, because you are paying quadratic prefill prices for degraded attention.

Occupancy is a measurable quantity, not a feeling. Log tokens in, cache hit rate, and evidence count per call, then graph task accuracy against occupancy in your evals. The point where accuracy bends downward is your model's effective window, and it usually sits far below the marketed one.

Middle Out Truncation

When history must shrink, the naive choice is dropping the oldest turns. That throws away the original goal, which is the highest value token block in the window. Middle out truncation inverts it: preserve the system prompt, the first user turn that stated the intent, and the recency tail, then compress everything between.

The algorithm is mechanical. Pin the head and tail blocks, score middle blocks with the memory function from Pillar 2, replace low scoring blocks with one line summaries, and reassemble. The U curve explains why this works: you are deliberately spending the worst real estate in the window on compressed material.

Cut from the middle and keep both ends intact. The first user turn and the latest evidence are the highest value tokens in the window.

The Compaction Ladder

Truncation is one rung on a longer ladder. Run the cheap rungs first and climb only when pressure continues.

1. Tool result clearing. Drop raw tool outputs whose summaries already sit in the scratchpad. Anthropic ships this as context editing, and it loses nothing because the pointer survives.

2. Middle out truncation. Compress middle history blocks while the head and tail stay pinned.

3. Trajectory summarization. Replace the full trace with a model written summary of progress so far, the compact command pattern from coding agents.

4. Structured note taking. The agent writes durable notes to a file outside the window and reads them back on demand, which moves memory out of the token budget entirely.

5. Sub agent isolation. Hand a self-contained subtask to a fresh agent with a clean window and accept only its summary back.

Each rung trades fidelity for space. Trajectory summarization loses detail but keeps decisions. Sub agent isolation loses shared context entirely, which is why it sits at the top.

Semantic Compression with LLMLingua

A research line treats compression itself as a model task. LLMLingua, published at EMNLP 2023, compresses prompts coarse to fine with a small model, reporting up to 20 times token reduction with little loss on benchmarks. LongLLMLingua, published at ACL 2024, makes the compression question aware and reorders documents to fight the lost-in-the-middle curve directly.

Treat semantic compression as a lossy channel and pin what cannot be regenerated. Identifiers, amounts, dates, and negations must survive verbatim, because a compressor that paraphrases "not refundable" into "refundable" has negative value. Keep pinned spans out of the compressor's reach.

Compress the middle, never the edges, and never the identifiers.

Budgeting keeps one agent inside its window. The final pillar keeps the agent honest about what the window contains.

Pillar 5 Deterministic Verification and Attribution Grounding

The first four pillars control what the model sees. This one controls what the model is allowed to do with it, using gates that run in code rather than instructions that run in prose.

The Three Gate Verification Hierarchy

The first gate runs before generation: the attribution gate. Every factual claim in the draft answer must map to an evidence reference currently in context. No reference, no claim. This single rule kills most policy hallucinations, because the model cannot cite a refund window that retrieval never returned.

The second gate runs before execution: the schema gate. Tool arguments are validated against the schema and a policy layer before dispatch: amount bounds, allowed accounts, required confirmations. A refund above the threshold routes to approval instead of the payments API.

The third gate runs after execution: read-after-write verification. After any write, the runtime re-reads the system of record and compares the result against the intended effect. A refund call that returns success but leaves the ledger unchanged fails the gate and triggers repair instead of a confirmation message.

Trust is a gate, not a vibe. Claims need references, writes need schemas, and writes need reads.

Evaluation Metrics for Context Quality

What gets gated gets measured. Four metrics cover the pipeline, and the RAGAS framework standardized the first two.

Context precision and recall grade Pillar 1. Position relevancy grades the placement work from Pillar 4. Write verification rate grades the third gate, and it is the metric leadership should see, because it counts money touching actions that were confirmed rather than assumed.

The Four Part Handoff Payload

When confidence drops below threshold or the task exceeds scope, the agent hands off. The handoff is a structured payload rather than a transcript dump, and the same shape works for a human escalation and a sub agent return:

{
  "intent": "Refund order 1042 for a late delivery",
  "done": ["Order verified as delivered 2026-08-18", "Policy v12 confirms 30 day window"],
  "blocker": "Refund above the 50 USD auto approval limit",
  "ask": "Approve an 84 USD refund or counter with store credit"
}

Four fields force the discipline that matters: what the agent was trying to do, what is already verified, what stopped it, and what it needs next. A receiving human or parent agent can act in one read, and the handoff itself becomes an episodic memory when the task closes.

The five pillars are framework agnostic, but you do not have to build them from scratch. The open source ecosystem already implements each piece.

Open Source Framework Implementations and Ecosystem Patterns

The pillars map cleanly onto the four frameworks most teams evaluate first. Each makes a different bet on where context state should live.

LangGraph

LangGraph models an agent as a StateGraph: nodes are functions, edges are control flow, and state is a typed object whose fields are channels with reducer functions. A messages channel with an append reducer gives you the transcript, while custom channels carry the scratchpad fields from Pillar 2. Checkpointers persist state per thread to SQLite or Postgres, which gives you episodic memory, crash recovery, human interrupts, and time travel replay. LangGraph's bet: context is explicit graph state, and the runtime owns persistence.

LlamaIndex

LlamaIndex started as a retrieval framework, and its context features concentrate on Pillar 1. The QueryFusionRetriever generates several query variants from the input and fuses the result lists with RRF, which covers hybrid search and fusion in one component. The LongContextReorder postprocessor reorders retrieved nodes so the most relevant content sits at the start and end of the evidence block, a direct implementation of the U curve placement rule.

Letta

Letta, formerly MemGPT, treats the context window the way an operating system treats RAM. Core memory lives in labeled blocks such as persona and human, each with a character limit, and the agent edits its own blocks through tool calls. Archival memory is an unbounded store the agent pages into the window through retrieval, and recall memory covers raw conversation history. Letta's bet: memory management is the agent's own job, mediated by tools, with the framework enforcing the hierarchy.

DSPy

DSPy replaces hand written prompts with declarative signatures that specify inputs and outputs, then compiles them into concrete context. Its MIPROv2 optimizer proposes instructions and few shot examples, scores them against your metric, and searches the space with Bayesian optimization. DSPy's bet: the context construction program is itself optimizable, so you tune it with data instead of editing strings by hand.

No framework gives you all five pillars out of the box, and every one of them still needs your watermarks, your TTLs, and your gates.

Frameworks own the plumbing. You own the policy.

The questions below come up every time this pipeline is presented to a new team.

Frequently asked questions

What is context engineering in plain terms?

Context engineering is deciding what a language model sees before it acts: the instructions, the evidence, the memory, the tool results, and the order they arrive in. Prompt engineering writes the instruction. Context engineering builds the information state around the instruction, on every turn, under a token budget.

The term stuck because it names where agent systems actually fail. The instruction is usually fine. The state around it is what drifts, bloats, and contradicts itself by turn nine.

How is context engineering different from prompt engineering?

A prompt is a static string you can diff and A/B test. Context is a per turn snapshot assembled by code: prefix, memory state, ranked evidence, compressed tool output, and budget counters.

The debugging differs too. A prompt defect reproduces with one call. A context defect needs the full trace of what retrieval returned, how the reranker ordered it, and what the scratchpad held at the step that went wrong.

Does a larger context window remove the need for retrieval?

No. Chroma's 2025 study of 18 frontier models found accuracy decays as input grows, well inside the advertised window, and the lost-in-the-middle curve penalizes anything placed mid window. Cost scales the same way: prefill compute grows quadratically with input length and the KV cache grows linearly.

Retrieval exists to keep the window small, fresh, and well placed. That need grows with the window rather than shrinking, because a bigger window tempts teams to stuff it, and stuffing is exactly what the physics punishes.

How much of the context window should an agent actually use?

A workable starting policy is the watermark pattern: full operation under 70 percent occupancy, pruning between 70 and 85 percent, and mandatory compaction above 85 percent. Tune the thresholds against your own evals, because models differ in how early mid window accuracy drops.

The rule that matters most is never making a model call while over the red watermark. Past that point you pay quadratic prefill prices for degraded attention, and every later turn inherits the damage.

What is the difference between RAG and context engineering?

RAG is one subsystem: it retrieves documents and injects them into the prompt. Context engineering is the whole pipeline around the model, which includes retrieval plus memory tiers, tool state isolation, budget policy, and verification gates.

A production agent runs RAG as the first of five pillars. Teams that stop at retrieval still lose state between turns, still bloat the prefix with tool schemas, and still have no gate between a generated claim and an executed write.

How do I detect context rot in a production agent?

Watch for three symptoms: the agent contradicts facts it verified earlier, it re-asks for identifiers it already confirmed, or accuracy decays as conversations lengthen while single-turn evals stay flat. Any one of the three points at the context rather than the model.

Confirm with instrumentation. Graph task accuracy against turn count and against context occupancy per call. A downward slope against either axis is rot, and the fix is pruning, placement, and fresher evidence rather than a bigger window.

When should a task move to a sub agent instead of a longer context?

Move a subtask when it is self-contained, token heavy, and only its conclusion matters to the parent. Reading fifty pages of logs to answer one question is the canonical case: the sub agent burns its own window and returns a paragraph.

The parent's context stays small and fresh, and the sub agent's transcript never enters it. The cost is that no nuance survives beyond the summary, so keep shared reasoning in the parent and push only mechanical breadth downward.

The Engineering Takeaway

Context engineering reduces to an audit you can run against any agent, in order: what enters the window is retrieval, what survives between turns is memory, what the agent can touch is tools, when the pipeline prunes is budget, and how the system verifies is gates.

Run the audit per release and per incident, in this order:

1. Retrieval. Are queries constructed per step, fused across BM25 and embeddings, situated with chunk context, and reranked before placement?

2. Memory. Does state live in a typed scratchpad with TTLs and a scoring function, or in prose the model re-derives every call?

3. Tools. Are schemas compact and loaded on demand, with raw outputs isolated behind references and idempotency keys on every write?

4. Budget. Are watermark zones defined, and does compaction run before the red zone rather than after the failure?

5. Verification. Does every claim carry a reference, does every write pass a schema gate, and does every write get read back?

Most production agent failures trace to one of the five answers being "we never decided."

The model rents you a window. Your pipeline decides what it holds.

Teams that treat context as engineered state are the ones whose agents still work on turn nine.