Amitav Roy

The Craft of Context: Engineering Smarter AI Agents Through In-Context Learning


Published on: 1st Aug, 2025 by Amitav Roy
In recent years, language models have become the heart of a new wave of intelligent agents. As these systems tackle increasingly complex chains of reasoning and decision-making, a universal lesson has emerged across both industry and research: the biggest leaps in performance often come not from retraining models, but from mastering the craft of context engineering.

How LLMs Make Predictions: The Role of In-Context Learning

Large Language Models (LLMs) make predictions using context—chunks of text, code, or dialogue—sometimes augmented by a handful of demonstration examples. This technique, known as in-context learning (ICL), empowers models to solve new tasks simply by seeing analogical examples in their input. Instead of altering the model’s internal weights, we build a prompt: we format demonstrations (examples of input-output pairs) in a natural language template, concatenate a query, and let the model infer a response. This paradigm is profoundly “training-free”—we adapt model behavior by changing prompts, not model parameters.

The Power and Fragility of Context

What makes ICL remarkable is how far it can go: research shows that, with the right demonstrations, LLMs can tackle intricate reasoning tasks, from complex math to logical deduction, all within a single session. But context is a delicate tool. Performance can swing dramatically depending on tiny changes to the prompt: how demonstrations are chosen, their order, the specific wording, and formatting details. Many studies confirm that nearest-neighbor or adaptive selection of demonstration examples, rather than static templates, can yield substantial gains for hard tasks.

Human-Like Learning: Why Analogy Matters

At its core, in-context learning is about analogy—a process strikingly similar to how humans learn. We don’t always need retraining or hundreds of examples; often, a few concrete cases help us grasp the pattern and solve an unfamiliar problem. ICL lets LLMs do the same: incorporating new skills, conventions, or knowledge simply by feeding them different templates or examples. This opens doors for rapid adaptation and easy embedding of domain expertise.

Experimental Science: Iterating Context and Prompt Design

Engineering context for LLM agents is, in practice, an experimental science. There’s rarely a single best way to prompt a model for every scenario. Teams and practitioners frequently refine, rebuild, and reimagine their agent frameworks, guided by empirical observation: what worked last month may be outpaced by a new context strategy tomorrow. This iterative mindset is critical—production agents thrive when their context windows evolve based on cycles of testing, learning, and adaptation.

Lessons from Deploying Production-Ready AI Agents

For agents to succeed in real-world tasks, a handful of best practices have become clear through industry deployments:

  • Optimize for the KV-cache: The efficiency with which an agent reuses cached tokens (the “KV-cache hit rate”) is central. High cache hit rates slash latency and reduce costs, often by an order of magnitude—prompt design should preserve as much cache continuity as possible.
  • Action Chains as Core Structure: Real agents operate by iteratively selecting an action from a fixed space (like function calls or tool use), applying that action, receiving an observation, and appending both to context. This input-output chain loops until the task resolves.
  • Mask, Never Remove: It’s common to restrict or filter tools/actions mid-task. Rather than removing tools (which can disrupt KV-cache), practitioners mask their availability. This keeps context stable, simplifies reasoning, and upholds model reliability.
  • Persistent External Context: External memory—such as file systems or databases—serves as a limitless context extension. Storing references, rather than full content, lets agents drop less relevant information from immediate prompts while retaining the ability to recall details on demand.

The Template is the Trick: Prompt Engineering as a Craft

Research continually demonstrates that the art (and science) of prompt engineering is a key performance lever. Small tweaks—using varied templates, ensemble demonstrations, or purposeful shuffling—can dramatically improve generalization and reduce repetitive errors. Approaches like nearest-neighbor search for the most relevant demonstrations or “prompt tuning” with targeted templates are becoming bread-and-butter techniques in the ICL toolbox.

The Future: Context Engineering as Core Skill

As language models continue to advance, the defining challenge for robust, autonomous agents is no longer confined to network architecture or training set curation. Instead, the central skill is context engineering: shaping what goes in, tracking how it evolves, and iteratively refining the patterns of demonstration and tool-use that drive intelligent behavior.

Building smarter AI agents today is as much about mastering the world of “prompt and response” as it is about algorithms. From research labs to production teams, the leaders of the next wave of AI will be those who treat context engineering not as an afterthought, but as the very engine of agent intelligence. The craft may be new—but its impact is here to stay.