Chain-of-thought (CoT)

The show-your-work analogy

Think of chain-of-thought like a math teacher asking students to show their work, not just the final answer. A student who writes “42” gets partial credit at best. A student who shows each calculation step—identifying the problem, breaking it into parts, solving each one, and combining the results—demonstrates real understanding and catches their own mistakes along the way. Chain-of-thought prompting asks the AI to do the same thing: think out loud before answering.

How chain-of-thought works

Chain-of-thought prompting was introduced by Jason Wei and colleagues at Google Research in January 2022 [1]. The core insight is simple: when you ask a language model to reason step by step, it decomposes complex problems into smaller, manageable pieces—just as a human would work through a multi-step calculation on paper.

There are two main forms:

Few-shot CoT provides the model with worked examples that include a question, a chain of reasoning, and the final answer. The model imitates the reasoning pattern shown in the examples. For instance:

Q: Roger has 5 tennis balls. He buys 2 more cans of 3 balls each. How many does he have now?
A: Roger started with 5 balls. 2 cans of 3 is 6. 5 + 6 = 11. The answer is 11.

The model sees this reasoning structure and applies it to new questions—decomposing each problem into steps rather than guessing at the final number.

Zero-shot CoT requires no examples at all. Simply appending “Let’s think step by step” to a prompt triggers the model to generate its own reasoning chain before answering. This was shown to be surprisingly effective for many reasoning tasks, even without worked demonstrations [1].

Why it matters for AI agents

Chain-of-thought is the cognitive foundation that makes modern AI agents capable of complex reasoning. When an agent plans a multi-step task—researching a topic, comparing options, calling tools, and synthesizing results—it uses chain-of-thought internally to break the goal into subtasks and evaluate each step.

The ReAct framework (Reason + Act), which underlies nearly every production agent in 2025–2026, interleaves chain-of-thought reasoning with tool calls [2]: the agent thinks about what to do next, takes an action, observes the result, and reasons about the next step. Without chain-of-thought, agents would lack the structured reasoning needed to handle real-world complexity.

Modern reasoning models—such as OpenAI’s o-series, Anthropic’s extended thinking, and Google’s Deep Think—have internalized chain-of-thought into the model architecture itself, performing multi-step deliberation inside the model rather than requiring explicit prompting for it.

Limitations

Scale-dependent effectiveness. Chain-of-thought gains are most pronounced in larger models (typically above 50 billion parameters). Smaller models may not benefit as much, or may produce reasoning chains that introduce more errors than they resolve [1].

Increased cost and latency. Generating intermediate reasoning steps uses more tokens and compute than a direct answer. For production systems, this adds to the inference cost and response time.

Not a guarantee of correctness. A model can produce a fluent, step-by-step reasoning chain that still arrives at a wrong answer. The intermediate steps provide a scaffold, not a proof—the model’s reasoning can be confidently wrong at each step.

Fragile with complexity. As problems grow more complex, the reasoning chain can drift, lose track of earlier steps, or introduce compounding errors. This is why agents often combine chain-of-thought with verification loops and tool-based fact-checking.

Sources

[1] Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” arXiv, January 28, 2022. The foundational paper introducing chain-of-thought prompting, demonstrating that few-shot CoT significantly improves reasoning on arithmetic, commonsense, and symbolic tasks.

[2] Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” arXiv, October 2022. The framework that interleaves reasoning and acting to enable agents to handle complex tasks.