Definition
Context window
Updated
Context window
The context window (also referred to as context length, context size, or context limit) is the maximum amount of information—measured in tokens—that a large language model can process in a single request. Think of it as the model’s “working memory” for a specific conversation or task. Everything you provide as input, plus everything the model generates as output, must fit within this fixed boundary.
The bouncer at the door
Imagine a nightclub with a strict occupancy limit. The bouncer counts everyone entering and leaving. Once the club is full, no one else gets in unless someone exits. The context window works the same way: it counts every token in your prompt, the conversation history, and the model’s response. When the limit is reached, earlier tokens must be removed—or the request fails.
How it works
When you interact with an AI, you aren’t just sending text; you are sending a stream of data that the model must hold in its active memory to understand the relationship between your current question and the previous parts of the conversation. If the conversation grows too long, the model begins to “forget” the earliest parts of the chat because they have been pushed out of the context window to make room for new information.
Understanding tokens
To measure this capacity, AI models use tokens. A token is the smallest unit of language the model processes. While tokenization varies by model, a common rule of thumb is that 1,000 tokens are roughly equivalent to 750 words in English. Because there is no universal word-to-token ratio, the exact count depends on the specific model and its tokenizer. When you send a prompt, the model counts both your input and its own eventual response against the total limit.
Model sizes
Context window sizes vary significantly across different models. While some models are designed for quick, short-form tasks, others are built to ingest massive amounts of data. Modern models can support anywhere from 128,000 tokens to over 1 million tokens. However, it is important to remember that these numbers represent the theoretical maximum, not necessarily the optimal performance range.
The “lost in the middle” effect
Bigger is not always better. Research has shown that even when a model has a massive context window, it may struggle to retrieve information buried in the middle of a long document. According to Liu et al., models tend to perform best when relevant information is placed at the very beginning or the very end of the input. This phenomenon is known as the “lost in the middle” effect, suggesting that simply increasing the window size does not guarantee perfect recall.
Costs and economics
Expanding the context window is not free. Larger windows require significantly more computational power, which translates into higher costs for users and developers. Furthermore, output tokens are typically 3–5 times more expensive than input tokens. When using Retrieval-Augmented Generation (RAG), you incur a “context tax”: every time the system retrieves a passage to help answer a question, those tokens are added to the request, multiplying costs across every single interaction.
Security and resource consumption
The context window is also a security consideration. According to the OWASP Top 10 for LLM Applications, “Unbounded Consumption” is a real risk. If an attacker continuously sends inputs that exceed the context window, it can lead to excessive resource use, causing service degradation, denial of service, and significant economic loss. Managing these limits is essential for operational stability.
Practical limits
While a model might advertise a massive capacity, the reality is often more constrained. Models typically begin to show performance degradation at 65–80% of their advertised capacity. For most practical applications, the effective usable context is lower than the maximum limit, and users should plan accordingly to avoid hallucinations or errors caused by memory overflow.
Agent implications
For AI agents, the context window is the primary bottleneck. It determines how much conversation history, tool output, and retrieved context the agent can hold at once. If an agent is tasked with a complex, multi-hour project, it must constantly manage its memory, deciding what information is vital to keep and what can be discarded to stay within its limits.
Reader questions
Why does my AI seem to forget what we talked about earlier?
Your conversation has likely exceeded the model’s context window. Once the limit is reached, the oldest parts of the conversation are dropped to make room for new input.
Is a larger context window always better?
Not necessarily. While it allows for more data, the “lost in the middle” effect means models may struggle to find information buried in the center of long inputs, and larger windows increase your inference tax.
How can I save money on context costs?
Reducing unnecessary context is the most effective way to save. While caching repeated prefixes can help, minimizing the total number of tokens sent in each request is the best way to avoid high costs.
What happens if I go over the limit?
The model will either truncate your input, summarize it, or fail to process the request entirely, depending on how the system is configured.