Inference

What is AI Inference?

In the world of artificial intelligence, inference is the process of using a machine learning model that has already been trained to generate useful outputs—such as predictions, text, or decisions—from new, unseen input data. If training is the period where an AI goes to school to learn, inference is the moment it graduates and starts doing its job in the real world [1][2].

The Chef Analogy: Training vs. Inference

To understand the difference, imagine a chef learning to cook. Training is the long, intensive process of culinary school: the chef practices recipes hundreds of times, makes mistakes, receives feedback from instructors, and adjusts their technique until they master the craft. This is a resource-heavy, episodic phase that happens before the restaurant ever opens [1][2].

Inference, by contrast, is the dinner service. The chef is no longer learning; they are applying their training to fulfill specific orders from customers. When a customer places an order (the input), the chef prepares the dish (the output) based on what they already know. The key constraint is speed—the kitchen must deliver each dish quickly and consistently, because customers are waiting [1].

How It Works: The Life of a Prompt

When you interact with a large language model, the system performs inference in a specific, two-part sequence:

• Tokenization: Your input is broken down into smaller units called tokens, which can be parts of words, whole words, or characters.
• Prefill Phase: The model processes your entire input prompt through its layers all at once to build context.
• Decode Phase: The model generates the response one token at a time. Each individual token generated is a single inference pass [3].

Inference vs. Training

Training and inference are fundamentally different operations, even though they use the same model architecture:

• Training involves both forward and backward passes through the model. The model makes a prediction, measures the error, then adjusts its internal weights via backpropagation to reduce that error. This process is episodic—it runs once per model version or fine-tuning cycle, typically over days or weeks. Training is optimized for throughput: processing as much data as possible per unit of time [1][2].
• Inference involves only the forward pass. The model’s weights are frozen; it simply processes new inputs and produces outputs. Inference is continuous—it runs 24/7 in production, scaling directly with user requests. Inference is optimized for latency: minimizing the time from a single input to a single, high-quality answer [2].

Key Metrics for Measuring Performance

Because inference happens in real-time under production load, engineers track specific metrics to ensure the system feels responsive:

• Time to First Token (TTFT): How quickly the model produces its first token after you submit a query. This is the primary indicator of perceived latency—short TTFT means the system feels snappy [4].
• Time Per Output Token (TPOT): The average time to generate each subsequent token after the first. This determines the overall pace of the response [4].
• Goodput: The amount of useful output the system delivers while still meeting its target latency thresholds. Goodput balances speed against capacity—high throughput means nothing if the responses arrive too slowly [4].

Why Inference Matters for AI Agents

As autonomous AI agents become more common, inference becomes the engine behind every action they take. Every time an agent decides which tool to call, formulates a response, or evaluates whether a task is complete, it is running an inference pass through its underlying large language model.

In complex agentic workflows, a single user prompt can trigger many downstream inference calls—planning, retrieval, tool execution, validation, synthesis—each one consuming computational resources. This volume of inference is the primary driver of the inference tax: the cumulative cost of running AI applications at scale, which forces developers to make trade-offs between model capability, speed, and operating expense.

Common Questions

Is inference the same as prediction?
In traditional machine learning, the terms overlap—a model “predicts” an output. In the context of large language models, “inference” is the broader term: it covers not just classification or regression but also text generation, reasoning, and tool use. Think of prediction as one type of output that inference can produce.

Why is inference optimized differently than training?
Training is a batch process where latency per individual data point does not matter—the goal is to process as much data as possible to learn efficiently. Inference is interactive: a user is waiting for a response, so the system must minimize the time from input to output. The hardware, software, and scheduling strategies used for inference are therefore tuned for responsiveness, not raw throughput [2].

Can a model run inference without being connected to the internet?
Yes. Once a model’s weights are loaded onto hardware (whether a cloud GPU or a local device), inference can happen entirely on-device without an internet connection. This is how some mobile AI features and on-premise enterprise deployments work.