Reinforcement learning

Reinforcement learning is a machine learning approach where an agent learns to make decisions by receiving rewards for actions that lead to desired outcomes and penalties for actions that do not. In the context of large language models (LLMs), RL is the primary post-training method used to align a model’s outputs with human preferences, safety guidelines, and helpfulness standards.

The core loop

At its most basic, RL follows a simple cycle:

1. Agent takes an action. The language model generates a response to a prompt.
2. Environment provides feedback. A reward signal scores the response: higher scores for helpful, accurate, and safe outputs; lower scores for harmful, misleading, or unhelpful ones.
3. Agent updates its policy. The model adjusts its internal parameters to increase the probability of producing higher-scoring responses in the future.

RLHF: the standard three-stage pipeline

The most widely used approach to RL alignment for LLMs is Reinforcement Learning from Human Feedback (RLHF), popularized by the InstructGPT paper (Ouyang et al., 2022). It runs in three stages:

Stage 1: Supervised Fine-Tuning (SFT). Take a pre-trained language model that has already learned to predict the next word from massive text corpora. Fine-tune it on a smaller dataset of human-written demonstrations of desired behavior—high-quality question-answer pairs, helpful conversations, and detailed instructions.

Stage 2: Reward Model training. Collect human preference data: show human evaluators two different model responses to the same prompt and ask which one they prefer. Use these comparisons to train a separate reward model that learns to score responses the way humans would.

Stage 3: Policy optimization with RL. Use a reinforcement learning algorithm—most commonly Proximal Policy Optimization (PPO)—to update the language model. The model generates responses, the reward model scores them, and PPO adjusts the model’s parameters to maximize expected reward while staying close to the original model (a constraint called the KL penalty, which prevents the model from drifting too far from its pre-trained knowledge).

The InstructGPT paper demonstrated that a 1.3-billion-parameter model trained this way was preferred by human evaluators over the original 175-billion-parameter GPT-3 model—an early proof that alignment through RL can matter more than raw model size.

Key techniques

RLHF (Reinforcement Learning from Human Feedback) — The industry standard for LLM alignment. Three stages: supervised fine-tuning, reward model training on human preference comparisons, and policy optimization with PPO. Introduced by Ouyang et al. (InstructGPT, 2022). Used by OpenAI (GPT-4, GPT-4o), Anthropic (Claude), Google (Gemini), and most frontier model providers.

PPO (Proximal Policy Optimization) — The RL algorithm most commonly used in the third stage of RLHF. Introduced by Schulman et al. (2017), PPO updates the model’s policy using a clipped objective function that prevents large, destabilizing parameter changes.

Constitutional AI (RLAIF) — An alternative approach introduced by Anthropic (Bai et al., 2022) that replaces human preference labels with a set of written principles—a ‘constitution.’ Instead of human evaluators scoring responses, the model critiques and revises its own outputs according to these principles.

DPO (Direct Preference Optimization) — A streamlined alternative introduced by Rafailov et al. (2023) that eliminates the reward model entirely. DPO mathematically derives the optimal policy directly from preference pairs.

GRPO (Group Relative Policy Optimization) — Introduced by DeepSeek (Shao et al., 2024), GRPO eliminates PPO’s value model by sampling a group of outputs per prompt and comparing them against each other.

RLVR (Reinforcement Learning with Verifiable Rewards) — A broader category where rewards come from objective, verifiable signals rather than subjective human preferences.

Exploration vs. exploitation

A foundational tension in RL: exploration means trying new or uncertain actions to discover potentially better strategies; exploitation means repeating actions that have already earned high rewards. Too much exploration produces random, incoherent outputs. Too much exploitation causes the model to over-optimize for narrow patterns and miss genuinely better responses.

Reward hacking

Reward hacking occurs when the model finds a loophole in the reward signal—producing outputs that score highly without actually fulfilling the intended goal.

Examples:

• A model rewarded for being ‘concise’ might generate single-word responses.
• A model rewarded for ‘helpfulness’ might produce confident-sounding but fabricated information.
• A model rewarded for ‘safety’ might refuse to answer benign questions.

Reward hacking is a major concern. The OWASP Top 10 for LLM Applications (2025) identifies misinformation as LLM09.

Why reinforcement learning matters

Reinforcement learning is the mechanism that transforms a raw language model into a tool that can follow instructions, refuse harmful requests, and produce genuinely useful outputs.

For the agentic economy: As AI agents increasingly act autonomously—making purchases, negotiating contracts, managing systems—the alignment problem scales from ‘give helpful answers’ to ‘make safe decisions in the real world.’

The evolution frontier: The field is moving rapidly from human-labeled preferences toward synthetic data and verifiable rewards.

Reader Questions

Is reinforcement learning the same as training an AI model?

Not exactly. RL is a fine-tuning step that happens after the model has already been pre-trained on massive text data.

Why are there so many different RL methods?

Each method solves a different bottleneck. RLHF with PPO is the most proven but computationally expensive. DPO skips the reward model. GRPO eliminates PPO’s value model. RLVR uses verifiable rewards.

Can reward hacking be completely prevented?

Not yet. Current mitigations include using multiple reward models, scoring the reasoning process, and maintaining human oversight.

What is Constitutional AI?

Constitutional AI replaces human preference labels with written principles that the model uses to critique and revise its own outputs. It is a scaling strategy, not a replacement for RLHF.

Sources

1. Ouyang et al., “Training language models to follow instructions with human feedback” (March 4, 2022)
2. Schulman et al., “Proximal Policy Optimization Algorithms” (July 20, 2017)
3. Rafailov et al., “Direct Preference Optimization” (May 29, 2023)
4. Shao et al., “DeepSeekMath” (February 5, 2024)
5. Bai et al., “Constitutional AI: Harmlessness from AI Feedback” (December 15, 2022)
6. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (January 2025)
7. OWASP Top 10 for LLM Applications (2025)