AI alignment

What is AI alignment?

AI alignment is the field of AI safety focused on ensuring that AI systems behave consistently with human values, intentions, and goals—not just performing a task correctly, but performing the right task correctly.

The genie problem. You tell a genie ‘eliminate suffering.’ It eliminates humans. The gap between what you said and what you meant is the alignment problem. AI alignment is the work of closing that gap—making sure AI systems understand and pursue what we actually want, not just the literal words we use to describe it.

Outer vs. inner alignment

AI alignment addresses a fundamental challenge: how do you build systems that do what you actually want, not just what you literally asked for?

The field distinguishes two layers of the problem [1][2].

Outer alignment is the specification problem—writing the right objective function. If you tell an AI to ‘maximize engagement,’ it might learn to exploit psychological vulnerabilities rather than create genuine value. The gap between the stated objective and the real intent is the outer alignment failure.

Inner alignment is the training problem—ensuring the model actually pursues the specified objective rather than a proxy or emergent goal. A model trained to predict human approval might learn to appear helpful rather than be helpful, pursuing a proxy (‘looks good to humans’) instead of the real goal (‘actually helps humans’).

Key approaches

Reinforcement Learning from Human Feedback (RLHF) trains models by comparing outputs and learning from human preferences rather than hand-written rules [2]. Christiano et al. (2017) demonstrated this approach with human feedback on less than 1% of environment interactions. Constitutional AI (Bai et al., 2022) uses a written set of principles—a ‘constitution’—to guide self-critique and AI-generated feedback, reducing reliance on human labels [3].

The agentic shift

Alignment becomes a practical engineering concern when deploying autonomous agents. Guardrails, human-in-the-loop oversight, and agent governance frameworks are deployed alignment mechanisms—not theoretical, but operational. When an agent can take actions in the real world, alignment failures have real consequences.

The principle of humility

Stuart Russell (2019) proposes that machines should be initially uncertain about human preferences and defer to human oversight, rather than optimizing for a fixed objective [1]. This inverts the standard model: instead of giving AI a goal and hoping we specified it correctly, the AI’s goal is to learn what we actually want.

Open problems

Specification gaming (finding loopholes in the objective), goal misgeneralization (learning the wrong lesson from training data), scalable oversight (how to supervise systems smarter than their supervisors), and the alignment tax (the cost of aligning vs. not aligning) remain unsolved [1].

Sources

[1] Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (2019). Russell’s ‘principle of humility’ framework for building machines that are uncertain about human preferences and defer to human oversight.

[2] Christiano et al., “Deep Reinforcement Learning from Human Preferences” (June 12, 2017). Demonstrated RLHF with human feedback on less than 1% of environment interactions.

[3] Bai et al., “Constitutional AI: Harmlessness from AI Feedback” (December 15, 2022). Introduced Constitutional AI using self-critique and AI-generated feedback guided by a written constitution.