RLHF is responsible for most of what makes modern LLMs useful in practice. The shift from raw language modelling to instruction-following behaviour was largely a consequence of learning from human preference signals. But RLHF has a fundamental structural bottleneck that limits how far it can scale: it depends on humans being in the loop.

The RLHF Bottleneck

Human annotation is slow, expensive, and inconsistent at the margins. A top-tier RLHF pipeline produces maybe tens of thousands of preference pairs per month. At GPT-4 scale, that's a rounding error relative to the pretraining data. More subtly, human annotators are bad at evaluating things they can't verify — long chains of mathematical reasoning, novel code, or expert-level domain knowledge. The reward model trained on human labels inherits all of these blind spots.

“A reward model is only as good as the humans whose preferences trained it. When the task exceeds human expertise, RLHF has nowhere to go.”

Enter Reinforcement Learning from Agent Feedback

The core insight behind RLAF is simple: replace human annotators with AI agents that can evaluate outputs more scalably, more consistently, and — in some domains — more accurately. This is already happening implicitly in systems like Constitutional AI (Claude), Scalable Oversight, and Debate. RLAF makes it explicit.

The Architecture

A minimal RLAF pipeline looks like this: a generator model proposes outputs, a critic agent evaluates those outputs against a rubric, and the preference signal from the critic drives a PPO or DPO update on the generator. The critic agent itself can be another LLM, a verifier (for code or math), or a multi-agent debate panel.

python
# Simplified RLAF training loop sketch
for step in range(max_steps):
    prompts = sample_prompt_batch()
    generations = generator.generate(prompts)
    
    # Critic agent evaluates — no humans here
    rewards = critic_agent.evaluate(prompts, generations)
    
    # PPO update on generator
    generator = ppo_step(generator, prompts, generations, rewards)

Where It Outperforms RLHF

  • Scalability: critics can evaluate millions of pairs per day — humans cannot
  • Domain depth: a specialised code-critic can evaluate correctness better than a generalist human annotator
  • Consistency: no annotator fatigue, no shift in standards across the dataset
  • Iteration speed: feedback loops can run overnight rather than over months

The Hard Problems

RLAF is not a free lunch. The risks are distinct from — and in some ways more dangerous than — RLHF's risks.

Reward Hacking Against the Critic

If the generator is trained against a fixed critic, it will eventually learn to exploit the critic's weaknesses rather than actually improve. The generator and critic need to co-evolve, or you need to use verifiable reward signals (like unit tests for code) that can't be gamed.

Mode Collapse

Agent feedback is more homogeneous than human feedback. This can cause the generator to collapse to a narrow mode of outputs that the critic happens to prefer — even if those outputs are not what end users want. Diversity regularisation and periodic human spot-checks are important safeguards.

Alignment Tax

Training against an AI critic risks optimising for AI-legible quality rather than human-desired quality. A model can become very good at satisfying its critic while drifting from genuine helpfulness. This is a subtle but important failure mode in long RLAF training runs.

The Synthesis: Hybrid Oversight

The most robust alignment approach combines RLHF and RLAF. Humans define the rubric and sample the edge cases; AI agents evaluate the bulk. This preserves the grounding of human judgment while gaining the scalability of automated feedback. Scalable Oversight research at Anthropic and OpenAI is converging toward exactly this hybrid.

Why This Matters for Production Systems

If you're fine-tuning models for enterprise use cases, you're already doing informal RLAF — you just might not call it that. Any time you use a stronger model to critique a weaker model's outputs and then train on the curated subset, you're doing RLAF. Making this explicit and systematic is one of the highest-leverage things you can do to improve model quality without human annotation bottlenecks.

✶

The shift from RLHF to RLAF is not a replacement — it's a maturation. We're learning to let AI systems participate in their own improvement, carefully and verifiably. That's the direction the field is going, and it's the most interesting engineering problem I know of right now.