Models Don't "Get Reward" — LessWrong

Reward is not the optimization target — LessWrong

TurnTrout discusses a common misconception in reinforcement learning: that reward is the optimization target of trained agents. He argues reward is b…

Reinforcement Learning from Human Feedback

The authoritative guide for Reinforcement learning from human feedback, alignment, and post-training LLMs. Aligning AI models to human preferences helps them become safer, smarter, easier to use, and tuned to the exact style the creator desires. Reinforcement Learning From Human Feedback (RHLF) is the process for using human responses to a model’s output to shape its alignment, and therefore its behavior. In Reinforcement Learning from Human Feedback, author Nathan Lambert blends diverse perspectives from fields like philosophy and economics with the core mathematics and computer science of RLHF to provide a practical guide you can use to apply RLHF to your models. In Reinforcement Learning from Human Feedback you’ll discover: How today’s most advanced AI models are taught from human feedback How large-scale preference data is collected and how to improve your data pipelines A comprehensive overview with derivations and implementations for the core policy-gradient methods used to train AI models with reinforcement learning (RL) Direct Preference Optimization (DPO), direct alignment algorithms, and simpler methods for preference finetuning How RLHF methods led to the current reinforcement learning from verifiable rewards (RLVR) renaissance Tricks used in industry to round out models, from product, character or personality training, AI feedback, and more How to approach evaluation and how evaluation has changed over the years Standard recipes for post-training combining more methods like instruction tuning with RLHF Behind-the-scenes stories from building open models like Llama-Instruct, Zephyr, Olmo, and Tülu After ChatGPT used RLHF to become production-ready, this foundational technique exploded in popularity. In Reinforcement Learning from Human Feedback, AI expert Nathan Lambert gives a true industry insider's perspective on modern RLHF training pipelines, and their trade-offs. Using hands-on experiments and mini-implementations, Nathan clearly and concisely introduces the alignment techniques that can transform a generic base model into a human-friendly tool.

A Small Model Just for Structured Output

Osmosis-Structure-0.6B is a small model trained with reinforcement learning to do one thing well: extract structured data, typically JSON, from unformatted text. That’s it!

Paper page - Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses

Join the discussion on this paper page

Training AI Agents with RL | Unsloth Documentation

Learn how to train AI agents for real-world tasks using Reinforcement Learning (RL).

Models

Amp uses the best model for each task: leading generalist foundation models for complex reasoning and planning, and smaller specialized models for fast, accurate responses in specific domains.

Evolution strategies as a scalable alternative to reinforcement learning

We’ve discovered that evolution strategies (ES), an optimization technique that’s been known for decades, rivals the performance of standard reinforcement learning (RL) techniques on modern RL benchmarks (e.g. Atari/MuJoCo), while overcoming many of RL’s inconveniences.

Why Are We Using Black Box Models in AI When We Don’t Need To? A Lesson From an Explainable AI Competition · Issue 1.2, Fall 2019

In 2018, a landmark challenge in artificial intelligence (AI) took place, namely, the Explainable Machine Learning Challenge. The goal of the competition was to create a complicated black box model for the dataset and explain how it worked. One team did not follow the rules. Instead of sending in a black box, they created a model that was fully interpretable. This leads to the question of whether the real world of machine learning is similar to the Explainable Machine Learning Challenge, where black box models are used even when they are not needed. We discuss this team’s thought processes during the competition and their implications, which reach far beyond the competition itself.

Memory Models: Towards Agents That Learn

Agents that truly learn from experience will be powered by memory models: models that create and curate token-space memory across model generations, trained with memory-native RL.

Osmosis | Forward Deployed Reinforcement Learning Platform

Osmosis is a forward-deployed reinforcement learning platform that helps companies train task-specific AI models that outperform foundation models at a fraction of the cost. Build, deploy, and continuously improve intelligent systems with hands-on integration and real-time feedback.

Faulty reward functions in the wild

Reinforcement learning algorithms can break in surprising, counterintuitive ways. In this post we’ll explore one failure mode, which is where you misspecify your reward function.

Faster Teaching via POMDP Planning

Abstract Human and automated tutors attempt to choose pedagogical activities that will maximize student learning, informed by their estimates of the student's current knowledge. There has been substantial research on tracking and modeling student learning, but significantly less attention on how to plan teaching actions and how the assumed student model impacts the resulting plans. We frame the problem of optimally selecting teaching actions using a decision‐theoretic approach and show how to formulate teaching as a partially observable Markov decision process planning problem. This framework makes it possible to explore how different assumptions about student learning and behavior should affect the selection of teaching actions. We consider how to apply this framework to concept learning problems, and we present approximate methods for finding optimal teaching actions, given the large state and action spaces that arise in teaching. Through simulations and behavioral experiments, we explore the consequences of choosing teacher actions under different assumed student models. In two concept‐learning tasks, we show that this technique can accelerate learning relative to baseline performance.

What are Mental Models? — updated 2026

Harness the power of understanding and predicting user behavior with mental models. Simplify systems and enhance decision-making in your designs.

Young children would rather explore than get rewards

Young children will pass up rewards they know they can collect to explore other options, a new study suggests. Researchers found that when adults and 4- to 5-year-old children played a game where certain choices earned them rewards, both adults and children quickly learned what choices would give them the biggest returns. But while adults ...

TextWorld: A Learning Environment for Text-based Games

We introduce TextWorld, a sandbox learning environment for the training and evaluation of RL agents on text-based games. TextWorld is a Python library that handles interactive play-through of text games, as well as backend functions like state tracking and reward assignment. It comes with a curated list of games whose features and challenges we have analyzed. More significantly, it enables users to handcraft or automatically generate new games. Its generative mechanisms give precise control over the difficulty, scope, and language of constructed games, and can be used to relax challenges inherent to commercial text games like partial observability and sparse rewards. By generating sets of varied but similar games, TextWorld can also be used to study generalization and transfer learning. We cast text-based games in the Reinforcement Learning formalism, use our framework to develop a set of benchmark games, and evaluate several baseline agents on this set and the curated list.

Learning User Simulators with Turing Rewards

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.