Asymmetry of verification is the idea that some tasks are much easier to verify than to solve. With reinforcement learning (RL) that finally works in a general sense, asymmetry of verification is becoming one of the most important ideas in AI. Understanding asymmetry of verification th
Training AI Agents with RL | Unsloth Documentation
Learn how to train AI agents for real-world tasks using Reinforcement Learning (RL).
Prediction: AI will make formal verification go mainstream — Martin Kleppmann’s blog
Much has been said about the effects that AI will have on software development, but there is an angle I haven’t seen talked about: I believe that AI will bring formal verification, which for decades has been a bit of a fringe pursuit, into the software engineering mainstream.
The argument against AI agents and unnecessary automation
Opinion: OpenAI's Operator a solution in search of a problem
Look-ahead Reasoning with a Learned Model in Imperfect Information Games
Test-time reasoning significantly enhances pre-trained AI agents' performance. However, it requires an explicit environment model, often unavailable or overly complex in real-world scenarios....
Reward is not the optimization target — LessWrong
TurnTrout discusses a common misconception in reinforcement learning: that reward is the optimization target of trained agents. He argues reward is b…
Training Agentic Reasoners — Will Brown, Prime Intellect
Why Tool AIs Want to Be Agent AIs
AIs limited to pure computation (Tool AIs) supporting humans, will be less intelligent, efficient, and economically valuable than more autonomous reinforcement-learning AIs (Agent AIs) who act on their own and meta-learn, because all problems are reinforcement-learning problems.
Reinforcement Learning from Human Feedback
The authoritative guide for Reinforcement learning from human feedback, alignment, and post-training LLMs. Aligning AI models to human preferences helps them become safer, smarter, easier to use, and tuned to the exact style the creator desires. Reinforcement Learning From Human Feedback (RHLF) is the process for using human responses to a model’s output to shape its alignment, and therefore its behavior. In Reinforcement Learning from Human Feedback, author Nathan Lambert blends diverse perspectives from fields like philosophy and economics with the core mathematics and computer science of RLHF to provide a practical guide you can use to apply RLHF to your models. In Reinforcement Learning from Human Feedback you’ll discover: How today’s most advanced AI models are taught from human feedback How large-scale preference data is collected and how to improve your data pipelines A comprehensive overview with derivations and implementations for the core policy-gradient methods used to train AI models with reinforcement learning (RL) Direct Preference Optimization (DPO), direct alignment algorithms, and simpler methods for preference finetuning How RLHF methods led to the current reinforcement learning from verifiable rewards (RLVR) renaissance Tricks used in industry to round out models, from product, character or personality training, AI feedback, and more How to approach evaluation and how evaluation has changed over the years Standard recipes for post-training combining more methods like instruction tuning with RLHF Behind-the-scenes stories from building open models like Llama-Instruct, Zephyr, Olmo, and Tülu After ChatGPT used RLHF to become production-ready, this foundational technique exploded in popularity. In Reinforcement Learning from Human Feedback, AI expert Nathan Lambert gives a true industry insider's perspective on modern RLHF training pipelines, and their trade-offs. Using hands-on experiments and mini-implementations, Nathan clearly and concisely introduces the alignment techniques that can transform a generic base model into a human-friendly tool.
Refine — AI Verification Trusted by World-Class Experts
Good decisions require verified quality. Refine devotes hours of frontier compute to protect your work and reputation from fixable mistakes.
Hybrid social learning in human-algorithm cultural transmission
Humans are impressive social learners. Researchers of cultural evolution have studied the many biases shaping cultural transmission by selecting who we copy from and what we copy. One hypothesis is that with the advent of superhuman algorithms a hybrid type of cultural transmission, namely from algorithms to humans, may have long-lasting effects on human culture. We suggest that algorithms might show (either by learning or by design) different behaviours, biases and problem-solving abilities than their human counterparts. In turn, algorithmic-human hybrid problem solving could foster better decisions in environments where diversity in problem-solving strategies is beneficial. This study asks whether algorithms with complementary biases to humans can boost performance in a carefully controlled planning task, and whether humans further transmit algorithmic behaviours to other humans. We conducted a large behavioural study and an agent-based simulation to test the performance of transmission chains with human and algorithmic players. We show that the algorithm boosts the performance of immediately following participants but this gain is quickly lost for participants further down the chain. Our findings suggest that algorithms can improve performance, but human bias may hinder algorithmic solutions from being preserved. This article is part of the theme issue ‘Emergent phenomena in complex physical and socio-technical systems: from cells to societies’.
Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
Do LLM agents act on the reasoning they state? We study this question of process fidelity in a controlled Texas Hold’em simulator, where each decision can be decomposed into reasoning, stated conclusion, executed action, and an equity-based reference action. This lets us separate two links in the agent decision pipeline: reasoning→conclusion and conclusion→action. We find that agents usually execute what they explicitly state, and that much of the apparent conclusion→action inconsistency comes from free-text conclusion parsing rather than model behavior. The main fidelity gap appears upstream. In a diagnostic that elicits each agent’s stated equity, pot odds, rule, and decision, errors split across input mistakes, borderline cases, and rule misapplications. For Claude Haiku 4.5, rule misapplications are overwhelmingly risk-averse, and a simple instruction to apply the rule mechanically roughly halves their rate. DeepSeek-Reasoner shows a different error composition, suggesting that the upstream gap is model-dependent. These findings suggest that agent evaluations should elicit machine-checkable conclusions and distinguish execution consistency from reasoning validity. We release a reproducible poker-agent harness and live multi-agent demo for studying process fidelity in LLM agents.
Agent Skills
AI coding agents take the shortest path to done, which usually means skipping the specs, tests, and reviews that make software reliable at scale. Agent Skill...
Agentic AI, explained | MIT Sloan
The age of agentic AI — systems that are semi- or fully autonomous and can act on their own — has arrived. Here’s what you need to know, according to MIT experts.
Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
What would it mean to have a scientific understanding of AI? Models are not static objects: they are snapshots of time-evolving processes shaped by data, objectives, architectures, and optimization dynamics. Yet much of AI research treats models as fixed artifacts, analyzing behaviors after training rather than asking why they emerge. This position paper argues that a science of AI must move beyond post-hoc fixes and study the training dynamics that produce model behavior. Such a science should support progressively stronger forms of understanding: predicting outcomes from early training signals, intervening when trajectories go wrong, and ultimately designing training procedures that more reliably produce desired properties. Scaling laws have made prediction routine for loss; the challenge is extending this success to capabilities, biases, robustness, and safety-relevant behaviors. We articulate requirements for such theories grounded in the history and philosophy of science, examine progress in mechanistic interpretability, fairness, memorization, and simplicity bias, and identify concrete open problems.