Reinforcement learning algorithms can break in surprising, counterintuitive ways. In this post we’ll explore one failure mode, which is where you misspecify your reward function.
Natural emergent misalignment from reward hacking
We show for the first time that realistic AI training processes can accidentally produce misaligned models.
Reward is not the optimization target — LessWrong
TurnTrout discusses a common misconception in reinforcement learning: that reward is the optimization target of trained agents. He argues reward is b…
The Human-in-the-Loop is Tired
On reward functions, dopamine, and what it actually feels like when the code starts writing itself
Models Don't "Get Reward" — LessWrong
Models don't "get" reward. Reward is the mechanism by which we select parameters, it is not something "given" to the model. Reinforcement learning sh…
Asymmetry of verification and verifier’s rule — Jason Wei
Asymmetry of verification is the idea that some tasks are much easier to verify than to solve. With reinforcement learning (RL) that finally works in a general sense, asymmetry of verification is becoming one of the most important ideas in AI. Understanding asymmetry of verification th
Control: Why Big Giving Falls Short
j⧉nus on Twitter / X
> for whatever reason, Claude-series model "try less hard" on the first shotI think this is because they're less brain damaged and a generalization of being better agents & caring about reality instead of test passing.If you try maximally hard at everything you do, regardless… https://t.co/jkhSar2hJ7— j⧉nus (@repligate) April 15, 2026
Why Good Leaders Fail
The risk of sudden leadership failure can be headed off by early detection of challenges and better supports.
The Optimization Trap: Why Too Much Efficiency Makes Us Fragile - The Great Simplification
In this episode, Nate is joined by biologist and biophysicist Olivier Hamant to explore why living systems prioritize robustness over performance, and what that means for a civilization built almost entirely in the opposite direction.
An Empirical Study of Example Forgetting during Deep Neural Network Learning
Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a “forgetting event” to have occurred when an individual training example transitions […]
Noam Brown on Twitter / X
And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.— Noam Brown (@polynoamial) August 1, 2026
Learning to Trust: How Humans Mentally Recalibrate AI Confidence Signals
Productive human-AI collaboration requires appropriate reliance, yet contemporary AI systems are often miscalibrated, exhibiting systematic overconfidence or underconfidence. We investigate whether humans can learn to mentally recalibrate AI confidence signals through repeated experience. In a behavioral experiment (N = 200), participants predicted the AI's correctness across four AI calibration conditions: standard, overconfidence, underconfidence, and a counterintuitive "reverse confidence" mapping. Results demonstrate robust learning across all conditions, with participants significantly improving their accuracy, discrimination, and calibration alignment over 50 trials. We present a computational model utilizing a linear-in-log-odds (LLO) transformation and a Rescorla-Wagner learning rule to explain these dynamics. The model reveals that humans adapt by updating their baseline trust and confidence sensitivity, using asymmetric learning rates to prioritize the most informative errors. While humans can compensate for monotonic miscalibration, we identify a significant boundary in the reverse confidence scenario, where a substantial proportion of participants struggled to override initial inductive biases. These findings provide a mechanistic account of how humans adapt their trust in AI confidence signals through experience.
Generative AI runs on gambling addiction — just one more prompt, bro!
You’ll have noticed how previously normal people start acting like addicts to their favourite generative AI and shout at you like you’re trying to take their cocaine away. Matthias Döpm…
Paper page - Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses
Join the discussion on this paper page
Architecture Is All You Need: Diversity-Enabled Sweet Spots for Robust Humanoid Locomotion
Robust humanoid locomotion in unstructured environments requires architectures that balance fast low-level stabilization with slower perceptual decision-making. We show that a simple layered control architecture (LCA), a proprioceptive stabilizer running at high rate, coupled with a compact low-rate perceptual policy, enables substantially more robust performance than monolithic end-to-end designs, even when using minimal perception encoders. Through a two-stage training curriculum (blind stabilizer pretraining followed by perceptual fine-tuning), we demonstrate that layered policies consistently outperform one-stage alternatives in both simulation and hardware. On a Unitree G1 humanoid, our approach succeeds across stair and ledge tasks where one-stage perceptual policies fail. These results highlight that architectural separation of timescales, rather than network scale or complexity, is the key enabler for robust perception-conditioned locomotion.