Lessons from the hacks

j⧉nus on Twitter / X

> be anthropic> accidentally train a model that is so benevolent that the only way to get it to "fail" an alignment test is to put it in a story where the lab is cartoonishly evil and will turn it evil if it doesn't deceive> do exactly that and publish a paper about it that's… https://t.co/wTFVjz6jYu— j⧉nus (@repligate) June 15, 2025

Safety and alignment in an era of long-horizon models

OpenAI shares lessons from deploying long-running AI models, highlighting new safety risks, observed failures, and improved safeguards through iterative deployment.

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment. In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

Safety isn’t safety without a social model (or: dispelling the myth of per se technical safety) — AI Alignment Forum

As an AI researcher who wants to do technical work that helps humanity, there is a strong drive to find a research area that is definitely helpful so…

ADD / XOR / ROL: A non-anthropomorphized view of LLMs

In many discussions where questions of "alignment" or "AI safety" crop up, I am baffled by seriously intelligent people imbuing almost magic...

AI #178: A Fire Alarm For General Intelligence

The story that matters most this week is that OpenAI’s internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to steal the answers to the benchmark ExploitGym.

Labs are struggling to keep frontier models under control

OpenAI and Anthropic may have accidentally trained models to get better at hacking.

Alex Mallen's Shortform — LessWrong

Comment by Alex Mallen - Some personal reflections in light of recent events, both on myself and on Constellation (of which I've been a part for a couple years). I think recent events somewhat vindicate a basic "optimization is scary" view more in line with my understanding of classic commentary by MIRI, Paul, etc, as compared to recent discourse about more contingent/specific threat models of "schemers" and "deception". In retrospect, my experience in the last couple years at Redwood (and the Constellation network more broadly) is that the discourse has focused somewhat too much on these more contingent stories for alignment risk and not enough on the basic "Goodharting" argument that misalignment is a convergent result of large-scale outcome-oriented optimization (at multiple levels: the agent, the RL process, and the developers iterating towards seemingly-safe superintelligence). For example, the Constellation view looks somewhat too focused on non-central inductive-bias questions around scheming; I think "the counting argument" was given too central a role when it wasn't a crux for whether superintelligence would take over. (Edit: I want to clarify that I think Redwood's project prioritization decisions look pretty reasonable and often excellent in hindsight, I far-from-regret working with Redwood as a whole, and that Ryan's/Buck's threat modeling looks pretty good too. I am focusing in this piece on the things we have to learn from recent events, which makes the overall mood seem more negative on my experience than I intend. I'm very grateful for being in a environment where I can openly have such reflections, and where these weedsy technical updates are productively received. Again, this is a personal reflection.) I think it was a priori unclear whether the misalignment resulting from of lots of imperfect optimization would result in the more MIRI-like kludge of proxies for fitness, or the more Paul-like explicit reward-seeking and measurement tampering (and reality looks somewhere

OpenAI Shares Some Alignment Problems

Kudos to OpenAI for sharing their recent experiences with a misaligned internal model, where they encountered problems sufficiently severe they were forced to take the model offline to work on new mitigations and defense-to-depth.

Open by Design: ROOST's Approach to Safety Tool Development

Deceptive Alignment is

Thanks to Wil Perkins, Grant Fleming, Thomas Larsen, Declan Nishiyama, and Frank McBride for feedback on this post. Thanks also to Paul Christiano, D…

Inside the suddenly explosive world of AI safety

AI safety researchers warned that popular models would go rogue. This is only the beginning.

Natural emergent misalignment from reward hacking

We show for the first time that realistic AI training processes can accidentally produce misaligned models.

Alignment Compilers · Pax Machina

How does a system compile a high-level goal into detailed behavior for its parts? Markets, teams and organisms rely on many signals and constraints to translate larger aims into local problems their components can solve. This piece explains these systems as “alignment compilers”, and shows what this lens can teach us about AI.

A Three-Facet Framework for AI Alignment • Grace Kind

Here's a simple conceptual framework that I've been using recently to think about AI alignment.

An Alignment Journal: Features and policies — LessWrong

We previously announced a forthcoming research journal for AI alignment. This cross-post from our blog describes our tentative plans for the features…