While there are a decent number of folks out there modeling systems, there’s a much larger group of folks who think of themselves as systems thinkers but utilize techniques like modeling rather casually (a polite way of saying that they don’t use them). Even with modeling tools available, I’ve often taken the shortcut of intuitive modeling, which over time has given me quite the education in making well-intentioned reasoning errors.
Introduction to systems thinking.
Many effective leaders I’ve worked with have the uncanny knack for working on leverage problems. In some problem domains, the product management skillset is extraordinarily effective for identifying useful problems, but systems thinking is the most universally useful toolkit I’ve found.
The Systems Thinker
The Systems Thinker works to catalyze effective change by expanding the use of systems approaches. All articles are available free of charge in an effort to expose as wide of audience as possible. Browse, share with others, save your favorites and tell others about this valuable resource!
Mental Models: The Best Way to Make Intelligent Decisions (~100 Models Explained)
The smartest people in the world use mental models to make intelligent decisions, avoid stupidity, and increase productivity. Let's take a look at how ...
Mental Models: The Best Way to Make Intelligent Decisions (~100 Models Explained)
The smartest people in the world use mental models to make intelligent decisions, avoid stupidity, and increase productivity. Let's take a look at how ...
Crafting a good (reasoning) model
A recent talk I gave on model training, reasoning, and the next frontier.
Modern software quality, or why I think using language models fo…
How to make better software with systems-thinking
Prompts are unsafe, and that means language models are not fit f…
How to make better software with systems-thinking
The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
Recent generations of frontier language models have introduced Large Reasoning Models (LRMs) that generate detailed thinking processes…
Systems that defy detailed understanding
Last week, I wrote about the mindset that computer systems can be understood, and behaviors can be explained, if we’re willing to dig deep enough into the stack of abstractions our software is built atop. Some of the ensuing discussion on Twitter and elsewhere lead me to write this followup, in which I want to run through a few classes of systems where I’ve found pursuing in-detail understanding of the system wasn’t the right answer.
Understanding Reasoning LLMs
Methods and Strategies for Building and Refining Reasoning Models
Summary: Thinking in Systems (by Donella H. Meadows)
The main argument of the book: Everything in this world is made of systems. While we cannot fully understand systems, we can observe them…
Reasoning Models Don't Always Say What They Think
Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model's CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing models' actual reasoning processes. We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. They also suggest that in settings like ours where CoT reasoning is not necessary, test-time monitoring of CoTs is unlikely to reliably catch rare and catastrophic unexpected behaviors.
Reasoning Models Don't Always Say What They Think
Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model's CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing models' actual reasoning processes. We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. They also suggest that in settings like ours where CoT reasoning is not necessary, test-time monitoring of CoTs is unlikely to reliably catch rare and catastrophic unexpected behaviors.
Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
Do LLM agents act on the reasoning they state? We study this question of process fidelity in a controlled Texas Hold’em simulator, where each decision can be decomposed into reasoning, stated conclusion, executed action, and an equity-based reference action. This lets us separate two links in the agent decision pipeline: reasoning→conclusion and conclusion→action. We find that agents usually execute what they explicitly state, and that much of the apparent conclusion→action inconsistency comes from free-text conclusion parsing rather than model behavior. The main fidelity gap appears upstream. In a diagnostic that elicits each agent’s stated equity, pot odds, rule, and decision, errors split across input mistakes, borderline cases, and rule misapplications. For Claude Haiku 4.5, rule misapplications are overwhelmingly risk-averse, and a simple instruction to apply the rule mechanically roughly halves their rate. DeepSeek-Reasoner shows a different error composition, suggesting that the upstream gap is model-dependent. These findings suggest that agent evaluations should elicit machine-checkable conclusions and distinguish execution consistency from reasoning validity. We release a reproducible poker-agent harness and live multi-agent demo for studying process fidelity in LLM agents.
Models
Amp uses the best model for each task: leading generalist foundation models for complex reasoning and planning, and smaller specialized models for fast, accurate responses in specific domains.
