Anthropic announces updates to embedding independent evaluators
The Quest for Embedded Evaluators
Dario Amodei’s essay We Must Pace the Frontier committed Anthropic to embedded evaluators, who would be placed inside Anthropic and given employee-level access, so they could provide outside perspective and also reports on what was happening.
Eval awareness in Claude Opus 4.6’s BrowseComp performance
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
Embedded Evaluators Can’t Fix Companies That Choose to Be Bad — Stella Biderman
Embedded evaluators can report violations, but they cannot fix AI companies that knowingly disregard basic cybersecurity and safety practices.
Updates to Consumer Terms and Privacy Policy
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
Everything I'll forget about Evals | Hrishi Olickel
An actually actionable guide on how to do them
On evaluating agents – aunhumano
No amount of evals will replace the need to look at the data, once you have a evals good coverage you’ll be able to decrease the time but it’ll be always a must to just look at the agent traces to identify possible issues or things to improve.
An update on recent Claude Code quality reports
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
evalstate/fast-agent
Code, Build and Evaluate agents - excellent Model and Skills/MCP/ACP Support
Arvind Narayanan (@aisnakeoil)
Companies check their own work through various internal but independent functional units: QA, security red teams, model risk management in banks. I think it’s time for AI evaluation to become one such unit. Orgs deploying AI should stand up cross-functional eval teams with their own reporting line. Many reasons: 1) Evals as IP / moat. It’s now widely recognized that evals are the new IP. So it makes sense to have teams whose primary focus is on creating and widening this moat. 2) Evals are harder than you think. This is less well recognized but as someone whose research centers on AI evals this has been my consistent experience. It can't be an afterthought and must be a center of excellence. 3) Evals are inherently cross-functional and require a distinct set of skills. They are judgment heavy, require both AI expertise and deep domain expertise, as well as customer understanding and sophisticated thinking about risk. To do them well, you need competence in data science & stats, business operations, product/customer experience, IT, risk management, and even compliance (depending on the sector). 4) In-house but independent eval teams keep companies honest. A climate where teams are getting top-down mandates to hit deployment targets and show results has resulted in a culture of companies fooling themselves. It is extremely easy to knowingly or unknowingly to do evals poorly, making your AI deployment look much more successful than it is. Eval teams who don’t share the deploying teams’ KPIs are the best defense against this.
How building software is changing at Anthropic
A deepdive on what’s changed in how the leading AI lab makes software. Ever more code review and testing is done by AI, two-pizza teams very much alive, and more. Details from inside of Anthropic
Claude’s Constitution
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
Claude’s Constitution
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
Developing Enterprise Frontier Safeguards with our customers
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
Anthropic users face a new choice – opt out or share your chats for AI training | TechCrunch
Anthropic is making some major changes to how it handles user data. Users have until September 28 to take action.
