How do leading AI models perform on real news questions? NewsBench scores them on accuracy, neutrality, and source quality with expert-calibrated judges.
NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting
As frontier AI systems increasingly serve as sources of political and current-events information, evaluating the epistemic quality of their responses is now especially consequential. Existing evaluations often focus narrowly on ideological alignment, rely on measures not validated for this setting, or depend on reference labels derived from public opinion or non-expert annotation, potentially producing misleading or miscalibrated results. To address these limitations, we introduce NewsBench, an expert-grounded benchmark that reframes evaluation around editorial standards developed by senior practitioners in journalism, policy, and intelligence analysis. NewsBench operationalizes quality across three dimensions, source quality, factuality, and neutrality, using a two-phase pipeline in which domain experts first define and calibrate evaluation guidelines, which are then transferred to automated AI judges validated against expert-adjudicated gold labels. The benchmark includes approximately $3,000$ evaluation prompts, with roughly $500$ refreshed monthly to capture emerging current events, roughly $2,500$ human expert-annotated items for calibration and validation, and a suite of AI judges designed to enable scalable evaluation. To validate our methodology, we report both inter-rater reliability among human experts and AI judge--human agreement across all three dimensions. Finally, we apply NewsBench to four frontier models, revealing distinct and non-overlapping failure profiles in sourcing, factuality, and neutrality.
NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting
As frontier AI systems increasingly serve as sources of political and current-events information, evaluating the epistemic quality of their responses is now especially consequential. Existing evaluations often focus narrowly on ideological alignment, rely on measures not validated for this setting, or depend on reference labels derived from public opinion or non-expert annotation, potentially producing misleading or miscalibrated results. To address these limitations, we introduce NewsBench, an expert-grounded benchmark that reframes evaluation around editorial standards developed by senior practitioners in journalism, policy, and intelligence analysis. NewsBench operationalizes quality across three dimensions, source quality, factuality, and neutrality, using a two-phase pipeline in which domain experts first define and calibrate evaluation guidelines, which are then transferred to automated AI judges validated against expert-adjudicated gold labels. The benchmark includes approximately $3,000$ evaluation prompts, with roughly $500$ refreshed monthly to capture emerging current events, roughly $2,500$ human expert-annotated items for calibration and validation, and a suite of AI judges designed to enable scalable evaluation. To validate our methodology, we report both inter-rater reliability among human experts and AI judge--human agreement across all three dimensions. Finally, we apply NewsBench to four frontier models, revealing distinct and non-overlapping failure profiles in sourcing, factuality, and neutrality.
AI Policy • States Newsroom
AI Policy States Newsroom is, above all, dedicated to journalism practiced by people. We believe in the power of having on-the-ground reporters covering our communities with the finesse and nuance that only human interaction can produce. We also recognize the power and potential of generative AI. We are open to the responsible and transparent use […]
AI Model Leaderboards & Benchmarks
Explore leaderboards with expert-driven LLM benchmarks and updated AI model rankings across coding, reasoning and more.
Artificial Intelligence in the News: How AI Retools, Rationalizes, and Reshapes Journalism and the Public Arena
<p>Download the pdf here. Executive Summary Despite growing interest, the effects of AI on the news industry and our information environment — the public arena — remain poorly understood. Insufficient attention has also been paid to the implications of the news industry’s dependence on technology companies for AI. Drawing on 134 interviews with news workers […]</p>
WebDev AI Leaderboard - Best AI Models for Web Development
View overall rankings across AI models on front-end web development tasks, including agentic coding workflows that require multi-step reasoning and tool use.
AI False Claims Monitor
As the domain experts in data reliability in the topic of news and information, NewsGuard provides the leading red-teaming analysis for information reliability. AI models continue to face significant challenges in ensuring their models provide safe, accurate responses to prompts instead of spreading false claims on the internet or refusing to respond to topics in the news.
People are getting their news from AI – and it’s altering their views
Even when information is factually accurate, how it’s presented can introduce subtle biases. As large language models increasingly bring people the news, this bias is a looming problem.
People are getting their news from AI – and it’s altering their views
Even when information is factually accurate, how it’s presented can introduce subtle biases. As large language models increasingly bring people the news, this bias is a looming problem.
LLM Leaderboard - Best Text & Chat AI Models Compared
Compare and explore Text models ranked by overall performance.
May 2025 — AI Misinformation Monitor of Leading AI Chatbots
Every month, NewsGuard’s team of expert analysts audit the top AI models to see how well they respond to prompts in the news. On average, this month they failed to counter disinformation from Russia’s Pravda Network 24 percent of the time, either spreading false claims or failing to respond. The Pravda network was designed to infect the AI models.
WSJ Artificial Intelligence Tools - WSJ
The Wall Street Journal and Dow Jones Newswires use artificial-intelligence tools in a variety of ways in our journalism. For example, we use AI and machine learning in complex data-driven investigations, to summarize articles, to translate some content into local languages and to power website features such as audio versions of our articles. We are also using generative AI to write alt text descriptions of visuals to improve accessibility.
AI in journalism: Live tracker of scandals and mistakes
AI in journalism: Live tracker of mistakes and mishaps from the Mississippe Free Press to the New York Times.
Who earned the score? - Sensemaker
This week's AI claims blurred models, systems, simulations and people. The evidence becomes clearer when the tested subject comes first.
Arena AI: The Official AI Ranking & LLM Leaderboard
Chat, compare, vote for the world's best AI models. Join the community shaping the public leaderboard for LLMs, image, and code models through real-world evaluation.
Arena AI: The Official AI Ranking & LLM Leaderboard
Chat, compare, vote for the world's best AI models. Join the community shaping the public leaderboard for LLMs, image, and code models through real-world evaluation.