In this blog, Rachel Coldicutt OBE, Executive Director, Careful Industries discusses our newly published literature review on the safe adoption of artificial intelligence in engineered systems.
Safety isn’t safety without a social model (or: dispelling the myth of per se technical safety) — AI Alignment Forum
As an AI researcher who wants to do technical work that helps humanity, there is a strong drive to find a research area that is definitely helpful so…
Artificial Intelligence Safety and Security License Requirements
Artificial Intelligence Safety and Security License Requirements Artificial Intelligence (AI) is now broadly capable, which can produce both benefits and dangers. AI will hasten the design and proliferation of bioweapons, cyberweapons, nuclear weapons, progressively more general intelligence,...
Inside the suddenly explosive world of AI safety
AI safety researchers warned that popular models would go rogue. This is only the beginning.
Effective Altruism Is Pushing a Dangerous Brand of ‘AI Safety’
This philosophy—supported by tech figures like Sam Bankman-Fried—fuels the AI research agenda, creating a harmful system in the name of saving humanity
AI for Science & Safety Nodes - Request for Proposals
Artificial intelligence is accelerating the pace of discovery across science and technology. But today’s AI ecosystem risks centralizing compute, talent, and decision-making power – concentrating capabilities in ways that could undermine both innovation and safety.
People know AI is risky. They just don’t know what to do about it. | TrendLife Blog
New international research from TrendLife reveals a wide gap between AI adoption and awareness of AI-powered threats.
More Is Different for AI
Machine learning is touching increasingly many aspects of our society, and its effect will only continue to grow. Given this, I and many others care about risks from future ML systems and how to mitigate them. When thinking about safety risks from ML, there are two common approaches, which I'
AI Researchers On AI Risk
I first became interested in AI risk back around 2007. At the time, most people’s response to the topic was “Haha, come back when anyone believes this besides random Internet crackpots.…
Rethinking security for the age of AI - The Official Microsoft Blog
Editor’s note: Updates with additional details on the model’s crash score. Why security needs a new cyber stack — Introducing Project Perception The physics of cybersecurity are changing. Autonomous systems can now reason, adapt and operate continuously. At the same time, the cost of offense is falling, while the volume, velocity and complexity of what...
A Safe Path to Open Weights
Strategic openness can strengthen AI safety and support broader access as defenses and safety science mature.
How a single tweet transformed the AI safety debate
Most people think AI is risky, but they don’t agree what to do next.
Human-Centered Artificial Intelligence: Three Fresh Ideas
Human-Centered AI (HCAI) is a promising direction for designing AI systems that support human self-efficacy, promote creativity, clarify responsibility, and facilitate social participation. These human aspirations also encourage consideration of privacy, security, environmental protection, social justice, and human rights. This commentary reverses the current emphasis on algorithms and AI methods, by putting humans at the center of systems design thinking, in effect, a second Copernican Revolution. It offers three ideas: (1) a two-dimensional HCAI framework, which shows how it is possible to have both high levels of human control AND high levels of automation, (2) a shift from emulating humans to empowering people with a plea to shift language, imagery, and metaphors away from portrayals of intelligent autonomous teammates towards descriptions of powerful tool-like appliances and tele-operated devices, and (3) a three-level governance structure that describes how software engineering teams can develop more reliable systems, how managers can emphasize a safety culture across an organization, and how industry-wide certification can promote trustworthy HCAI systems. These ideas will be challenged by some, refined by others, extended to accommodate new technologies, and validated with quantitative and qualitative research. They offer a reframe -- a chance to restart design discussions for products and services -- which could bring greater benefits to individuals, families, communities, businesses, and society.
Safety and alignment in an era of long-horizon models
OpenAI shares lessons from deploying long-running AI models, highlighting new safety risks, observed failures, and improved safeguards through iterative deployment.
Safety Co-Option and Compromised National Security: The Self-Fulfilling Prophecy of Weakened AI Risk Thresholds
Risk thresholds provide a measure of the level of risk exposure that a society or individual is willing to withstand, ultimately shaping how we determine the safety of technological systems. Against the backdrop of the Cold War, the first risk analyses, such as those devised for nuclear systems, cemented societally accepted risk thresholds against which safety-critical and defense systems are now evaluated. But today, the appropriate risk tolerances for AI systems have yet to be agreed on by global governing efforts, despite the need for democratic deliberation regarding the acceptable levels of harm to human life. Absent such AI risk thresholds, AI technologists-primarily industry labs, as well as "AI safety" focused organizations-have instead advocated for risk tolerances skewed by a purported AI arms race and speculative "existential" risks, taking over the arbitration of risk determinations with life-or-death consequences, subverting democratic processes. In this paper, we demonstrate how such approaches have allowed AI technologists to engage in "safety revisionism," substituting traditional safety methods and terminology with ill-defined alternatives that vie for the accelerated adoption of military AI uses at the cost of lowered safety and security thresholds. We explore how the current trajectory for AI risk determination and evaluation for foundation model use within national security is poised for a race to the bottom, to the detriment of the US's national security interests. Safety-critical and defense systems must comply with assurance frameworks that are aligned with established risk thresholds, and foundation models are no exception. As such, development of evaluation frameworks for AI-based military systems must preserve the safety and security of US critical and defense infrastructure, and remain in alignment with international humanitarian law.
