Building an LLM safe design system

Our quest to build a scalable, LLM-safe design system

The lethal trifecta for AI agents: private data, untrusted content, and external communication

If you are a user of LLM systems that use tools (you can call them “AI agents” if you like) it is critically important that you understand the risk of …

Reuse your existing hardware to run LLMs privately and securely.

Trellis lets you run large language models on your organization's compute. Scale and data privacy, choose both.

Forcing Generative Models to Degenerate Ones: The Power of Data...

Growing applications of large language models (LLMs) trained by a third party raise serious concerns on the security vulnerability of LLMs.It has been demonstrated that malicious actors can...

A small number of samples can poison LLMs of any size

Anthropic research on data-poisoning attacks in large language models

MCP and LLM Security Research Briefing | Wiz Blog

Explore the evolving Model Context Protocol (MCP), its security risks, and how to prepare for safe adoption as LLMs connect to external systems.

Extensible Software in the age of LLMs | Jeremy Morrell

Solid core + capability-based sandboxes + LLMs = Users with superpowers

Defeating Prompt Injections by Design

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an untrusted environment. However, LLM agents are vulnerable to prompt injection attacks when handling untrusted data. In this paper we propose CaMeL, a robust defense that creates a protective system layer around the LLM, securing it even when underlying models are susceptible to attacks. To operate, CaMeL explicitly extracts the control and data flows from the (trusted) query; therefore, the untrusted data retrieved by the LLM can never impact the program flow. To further improve security, CaMeL uses a notion of a capability to prevent the exfiltration of private data over unauthorized data flows by enforcing security policies when tools are called. We demonstrate effectiveness of CaMeL by solving $77\%$ of tasks with provable security (compared to $84\%$ with an undefended system) in AgentDojo. We release CaMeL at https://github.com/google-research/camel-prompt-injection.

Model Context Protocol has prompt injection security problems

As more people start hacking around with implementations of MCP (the Model Context Protocol, a new standard for making tools available to LLM-powered systems) the security implications of tools built …

Vulnerability Reports Are Not Special Anymore

We needed the insight and confidentiality to protect our users, but now that anyone can get the same results from LLM?

Developer Ecosystems for Software Safety

How to design and implement information systems so they are safe and secure is a complex topic. Both high-level design principles and implementation guidance for software safety and security are well established and broadly accepted. For example, Jerome Saltzer and Michael Schroeder’s seminal overview of principles of secure design was published almost 50 years ago,10 and various community and governmental bodies have published comprehensive best practices about how to avoid common software weaknesses—for example, Common Weakness Enumeration (CWE)a and Open Worldwide Application Security Project (OWASP) Cheat Sheet Series.b

We oppose DRM. | Defective by Design

The Ethical Tech Giving Guide replaces DRM-laden software and devices that trample user freedom and privacy with products and programs that you can trust.

The /llms.txt file – llms-txt

A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

Build agents, not pipelines

There are only two ways to use LLMs in a computer program: as part of a pipeline, or as an agent. In other words, either you express the control flow of the program in code, or you give a LLM tools and allow it to manage the control flow itself1.

Guardian Angels: LLM Personalization for Productivity and Security

I propose an approach for highly personalized LLMs, for near-future productivity gains and personal info/cybersecurity against increasingly powerful LLMs: they should, in the spirit of uploading, try to emulate the user’s values and preferences in order to amplify the principal—not replace them. I discuss a package of techniques and proposals to accomplish such ‘guardian angels’; dynamic evaluation of LLMs combined with active learning and elicitation and heavy inner-monologue search/data-augmentation.