Columbia-NLP-Lab/PAPILLON

Clio: Privacy-Preserving Insights into Real-World AI Use

How are AI assistants being used in the real world? While model providers in theory have a window into this impact via their users' data, both privacy concerns and practical challenges have made analyzing this data difficult. To address these issues, we present Clio (Claude insights and observations), a privacy-preserving platform that uses AI assistants themselves to analyze and surface aggregated usage patterns across millions of conversations, without the need for human reviewers to read raw conversations. We validate this can be done with a high degree of accuracy and privacy by conducting extensive evaluations. We demonstrate Clio's usefulness in two broad ways. First, we share insights about how models are being used in the real world from one million Claude.ai Free and Pro conversations, ranging from providing advice on hairstyles to providing guidance on Git operations and concepts. We also identify the most common high-level use cases on Claude.ai (coding, writing, and research tasks) as well as patterns that differ across languages (e.g., conversations in Japanese discuss elder care and aging populations at higher-than-typical rates). Second, we use Clio to make our systems safer by identifying coordinated attempts to abuse our systems, monitoring for unknown unknowns during critical periods like launches of new capabilities or major world events, and improving our existing monitoring systems. We also discuss the limitations of our approach, as well as risks and ethical concerns. By enabling analysis of real-world AI usage, Clio provides a scalable platform for empirically grounded AI safety and governance.

PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of $1{,}000$ users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.

Data Minimisation: a Language-Based Approach (Long Version)

Data minimisation is a privacy-enhancing principle considered as one of the pillars of personal data regulations. This principle dictates that personal data collected should be no more than...

User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies

Hundreds of millions of people now regularly interact with large language models via chatbots. Model developers are eager to acquire new sources of high-quality training data as they race to improve model capabilities and win market share. This paper analyzes the privacy policies of six U.S. frontier AI developers to understand how they use their users' chats to train models. Drawing primarily on the California Consumer Privacy Act, we develop a novel qualitative coding schema that we apply to each developer's relevant privacy policies to compare data collection and use practices across the six companies. We find that all six developers appear to employ their users' chat data to train and improve their models by default, and that some retain this data indefinitely. Developers may collect and train on personal information disclosed in chats, including sensitive information such as biometric and health data, as well as files uploaded by users. Four of the six companies we examined appear to include children's chat data for model training, as well as customer data from other products. On the whole, developers' privacy policies often lack essential information about their practices, highlighting the need for greater transparency and accountability. We address the implications of users' lack of consent for the use of their chat data for model training, data security issues arising from indefinite chat data retention, and training on children's chat data. We conclude by providing recommendations to policymakers and developers to address the data privacy challenges posed by LLM-powered chatbots.

User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies

Hundreds of millions of people now regularly interact with large language models via chatbots. Model developers are eager to acquire new sources of high-quality training data as they race to improve model capabilities and win market share. This paper analyzes the privacy policies of six U.S. frontier AI developers to understand how they use their users' chats to train models. Drawing primarily on the California Consumer Privacy Act, we develop a novel qualitative coding schema that we apply to each developer's relevant privacy policies to compare data collection and use practices across the six companies. We find that all six developers appear to employ their users' chat data to train and improve their models by default, and that some retain this data indefinitely. Developers may collect and train on personal information disclosed in chats, including sensitive information such as biometric and health data, as well as files uploaded by users. Four of the six companies we examined appear to include children's chat data for model training, as well as customer data from other products. On the whole, developers' privacy policies often lack essential information about their practices, highlighting the need for greater transparency and accountability. We address the implications of users' lack of consent for the use of their chat data for model training, data security issues arising from indefinite chat data retention, and training on children's chat data. We conclude by providing recommendations to policymakers and developers to address the data privacy challenges posed by LLM-powered chatbots.

The Consent Layer: Using ligatures to make web text expensive to scrape without asking

ShieldFont is an open-source creative technology project that offers a practical opt-out from unauthorized AI training and disrupts what is collected when that choice is ignored. It swaps 45.8% of content words (around 24.4% of all words) in a page's source code for other (partially) random words, while the font restores the original text on screen. Readers see the work as intended; mass scrapers collect an altered version. In testing, shielding caused over 90% of pages that would otherwise pass the quality filter to be rejected, keeping them out of the training pipeline. Of those that still passed, 19.4% of all words conveyed false meaning, adding noise to unauthorized AI training datasets. This paper's goal is to walk newcomers through the whole process, in plain language and in order: the project's rationale, how it was built, the results, how to deploy it, and where to contribute.

Project Proposal - Social-PETs.EU | Notion

🇪🇺 EU Horizon Call: Privacy Enhancing Technologies

Introducing OpenAI Privacy Filter

OpenAI Privacy Filter is an open-weight model for detecting and redacting personally identifiable information (PII) in text with state-of-the-art accuracy

Permissioned Data PDS Lexicons

I wanted to share a first pass of the PDS lexicons for the permissioned data protocol to get some feedback while the paint’s still wet! You can check them out on my working branch: Comparing main...permissioned-data · bluesky-social/atproto · GitHub They’re all under the com.atproto.space namespace: atproto/lexicons/com/atproto/space at permissioned-data · bluesky-social/atproto · GitHub Confidence rating on Lexicons is probably like ~70-80%. Broad strokes, I think the shape is there. Though ...

fullmoon: local intelligence

chat with private and local large language models

Private Post-Training and Inference for Frontier Models

A technical deep dive of Silo, our local-like privacy stack for cloud-based training and inference of trillion-parameter models.

Forcing Generative Models to Degenerate Ones: The Power of Data...

Growing applications of large language models (LLMs) trained by a third party raise serious concerns on the security vulnerability of LLMs.It has been demonstrated that malicious actors can...

Permissioned Spaces Made Pretty — Lexidraw

A drawing by @essentialrandom.bsky.social on Lexidraw

Evaluating permissioned spaces for community contexts - meri's missives

A look at Bluesky's proposed approach to private data on ATProto