Differential Privacy - Microsoft Research

Differential Privacy

A robust yet accessible introduction to the idea, history, and key applications of differential privacy—the gold standard of algorithmic privacy protection

Privacy Architectures: Reasoning About Data Minimisation and Integrity

Privacy by design will become a legal obligation in the European Community if the Data Protection Regulation eventually gets adopted. However, taking into account privacy requirements in the design of a system is a challenging task. We propose an approach based on the specification of privacy architectures and focus on a key aspect of privacy, data minimisation, and its tension with integrity requirements. We illustrate our formal framework through a smart metering case study.

Permissioned Data Shapes: Private Events - Nick's Blog

Stratos: Lets get Technical

A deeper look at permission private data with Stratos

Data Minimisation in Communication Protocols: A Formal Analysis...

With the growing amount of personal information exchanged over the Internet, privacy is becoming more and more a concern for users. One of the key principles in protecting privacy is data...

Privacy by design: a formal framework for the analysis of architectural choices

The privacy by design approach has already been applied in different areas. We believe that the next challenge in this area today is to go beyond individual cases and to provide methodologies to explore the design space in a systematic way. As a first step in this direction, we focus in this paper on the data minimization principle and consider different options using decentralized architectures in which actors do not necessarily trust each other. We propose a framework to express the parameters to be taken into account (the service to be performed, the actors involved, their respective requirements, etc.) and an inference system to derive properties such as the possibility for an actor to detect potential errors (or frauds) in the computation of a variable. This inference system can be used in the design phase to check if an architecture meets the requirements of the parties or to point out conflicting requirements.

Privacy by Design

In view of rapid and dramatic technological change, it is important to take the special requirements of privacy protection into account early on, because new technological systems often contain hidden dangers which are very difficult to overcome after the basic design has been worked out. So it makes all the more sense to identify and examine possible data protection problems when designing new technology and to incorporate privacy protection into the overall design, instead of having to come up with laborious and time-consuming “patches” later on. This approach is known as “Privacy by Design” (PbD).

Configurable Per-Query Data Minimization for Privacy-Compliant Web APIs

The purpose of regulatory data minimization obligations is to limit personal data to the absolute minimum necessary for a given context. Beyond the initial data collection, storage, and...

openPDS/SafeAnswers - The privacy-preserving Personal Data Store

Protecting the Privacy of Metadata through SafeAnswers

Data Minimisation: a Language-Based Approach (Long Version)

Data minimisation is a privacy-enhancing principle considered as one of the pillars of personal data regulations. This principle dictates that personal data collected should be no more than...

Unlinkable Inference as a User Privacy Architecture

Unlinkable inference is a technique that provably sandboxes your AI requests from each other and from your identity. We discuss its building blocks, applications, and how it fits into the broader landscape of private personal intelligence.

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.

Vertical Federated Learning: Concepts, Advances, and Challenges

Vertical Federated Learning (VFL) is a federated learning setting where multiple parties with different features about the same set of users jointly train machine learning models without exposing their raw data or model parameters. Motivated by the rapid growth in VFL research and real-world applications, we provide a comprehensive review of the concept and algorithms of VFL, as well as current advances and challenges in various aspects, including effectiveness, efficiency, and privacy. We provide an exhaustive categorization for VFL settings and privacy-preserving protocols and comprehensively analyze the privacy attacks and defense strategies for each protocol. In the end, we propose a unified framework, termed VFLow, which considers the VFL problem under communication, computation, privacy, as well as effectiveness and fairness constraints. Finally, we review the most recent advances in industrial applications, highlighting open challenges and future directions for VFL.

Confidential Inference via Trusted Virtual Machines

Announcing a new collaborative research paper on Confidential Inference, a set of tools to improve the security of our model weights and of our users' data