As AI agents become increasingly autonomous in performing tasks for users, ensuring they respect privacy norms is more important than ever. Central to this effort is the concept of contextual integrity, which views privacy as the appropriateness of information flow within specific social settings. For AI systems, this means sharing only relevant information based on the situation, including who is involved, what data is being shared, and why it’s necessary. For example, an AI assistant booking a medical appointment should share details like the patient’s name and medical history but avoid unnecessary information such as insurance details. Similarly, when making lunch reservations, the AI should use available times and preferences without disclosing personal emails or other unrelated data. Maintaining these boundaries is essential for building and preserving user trust.

The Challenge of Contextual Awareness in AI

Today’s large language models (LLMs) often lack the nuanced understanding of context required to uphold privacy norms effectively. This deficiency can lead to inadvertent disclosures of sensitive information, even without explicit malicious prompts. This highlights a broader challenge: AI systems need mechanisms to determine what information is appropriate to include in various scenarios and when to share it. Addressing this gap is critical for ensuring AI aligns with users’ expectations and privacy standards.

Research Approaches to Improve Privacy in AI

Researchers at Microsoft are developing methods to embed contextual integrity into AI systems, helping them manage information more responsibly based on situational norms. Two notable efforts exemplify this approach:

Privacy in Action: The paper “Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents,” accepted at EMNLP 2025, introduces PrivacyChecker. This lightweight module can be integrated into AI agents, enhancing their sensitivity to privacy boundaries. It also offers a new way to evaluate privacy risks by transforming static benchmarks into dynamic environments, revealing higher privacy vulnerabilities during real-world interactions.

Contextual Reasoning and Reinforcement Learning: The paper “Contextual Integrity in LLMs via Reasoning and Reinforcement Learning,” accepted at NeurIPS 2025, explores a different strategy. It treats the enforcement of privacy norms as a reasoning task, requiring AI to carefully analyze the context, the information involved, and the social roles of those involved. This method aims to develop AI that can adaptively apply privacy standards through improved understanding and decision-making.

Both efforts aim to equip AI systems with a deeper sensitivity to the norms governing information sharing, ultimately fostering more trustworthy and privacy-conscious AI interactions.

Inspired by

• https://www.microsoft.com/en-us/research/blog/reducing-privacy-leaks-in-ai-two-approaches-to-contextual-integrity/