Managing Privacy in Developer Workflows

Isabel Barberá: "This document provides practical guidance and tools for developers and users of Large Language Model (LLM) based systems to manage privacy risks associated with these technologies. The risk management methodology outlined in this document is designed to help developers and users systematically identify, assess, and mitigate privacy and data protection risks, supporting the responsible development and deployment of LLM systems. This guidance also supports the requirements of the GDPR Article 25 Data protection by design and by default and Article 32 Security of processing by offering technical and organizational measures to help ensure an appropriate level of security and data protection. However, the guidance is not intended to replace a Data Protection Impact Assessment (DPIA) as required under Article 35 of the GDPR. Instead, it complements the DPIA process by addressing privacy risks specific to LLM systems, thereby enhancing the robustness of such assessments. Guidance for Readers > For Developers: Use this guidance to integrate privacy risk management into the development lifecycle and deployment of your LLM based systems, from understanding data flows to how to implement risk identification and mitigation measures. > For Users: Refer to this document to evaluate the privacy risks associated with LLM systems you plan to deploy and use, helping you adopt responsible practices and protect individuals’ privacy. " >For Decision-makers: The structured methodology and use case examples will help you assess the compliance of LLM systems and make informed risk-based decision" European Data Protection Board

If you are an organisation using AI or you are an AI developer, the Australian privacy regulator has just published some vital information about AI and your privacy obligations. Here is a summary of the new guides for businesses published today by the Office of the Australian Information Commissioner which articulate how Australian privacy law applies to AI and set out the regulator’s expectations. The first guide is aimed to help businesses comply with their privacy obligations when using commercially available AI products and help them to select an appropriate product. The second provides privacy guidance to developers using personal information to train generative AI models. GUIDE ONE: Guidance on privacy and the use of commercially available AI products Top five takeaways * Privacy obligations will apply to any personal information input into an AI system, as well as the output data generated by AI (where it contains personal information).  * Businesses should update their privacy policies and notifications with clear and transparent information about their use of AI * If AI systems are used to generate or infer personal information, including images, this is a collection of personal information and must comply with APP 3 (which deals with collection of personal info). * If personal information is being input into an AI system, APP 6 requires entities to only use or disclose the information for the primary purpose for which it was collected. * As a matter of best practice, the OAIC recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools. GUIDE 2: Guidance on privacy and developing and training generative AI models Top five takeaways * Developers must take reasonable steps to ensure accuracy in generative AI models. * Just because data is publicly available or otherwise accessible does not mean it can legally be used to train or fine-tune generative AI models or systems.. * Developers must take particular care with sensitive information, which generally requires consent to be collected. * Where developers are seeking to use personal information that they already hold for the purpose of training an AI model, and this was not a primary purpose of collection, they need to carefully consider their privacy obligations. * Where a developer cannot clearly establish that a secondary use for an AI-related purpose was within reasonable expectations and related to a primary purpose, to avoid regulatory risk they should seek consent for that use and/or offer individuals a meaningful and informed ability to opt-out of such a use. https://lnkd.in/gX_FrtS9

Since the ’90s I’ve built, shipped, and occasionally exploited just about every kind of identity control. We’re now pretty good at building gates around privilege, but not nearly as good at removing it once the job is done. This hurts in 2025. Privileged access no longer lives only with well-defined admin accounts. It threads through every developer workflow, CI/CD script, SaaS connector, and microservice. The result: standing privilege is inevitable, an orphaned token here, a break-glass account there, quietly turning into “forever creds.” Here’s what’s working in the field: → One JIT policy engine that spans cloud, SaaS, and on-prem - no more cloud-specific silos.  ↳ Same approval workflow everywhere, so nobody bypasses “the one tricky platform.”  ↳ Central log stream = single source of truth for auditors and threat hunters. → Bundle-based access: server + DB + repo granted (and revoked) as one unit.  ↳ Devs get everything they need in one click - no shadow roles spun up on the side.  ↳ When the bundle expires, all linked privileges disappear, killing stragglers. → Continuous discovery & auto-kill for any threat that slips through #1 or #2.  ↳ Scan surfaces for compromised creds, role drifts, and partially off-boarded accounts.  ↳ Privilege paths are ranked by risk so teams can cut off the dangerous ones first. Killing standing privilege isn’t a tech mystery anymore, it’s an operational discipline.  What else would you put on the “modern PAM” checklist?

Privacy isn’t a policy layer in AI. It’s a design constraint. The new EDPB guidance on LLMs doesn’t just outline risks. It gives builders, buyers, and decision-makers a usable blueprint for engineering privacy - not just documenting it. The key shift? → Yesterday: Protect inputs → Today: Audit the entire pipeline → Tomorrow: Design for privacy observability at runtime The real risk isn’t malicious intent. It’s silent propagation through opaque systems. In most LLM systems, sensitive data leaks not because someone intended harm but because no one mapped the flows, tested outputs, or scoped where memory could resurface prior inputs. This guidance helps close that gap. And here’s how to apply it: For Developers: • Map how personal data enters, transforms, and persists • Identify points of memorization, retention, or leakage • Use the framework to embed mitigation into each phase: pretraining, fine-tuning, inference, RAG, feedback For Users & Deployers: • Don’t treat LLMs as black boxes. Ask if data is stored, recalled, or used to retrain • Evaluate vendor claims with structured questions from the report • Build internal governance that tracks model behaviors over time For Decision-Makers & Risk Owners: • Use this to complement your DPIAs with LLM-specific threat modeling • Shift privacy thinking from legal compliance to architectural accountability • Set organizational standards for “commercial-safe” LLM usage This isn’t about slowing innovation. It’s about future-proofing it. Because the next phase of AI scale won’t just be powered by better models. It will be constrained and enabled by how seriously we engineer for trust. Thanks European Data Protection Board, Isabel Barberá H/T Peter Slattery, PhD

The story I keep hearing: privacy teams buried in manual reviews, endless forms, and disconnected tools. All necessary, but none of it strategic. It’s compliance as admin, not as leadership. This happens because of a persistent context gap — privacy teams know what data exists, but not the why. That missing context drives the over-reliance on assessments, long review cycles, and duplicated work across teams. If I were leading a privacy function today, here’s what I’d prioritise 👇 1/ Close the context gap before assessing Stop triggering assessments just to find answers. There is emerging AI tooling to connect data from projects, systems, and vendors. 2/ Automate vendor and contract triage Let AI run first-line checks for sub-processors, liability, and transfer risks, freeing teams to focus on the outliers. 3/ Build DSR operations you can trust Automate and track every action and time-to-closure. 4/ Make accountability visible Assign clear owners for systems, datasets, and escalation paths, ensuring human oversight remains in place. 5/ Embed privacy where work happens Governance shouldn’t live in isolation. Bring privacy and AI checks directly into development, procurement, and project workflows so compliance becomes a natural outcome of collaboration. In my recent conversation with Sergio Maldonado on the Masters of Privacy podcast, we discussed how modern privacy teams can close this gap and move from maintenance to impact. The future of privacy operations isn’t about more assessments. It’s about context-aware programs where automation and AI provide the foundation, and built-in know-how provides confidence. Listen to the full episode: Spotify: https://lnkd.in/d4CXx47J YouTube: https://lnkd.in/dd2jpbeH Apple Podcasts: https://lnkd.in/dRD4dkMV

🧪 Let’s talk about something many organizations quietly overlook: data masking in non-production environments. It’s common practice to replicate production data for testing, development, analytics, and training. But what’s not as common? Applying the same level of security controls to these environments as we do to production. The result? Sensitive information—PII, financial records, healthcare data—ends up in the hands of developers, QA testers, contractors, and analysts without the clearance or safeguards required. That’s not just a security gap. It’s a compliance risk—especially under GDPR, HIPAA, PCI DSS, and dozens of other global privacy laws. From what I’ve seen across industries, many data teams still rely on manual methods or basic redaction strategies that don’t scale. The real need is a holistic approach: 🔍 Discover where sensitive data lives (structured, unstructured, semi-structured) 🔐 Apply consistent, policy-based masking rules 🔄 Preserve referential integrity so data remains usable for dev/test 📜 Maintain audit trails to show compliance readiness If your teams are moving data across environments—and most are—this is a conversation worth having. Curious to hear from others: How are you handling data protection in dev/test environments? Is your masking strategy automated and consistent across the org? Have you run into challenges with maintaining format and usability? Let’s exchange ideas. Because in an era of data abundance, privacy must scale with innovation. https://lnkd.in/gVTyijZe #DataSecurity #DataMasking #DevOps #Compliance #PrivacyByDesign #EnterpriseIT #InfoSec #GDPR #HIPAA #NonProductionData #DigitalTrust #DataProtection

AI success isn’t just about innovation - it’s about governance, trust, and accountability. I've seen too many promising AI projects stall because these foundational policies were an afterthought, not a priority. Learn from those mistakes. Here are the 16 foundational AI policies that every enterprise should implement: ➞ 1. Data Privacy: Prevent sensitive data from leaking into prompts or models. Classify data (Public, Internal, Confidential) before AI usage. ➞ 2. Access Control: Stop unauthorized access to AI systems. Use role-based access and least-privilege principles for all AI tools. ➞ 3. Model Usage: Ensure teams use only approved AI models. Maintain an internal “model catalog” with ownership and review logs. ➞ 4. Prompt Handling: Block confidential information from leaking through prompts. Use redaction and filters to sanitize inputs automatically. ➞ 5. Data Retention: Keep your AI logs compliant and secure. Define deletion timelines for logs, outputs, and prompts. ➞ 6. AI Security: Prevent prompt injection and jailbreaks. Run adversarial testing before deploying AI systems. ➞ 7. Human-in-the-Loop: Add human oversight to avoid irreversible AI errors. Set approval steps for critical or sensitive AI actions. ➞ 8. Explainability: Justify AI-driven decisions transparently. Require “why this output” traceability for regulated workflows. ➞ 9. Audit Logging: Without logs, you can’t debug or prove compliance. Log every prompt, model, output, and decision event. ➞ 10. Bias & Fairness: Avoid biased AI outputs that harm users or breach laws. Run fairness testing across diverse user groups and use cases. ➞ 11. Model Evaluation: Don’t let “good-looking” models fail in production. Use pre-defined benchmarks before deployment. ➞ 12. Monitoring & Drift: Models degrade silently over time. Track performance drift metrics weekly to maintain reliability. ➞ 13. Vendor Governance: External AI providers can introduce hidden risks. Perform security and privacy reviews before onboarding vendors. ➞ 14. IP Protection: Protect internal IP from external model exposure. Define what data cannot be shared with third-party AI tools. ➞ 15. Incident Response: Every AI failure needs a containment plan. Create a “kill switch” and escalation playbook for quick action. ➞ 16. Responsible AI: Ensure AI is built and used ethically. Publish internal AI principles and enforce them in reviews. AI without policy is chaos. Strong governance isn’t bureaucracy - it’s your competitive edge in the AI era. 🔁 Repost if you're building for the real world, not just connected demos. ➕ Follow Nick Tudor for more insights on AI + IoT that actually ship.

Even though MCP often sits on familiar OAuth flows, there’s one important invariant: Every call to an MCP server is made by an agent. That gives you a single, well-defined control point to extend your existing policy-as-code and authorization models to be explicitly agent-aware. Two practical principles we lean into with customers: Principle 1. Give agents less power than the humans they represent. In most setups, agents act on behalf of users and carry user identity. That’s convenient—but risky. A solution is to exchange and descope tokens so agents only get the minimum permissions required for their task. Example: The AWS MCP has a lot of tools. Allow a developer to use all those tools, but put guardrails on how their coding agent can use those tools. Stand up a token exchange that maps internal IDP tokens (e.g. Okta) to an AWS IDP token for the developer (using the identity federation features of Okta and AWS), but descope the token to be read-only. Principle 2. Extend your authorization model—don’t replace it. In most organizations, agents work on behalf of users, and so the authorization flow should still rely on the user’s claims. But, it’s reasonable (and often necessary) to add agent-specific authorization policies to the workflow. Example: If you want a read-only version of an MCP server, but it isn’t practical to descope user claims, you can set up a policy to constrain the specific use of given tools. Enterprises need to start defining org-wide authorization policies that augment existing policies, and that’s where Stacklok is extending the policy-as-code boundary to include tool calling. A lot of the early attention on MCP focused on how it introduced security challenges. We see MCP’s potential to unlock new security solutions for agentic workflows.

🔁 If you want privacy risk assessments to matter, stop treating them like paperwork and start treating them like part of how you build. Here’s what I’ve seen actually work: ⚙️ Tie the assessment trigger into product or project intake. Use a checkbox, intake form, or ticket. It doesn’t have to be fancy. 🧠 Legal and privacy shouldn’t be the only ones spotting risk. PMs, engineers, and data science leads need to know what qualifies as “high risk” and when to raise it. 📄 Risk assessments work best when they're short, specific, and reviewed alongside design or architecture; not tacked on after a decision’s already made. 📣 If you’re doing real assessments, share them internally. When others can see how a decision was made (and how a risk was handled), the org learns faster. Don't bolt privacy on at the end. Build the questions into how your team ships. You’ll get better decisions and fewer regrets. #PrivacyByDesign #RiskAssessment #ProductDevelopment #CPRA