Privacy Considerations for Software Developers

If you are an organisation using AI or you are an AI developer, the Australian privacy regulator has just published some vital information about AI and your privacy obligations. Here is a summary of the new guides for businesses published today by the Office of the Australian Information Commissioner which articulate how Australian privacy law applies to AI and set out the regulator’s expectations. The first guide is aimed to help businesses comply with their privacy obligations when using commercially available AI products and help them to select an appropriate product. The second provides privacy guidance to developers using personal information to train generative AI models. GUIDE ONE: Guidance on privacy and the use of commercially available AI products Top five takeaways * Privacy obligations will apply to any personal information input into an AI system, as well as the output data generated by AI (where it contains personal information).  * Businesses should update their privacy policies and notifications with clear and transparent information about their use of AI * If AI systems are used to generate or infer personal information, including images, this is a collection of personal information and must comply with APP 3 (which deals with collection of personal info). * If personal information is being input into an AI system, APP 6 requires entities to only use or disclose the information for the primary purpose for which it was collected. * As a matter of best practice, the OAIC recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools. GUIDE 2: Guidance on privacy and developing and training generative AI models Top five takeaways * Developers must take reasonable steps to ensure accuracy in generative AI models. * Just because data is publicly available or otherwise accessible does not mean it can legally be used to train or fine-tune generative AI models or systems.. * Developers must take particular care with sensitive information, which generally requires consent to be collected. * Where developers are seeking to use personal information that they already hold for the purpose of training an AI model, and this was not a primary purpose of collection, they need to carefully consider their privacy obligations. * Where a developer cannot clearly establish that a secondary use for an AI-related purpose was within reasonable expectations and related to a primary purpose, to avoid regulatory risk they should seek consent for that use and/or offer individuals a meaningful and informed ability to opt-out of such a use. https://lnkd.in/gX_FrtS9

The EDPB recently published a report on AI Privacy Risks and Mitigations in LLMs.   This is one of the most practical and detailed resources I've seen from the EDPB, with extensive guidance for developers and deployers. The report walks through privacy risks associated with LLMs across the AI lifecycle, from data collection and training to deployment and retirement, and offers practical tips for identifying, measuring, and mitigating risks.   Here's a quick summary of some of the key mitigations mentioned in the report:   For providers: • Fine-tune LLMs on curated, high-quality datasets and limit the scope of model outputs to relevant and up-to-date information. • Use robust anonymisation techniques and automated tools to detect and remove personal data from training data. • Apply input filters and user warnings during deployment to discourage users from entering personal data, as well as automated detection methods to flag or anonymise sensitive input data before it is processed. • Clearly inform users about how their data will be processed through privacy policies, instructions, warning or disclaimers in the user interface. • Encrypt user inputs and outputs during transmission and storage to protect data from unauthorized access. • Protect against prompt injection and jailbreaking by validating inputs, monitoring LLMs for abnormal input behaviour, and limiting the amount of text a user can input. • Apply content filtering and human review processes to flag sensitive or inappropriate outputs. • Limit data logging and provide configurable options to deployers regarding log retention. • Offer easy-to-use opt-in/opt-out options for users whose feedback data might be used for retraining.   For deployers: • Enforce strong authentication to restrict access to the input interface and protect session data. • Mitigate adversarial attacks by adding a layer for input sanitization and filtering, monitoring and logging user queries to detect unusual patterns. • Work with providers to ensure they do not retain or misuse sensitive input data. • Guide users to avoid sharing unnecessary personal data through clear instructions, training and warnings. • Educate employees and end users on proper usage, including the appropriate use of outputs and phishing techniques that could trick individuals into revealing sensitive information. • Ensure employees and end users avoid overreliance on LLMs for critical or high-stakes decisions without verification, and ensure outputs are reviewed by humans before implementation or dissemination. • Securely store outputs and restrict access to authorised personnel and systems.   This is a rare example where the EDPB strikes a good balance between practical safeguards and legal expectations. Link to the report included in the comments.   #AIprivacy #LLMs #dataprotection #AIgovernance #EDPB #privacybydesign #GDPR

FTC Highlights Key Practices to Mitigate Cybersecurity Risks in Product Development As technology evolves, so do digital threats. The Federal Trade Commission (FTC) recently released vital recommendations to address cybersecurity risks linked to the development of AI, targeted advertising, and other data-intensive products. These risks stem from companies creating "valuable pools" of personal information that bad actors can exploit. Core Recommendations: Data Management - Enforce data retention schedules to limit unnecessary data storage. - Mandate deletion of improperly collected or retained data, including algorithms trained on such data. - Encrypt sensitive data to prevent unauthorized access. Secure Software Development: - Adopt “secure by design” principles, such as using memory-safe programming languages. - Conduct rigorous pre-release testing to identify vulnerabilities early. - Secure external product access with monitoring and intrusion detection systems. Human-Centric Product Design: - Implement phishing-resistant multi-factor authentication (MFA). - Enforce least-privilege access controls for employees handling sensitive data. - Avoid deceptive design patterns (e.g., "dark patterns") that compromise user privacy. The FTC underscores the importance of addressing systemic vulnerabilities and safeguarding consumers from digital security threats. With these actionable steps, companies can better protect data, ensure privacy, and enhance trust. Read the full details and explore related enforcement actions here: https://buff.ly/3PpuavB

𝗗𝗮𝘁𝗮 𝗽𝗿𝗶𝘃𝗮𝗰𝘆 𝗶𝘀𝗻’𝘁 𝗷𝘂𝘀𝘁 𝗮 𝗹𝗲𝗴𝗮𝗹 𝗱𝗲𝗽𝗮𝗿𝘁𝗺𝗲𝗻𝘁 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 - it’s something AI developers need to think about while building. As we wrap up the final week of the 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗔𝗜 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿 𝗖𝗲𝗿𝘁𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗣𝗿𝗼𝗴𝗿𝗮𝗺, today’s lesson focuses on one of the most important (and most often overlooked) topics: 𝗰𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲 𝗮𝗻𝗱 𝗱𝗮𝘁𝗮 𝗽𝗿𝗶𝘃𝗮𝗰𝘆 𝗶𝗻 𝗮𝗴𝗲𝗻𝘁𝗶𝗰 𝘀𝘆𝘀𝘁𝗲𝗺𝘀. 𝗜𝘁’𝘀 𝗲𝗮𝘀𝗶𝗲𝗿 𝘁𝗵𝗮𝗻 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝘁𝗼 𝗹𝗲𝗮𝗸 𝘀𝗲𝗻𝘀𝗶𝘁𝗶𝘃𝗲 𝗱𝗮𝘁𝗮 in LLM-based systems: - A RAG assistant pulls a patient case study and quotes it in a live response. - A chatbot stores user conversations without clear consent and guardrails. - A logging tool captures names, emails, or health info and sends it to a third-party service. These aren't rare edge cases. They’re common mistakes. And if you're not paying attention to 𝗚𝗗𝗣𝗥, 𝗛𝗜𝗣𝗔𝗔, 𝗦𝗢𝗖 𝟮, or basic privacy principles - your system could be non-compliant by default. In this lesson, we break down: - What these frameworks actually expect from developers (no legalese) - Where agentic systems unintentionally leak data - What responsible design looks like (with examples and a developer checklist) - How to build privacy-aware AI systems from day one - without slowing down development 𝗥𝗲𝗮𝗱 𝘁𝗵𝗲 𝗳𝘂𝗹𝗹 𝗹𝗲𝘀𝘀𝗼𝗻 𝗵𝗲𝗿𝗲 (𝗻𝗼 𝘀𝗶𝗴𝗻-𝘂𝗽 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗱): https://lnkd.in/d7vbRp5H This is part of the Agentic AI Developer Certification Program - a free, 12-week applied learning experience from Ready Tensor. 𝗟𝗲𝗮𝗿𝗻 𝗺𝗼𝗿𝗲 𝗮𝗻𝗱 𝗷𝗼𝗶𝗻 𝘁𝗵𝗲 𝗻𝗲𝘅𝘁 𝗰𝗼𝗵𝗼𝗿𝘁: https://lnkd.in/g23KZ9yH

This Stanford study examined how six major AI companies (Anthropic, OpenAI, Google, Meta, Microsoft, and Amazon) handle user data from chatbot conversations.  Here are the main privacy concerns. 👀 All six companies use chat data for training by default, though some allow opt-out 👀 Data retention is often indefinite, with personal information stored long-term 👀 Cross-platform data merging occurs at multi-product companies (Google, Meta, Microsoft, Amazon) 👀 Children's data is handled inconsistently, with most companies not adequately protecting minors 👀 Limited transparency in privacy policies, which are complex and hard to understand and often lack crucial details about actual practices Practical Takeaways for Acceptable Use Policy and Training for nonprofits in using generative AI: ✅ Assume anything you share will be used for training - sensitive information, uploaded files, health details, biometric data, etc. ✅ Opt out when possible - proactively disable data collection for training (Meta is the one where you cannot) ✅ Information cascades through ecosystems - your inputs can lead to inferences that affect ads, recommendations, and potentially insurance or other third parties ✅ Special concern for children's data - age verification and consent protections are inconsistent Some questions to consider in acceptable use policies and to incorporate in any training. ❓ What types of sensitive information might your nonprofit staff  share with generative AI?  ❓ Does your nonprofit currently specifically identify what is considered “sensitive information” (beyond PID) and should not be shared with GenerativeAI ? Is this incorporated into training? ❓ Are you working with children, people with health conditions, or others whose data could be particularly harmful if leaked or misused? ❓ What would be the consequences if sensitive information or strategic organizational data ended up being used to train AI models? How might this affect trust, compliance, or your mission? How is this communicated in training and policy? Across the board, the Stanford research points that developers’ privacy policies lack essential information about their practices. They recommend policymakers and developers address data privacy challenges posed by LLM-powered chatbots through comprehensive federal privacy regulation, affirmative opt-in for model training, and filtering personal information from chat inputs by default. “We need to promote innovation in privacy-preserving AI, so that user privacy isn’t an afterthought." How are you advocating for privacy-preserving AI? How are you educating your staff to navigate this challenge? https://lnkd.in/g3RmbEwD

Isabel Barberá: "This document provides practical guidance and tools for developers and users of Large Language Model (LLM) based systems to manage privacy risks associated with these technologies. The risk management methodology outlined in this document is designed to help developers and users systematically identify, assess, and mitigate privacy and data protection risks, supporting the responsible development and deployment of LLM systems. This guidance also supports the requirements of the GDPR Article 25 Data protection by design and by default and Article 32 Security of processing by offering technical and organizational measures to help ensure an appropriate level of security and data protection. However, the guidance is not intended to replace a Data Protection Impact Assessment (DPIA) as required under Article 35 of the GDPR. Instead, it complements the DPIA process by addressing privacy risks specific to LLM systems, thereby enhancing the robustness of such assessments. Guidance for Readers > For Developers: Use this guidance to integrate privacy risk management into the development lifecycle and deployment of your LLM based systems, from understanding data flows to how to implement risk identification and mitigation measures. > For Users: Refer to this document to evaluate the privacy risks associated with LLM systems you plan to deploy and use, helping you adopt responsible practices and protect individuals’ privacy. " >For Decision-makers: The structured methodology and use case examples will help you assess the compliance of LLM systems and make informed risk-based decision" European Data Protection Board

In an era where digital tools play a crucial role in our personal safety, ensuring the security of user data within safety mobile apps is more important than ever. As these apps handle sensitive information, robust cybersecurity measures are essential to protect users from potential threats. Here’s why data security matters and how developers can ensure user information is protected: Safety apps often collect sensitive personal information, such as location data and emergency contacts, making the protection of this data crucial for maintaining user trust and privacy. To ensure data security, developers can employ strong encryption methods for data storage and transmission, such as end-to-end encryption, to prevent unauthorized access. Regular security audits and vulnerability assessments are essential for identifying potential security risks, allowing developers to proactively address these issues before they are exploited. Implementing multi-factor authentication (MFA) provides an additional layer of security by ensuring only authorized users can access the app and its features. Clear and transparent privacy policies are vital for informing users about how their data is collected, used, and protected, thus building trust and empowering them to make informed decisions. Regular updates and security patches are necessary to address vulnerabilities and defend against emerging threats, while user education on best practices, like setting strong passwords and recognizing phishing attempts, further enhances data security and empowers users to protect their information. #Cybersecurity #DataProtection #SafetyApps #Privacy #TechForGood

Privacy isn’t a policy layer in AI. It’s a design constraint. The new EDPB guidance on LLMs doesn’t just outline risks. It gives builders, buyers, and decision-makers a usable blueprint for engineering privacy - not just documenting it. The key shift? → Yesterday: Protect inputs → Today: Audit the entire pipeline → Tomorrow: Design for privacy observability at runtime The real risk isn’t malicious intent. It’s silent propagation through opaque systems. In most LLM systems, sensitive data leaks not because someone intended harm but because no one mapped the flows, tested outputs, or scoped where memory could resurface prior inputs. This guidance helps close that gap. And here’s how to apply it: For Developers: • Map how personal data enters, transforms, and persists • Identify points of memorization, retention, or leakage • Use the framework to embed mitigation into each phase: pretraining, fine-tuning, inference, RAG, feedback For Users & Deployers: • Don’t treat LLMs as black boxes. Ask if data is stored, recalled, or used to retrain • Evaluate vendor claims with structured questions from the report • Build internal governance that tracks model behaviors over time For Decision-Makers & Risk Owners: • Use this to complement your DPIAs with LLM-specific threat modeling • Shift privacy thinking from legal compliance to architectural accountability • Set organizational standards for “commercial-safe” LLM usage This isn’t about slowing innovation. It’s about future-proofing it. Because the next phase of AI scale won’t just be powered by better models. It will be constrained and enabled by how seriously we engineer for trust. Thanks European Data Protection Board, Isabel Barberá H/T Peter Slattery, PhD

The Case for App Scanning and SDK Governance: Lessons from Texas Lawsuit The State of Texas has filed a lawsuit against a large insurance company and its analytics subsidiary for alleged violations of the Texas Data Privacy and Security Act (TDPSA), the Data Broker Law, and the Texas Insurance Code. What happened: - A large insurance company and its analytics subsidiary created a Software Development Kit (SDK), that was embedded into third-party apps offering location-based services. - This SDK secretly collected sensitive user data, including precise locations, speed, direction, and other phone sensor data, without users' awareness. - The collected data was used to create a massive driving behaviour database covering millions of users. - This data was monetized, influencing insurance premiums and policies, often without users' knowledge or consent. - Users were not informed about how their data was being collected or shared, and privacy policies were not clear or accessible. Key issues: 1) No user consent: People did not know their data was being collected or sold. 2) Inaccurate profiling: The SDK often mistook passengers or other scenarios as "bad driving," leading to misleading profiles. 3 ) Non-compliance: The analytics subsidiary failed to register as a data broker, as required by Texas law. Why this matters: This case highlights the risks of hidden data collection in apps. It shows how companies can misuse sensitive data and the importance of protecting user privacy through stronger controls. The way forward: To effectively address these risks, organizations must take assertive action by implementing the following measures - a) Conduct regular mobile app scanning: Analyze apps weekly or bi-weekly to identify permissions, embedded SDKs, and dataflows. b) Govern SDKs effectively: Establish strict policies for integrating and monitoring SDKs. Require transparency from SDK providers about what data is collected, how it is used, and who it is shared with. Avoid SDKs that fail to meet these standards. c) Monitor hidden dataflows: SDKs often operate in the background and can rely on permissions obtained by the app to collect sensitive data. Regularly audit these dataflows to uncover any implicit collection or sharing practices and address potential violations proactively. d) Communicate transparently with users: Update #privacy policies to clearly explain what data is collected, how it will be used, and who it will be shared with. Obtain explicit consent before collecting or sharing sensitive data. The risks of hidden #dataflows and implicit data collection are significant, especially as #SDKs become more complex. How frequently does your team #audit apps for SDK behaviors and permissions? What tools or strategies have you found most effective in uncovering hidden #datasharing?

‼️📱The CNIL - Commission Nationale de l'Informatique et des Libertés has released its final recommendations for mobile application developers to help them ensure privacy protection. These guidelines, which will be enforced starting in 2025, target all parties involved in creating and distributing mobile apps, including app publishers, developers, SDK providers, operating system vendors, and app store providers. 📍The recommendations emphasise that all stakeholders must cooperate to protect personal data throughout the app development process. The document outlines the responsibilities of each group, helping them navigate legal requirements and collaborate effectively to guarantee data protection. For instance, app publishers make the software available to users, while developers are responsible for writing the app's code. SDK providers offer pre-built functionalities like audience measurement tools while operating system providers such as iOS and Android enable apps to function on mobile devices. 📍One of the primary goals of the recommendations is to clarify each stakeholder's role and ensure they understand their responsibilities. This includes advice on how to inform users about their data use better. User consent is another critical focus, especially when apps request data for purposes beyond the app's functionality, such as targeted advertising. The CNIL stresses that consent must be freely given, and users should be able to refuse or withdraw consent at any time without facing hurdles. 📍To combat the overwhelming nature of consent requests, CNIL advises developers to collect consent in a way that is contextual and easier for users to understand. This means seeking consent based on the user's actions at appropriate moments rather than bombarding them with requests upfront. Additionally, while technical permissions (such as access to location data or camera) allow apps to function, they do not necessarily constitute legal consent under data protection laws. Therefore, developers must implement a Consent Management Platform alongside technical permissions. 📍CNIL clarifies when developers are considered data processors or data controllers under the regulation. If a developer provides the app's code and has no further role in its operation or data processing, they are not responsible under GDPR. However, if they process data on behalf of the publisher, they are considered data processors and must ensure the app's design complies with GDPR's data protection principles if the developer uses personal data for their purposes, such as improving other apps or offering new services. In that case, they may be classified as a data controller and must obtain the app publisher's approval before using the data for additional purposes. #mobileapps #gdpr #privacy