The Right To Be Forgotten vs Audit Trail Mandates: A Tech-Law Expert’s Guidance for Log Management

The scenario is not uncommon: A user or customer wants to erase all traces of their past interactions with your service, as allowed by law. Yet you need to maintain detailed logs for audits or security investigations, as required by law.

You may already have faced such seemingly conflicting requirements from two or more mandates regarding how your company manages log data. Proper log governance can help you thread the needle and shield your organization from the risk of legal risk and reputational damage.

For log data management practices there are generally four types of forcing functions that may put you in the crosshairs of costly penalties, legal entanglements, and/or organizational harm:

• Hard Law. Codified laws and regulations such as the EU’s GDPR, and the U.S. CCPA and HIPAA were created to protect individual privacy rights. As such they prescribe the processing of personal data in computer systems by companies, including log data management. Specifically, this class of laws affords persons the right to have their personally identifying information erased, deleted, or otherwise exercise their ‘right to be forgotten.’ Other laws like Sarbanes-Oxley (SOX) were enacted to protect investors from fraudulent accounting at public companies. While SOX doesn’t specify log management requirements, it does however require companies to implement internal controls and procedures that are reasonable and effective in preventing and ensuring the accuracy and reliability of their financial statements. In practice, this means that regulators and auditors will assess whether companies follow industry standards and best practices such as are set out by ISO 27001, NIST 800-53, and SOC2. All of these indicate the maintenance of logs of user and account activities, exceptions, and events as part of organizations’ information management systems. Whether the purpose is privacy- or organizational security-centric, these mandates have teeth, whether it be criminal and civil fines, suspension of business activities, and/or legal action by the government and in some cases private rights of action.

• Soft Law. These requirements derive not from formal statutes adopted through the legislative process or regs issued by the executive branch but by industry standards and best practices. Foremost examples include FIPPs (Fair Information Practice Principles) and PCI-DSS (Payment Card Industry Data Security Standard). Most if not all privacy laws & regulations are founded on FIPPS and include the right to deletion as one of the core principles. PCI, created by major credit card companies to protect cardholder data, is a set of security standards that companies must comply with in order to process credit card payments. Non-compliance with PCI and FIPPS-based hard law can and do result in significant penalties, fines, litigation, and reputational damage.

• Judge-made Law. AKA case law and litigation, this is a body of law rooted in judicial decisions in individual court cases that may set precedents for security and data management. Poor log management can contribute to whether companies are found to have breached a duty of care or acted negligently in preventing or responding to security incidents and/or handling of personal information. Litigation costs and associated reputational damages are significant sticks for responsible log management, perhaps more weighty than even hard law discussed above.

• Private Law. AKA contractual obligations, log management requirements often arise from agreements between companies and their service providers, clients/customers, and partners. These obligations can dictate the standards and protocols for log management, ensuring adherence to legal, regulatory, and industry standards. They may specify data ownership, access rights, and the retention period for logs, aligning with data privacy laws and the preferences of the parties involved. Contracts often include audit rights to verify compliance with log management practices and outline procedures for incident response, notification, and resolution in the event of security incidents. They may also define liability and indemnification provisions related to log management, and set service level agreements (SLAs) to ensure the availability, integrity, and confidentiality of logged data. Breach of contract can result in various and substantial penalties and damages.

Because this patchwork of requirements originates from differing purposes that are engineered to protect different stakeholders it’s unsurprising that there can be collisions when multiple interests are at stake: corporate data security versus individual privacy rights being front and center in this discussion. The overlapping reach of these requirements can collide at the data retention level. Can you delete personal histories on request without shooting holes in your audit trail? And how do you architect for compliance without finding you’ve made it impossible to fully forget a specific identity?

As you already suspect, there’s no silver bullet approach. But you can drastically reduce the uncertainty of violating these mandates: an ounce of log governance is worth a pound of risk management cure. Log governance is the set of policies, processes, and technical solutions that should dictate the boundaries of your dials. Log governance done right demonstrates due care in your organization’s tradeoff decisions between security compliance and privacy compliance.

A Pragmatic Log Governance Blueprint

• Inventory. Catalog what's collected in your logs and why, minimizing unnecessary personal data. During ingestion, tag or classify data based on its type or sensitivity. This way if a deletion request is received, the system can quickly identify and remove data based on these tags. Ideally use tools that can automatically redact sensitive information from the get-go.

• Granular Management. Ensure your log repositories support the deletion of individual records based on filtering criteria. "Delete" too often only flags events to be skipped in queries. Maintaining an immutable audit trail of processed customer deletion requests can mitigate against disparaging PCI audit reviews when there is ‘missing’ data and the integrity of the logs comes under question.

• Data Minimization or Anonymization. Can your business live without user journeys? If not, you can mask, truncate, tokenize, or use irreversible hashing on sensitive fields such as personal data, including IP addresses. Be aware that you are not off the hook if it’s reasonable to match-link data and re-identify sensitive personal data.

• Automated and Purpose-driven Retention. Craft clear retention policies aligned to legal obligations and operational needs (including client/customer requirements). Automate deletion or archiving of logs and/or fields beyond their retention period. Since your privacy obligations may not terminate when logs are shared with authorized third parties, make sure that they are legally bound by these retention policies, and have an expert deep-dive into their tech to make sure they can actually comply.

• Secure Storage & Access Controls. Adhere to industry-standard security measures to protect log data against unauthorized access or breaches of sensitive personal information. For logs containing sensitive data, employ strong encryption and strict role-based access controls. Manage encryption keys separately and revoke them upon expiration of the retention period to permanently render data inaccessible. Consider applying granular data access controls as part of role definitions.

• Audits & Monitoring. Regularly monitor and audit log data access, modification, and deletions, so you can spot issues before they’re brought to you by surprise. Work with legal/risk experts to ensure these practices are up-to-date with external legal requirements and aligned with your company’s internal data governance plan.

• Transparency & User Rights Management. Provide clear notice to users, customers, and clients about log data practices and retention periods, including what data is being stored, for how long, and for what purposes. Ensure clients have mechanisms to request data access, modifications, or deletions as required by law. Consider utilizing APIs that can handle data rights requests programmatically, so for example when integrated with a Customer Relationship Management (CRM) system, a deletion request in the CRM could trigger a corresponding deletion in the log(s).

• Enlist Subject Matter/Expert Guidance. Design, develop, validate and deploy your log governance with your organization’s appropriate risk control authority. Requirements evolve, so this should be an iterative and ongoing activity.

Future-proof against coming rules for AI

Reconciling the competing data retention and privacy drivers can be challenging, but it’s achievable with the right log governance game plan. Due care is what auditors on all sides fundamentally look for. Regulators are not out to penalize companies that act in good faith. If you can evidence an earnest, ex ante (based on reasonable forecasts) attempt to balance auditors’ requirements, you’ll go far in preempting legal risk.

Note also that none of these often conflicting requirements are absolute and may be limited by exceptions. For example, under GDPR data retention for genuine business, audit, or legal purposes are valid reasons for data processing and can override erasure requirements. The right to deletion under the CCPA also has exceptions, such as when the personal information is necessary for completing a transaction or providing a service requested by the consumer.

The payoff will extend beyond resolving current known-known conflicts to future-proof against a rapidly oncoming train: oversight and regulation of AI. You needn’t be a lawyer to realize that the risks and harms sure to emanate from companies’ enamor with AI today will place more premium on logs tomorrow as a tool for AI forensics and a target of AI risk management. The best way to stay ahead of data regulations is with your own program of log governance.

Erin Kenneally is a licensed attorney & scientist by training and trusted technology risk innovation leader by trade. She has held various leadership positions within the government (U.S. Dept of Homeland Security), R&D (San Diego Supercomputer Center), and industry (cyber insurance risk measurement and modeling) sectors. Kenneally’s expertise lies in translating and harmonizing sociotechnical capability requirements at the crossroads of technology, law, policy, and ethics. She is currently the CEO & Founder of Elchemy.

Right To Be Forgotten is language specific to GDPR. Other mandates talk about erasure or deletion, but RTBF has become a commonly-used term.