Building an Automated Data Retention and Erasure Pipeline

Section 8(7) creates an erasure obligation that fires on two distinct triggers: the purpose for which data was collected is no longer being served, or the Data Principal withdraws consent — whichever happens earlier. These are not the same engineering problem. Purpose expiry is typically time-based and predictable, like an inactive account past a defined threshold, and can be handled by a scheduled retention job. Consent withdrawal is event-based and unpredictable, arriving at any time, and needs a reactive pipeline that erasure kicks off immediately rather than waiting for the next scheduled sweep.

A pipeline that only handles one of these will look compliant in a demo and fail the first time someone withdraws consent mid-cycle and the data sits untouched until the next scheduled job runs, if there is one at all. Design for both triggers from the start, sharing the same downstream erasure execution logic but fed by two different upstream sources: a scheduler for time-based expiry, and an event listener for consent state changes.

The single biggest source of retention pipeline failure is not knowing, at deletion time, why a given record was originally collected or what retention rule applies to it. Retroactively figuring this out for a table with mixed-purpose data is close to impossible at scale. The fix is to tag data with its purpose and retention basis when it is first written, as metadata alongside the record, so the retention engine can query 'what applies to this record' directly instead of inferring it later.

This tagging needs to survive the record through its lifecycle: if data moves from a transactional store into a data warehouse, an analytics pipeline, or a cache, the purpose and retention metadata should travel with it, not get dropped at the first ETL step. Otherwise you end up with a retention pipeline that works cleanly on the primary database and silently does nothing for the four downstream copies of the same data.

Section 8(7) requires the Data Fiduciary to cause any Data Processor engaged with that data to also erase it, which means your erasure pipeline cannot stop at your own infrastructure boundary. Every processor integration needs a corresponding erasure API call or contractual mechanism, triggered from the same event that starts your internal erasure, and — critically — a way to confirm the processor actually completed it rather than just accepted the request.

This is where a lot of pipelines are incomplete: they fire a deletion webhook to a processor and consider the job done, with no reconciliation step. Build in a status check, even a periodic one, that confirms processor-side erasure completed, and treat processors who cannot provide any confirmation mechanism as a gap to flag during vendor assessment rather than something to quietly accept.

Not every erasure request can execute immediately in full. Legal holds, active disputes, and statutory retention obligations under other laws can override a DPDP erasure trigger temporarily, and the pipeline needs an explicit hold state that pauses erasure with a recorded reason, rather than silently skipping records that later get missed entirely. An erasure pipeline with no hold mechanism tends to get manual overrides bolted onto it under pressure, which is exactly the kind of undocumented exception that fails an audit.

Backups are the other structural exception, and they deserve their own design rather than being treated as an edge case of the primary pipeline — the interaction between backup retention and erasure rights is significant enough to warrant a dedicated approach.

The Retention Planner tool on this site helps map purpose-by-purpose retention periods before you build the pipeline logic around them, which makes the tagging-at-ingestion step in this pipeline considerably more accurate.