Limitless Wealth
Article

Developing against production data without holding it

Every team wants realistic data in development, because synthetic seeds hide the bugs that only appear at real scale and real messiness. Almost every team gets it the easy way: a production dump restored somewhere less protected. That works until it is the thing on the incident report.

Why the shortcut is worse than it looks

A production copy in staging is not simply a smaller risk than production. In several respects it is a larger one.

  • Access is broader. Production access is usually restricted and audited. Staging is where contractors, new starters and third-party tools get credentials on day one.
  • Controls are weaker. Staging rarely has the same network restrictions, logging or alerting. A dump sitting in object storage often has none at all.
  • Copies multiply. One restore becomes a local copy on four laptops, a snapshot in CI, and an old bucket nobody remembers.
  • Email is live. The most common version of this incident is not a breach at all - it is a test run that sends real messages to real customers from an environment nobody thought was connected to anything.

Under GDPR the data has not stopped being personal data because it moved environments. The obligations follow the data.

An anonymisation pipeline

The approach that works is a scheduled job producing an anonymised snapshot that anyone can use freely, with production access required only to create it, never to consume it.

  1. Dump production, inside the production security boundary.
  2. Restore into a temporary isolated database.
  3. Run anonymisation against that copy.
  4. Verify, then export the anonymised result to storage.
  5. Destroy the temporary database.
  6. Developers pull the anonymised snapshot with a single command.

The property that matters: the raw dump never leaves the production boundary. Only the anonymised artefact does. Triggering a fresh snapshot is permission-gated; using one is not.

Anonymisation that survives contact with reality

The naive version overwrites every name with "Test User" and every email with test@example.com. It is safe and it is useless - you have destroyed exactly the variation you wanted the data for.

Better rules, in rough order of importance:

Keep the shape, replace the content. A name becomes a different plausible name of similar length. An address becomes a real-format address in the same country. Sort orders, pagination behaviour and layout bugs stay reproducible.

Be deterministic. The same input should produce the same output every run. Then a bug found on last week's snapshot is still findable on this week's, and support can match a report to a record without ever seeing the real value.

Preserve referential integrity. If a customer email appears in three tables, all three must become the same replacement, or joins break and half your application stops working.

Keep distributions. Do not replace every order value with £10. Percentile-heavy bugs live in the tail, and a uniform dataset hides them.

Neutralise the outbound channels. Rewrite every email address and phone number to a domain and range you control. This is the single highest-value line in the whole pipeline.

An anonymiser that makes the data useless will be bypassed within a month, and then you have the original problem plus a false sense of having solved it.

Fail closed

The dangerous failure is a new column arriving that nobody added a rule for. Someone adds national_insurance_number, the anonymiser does not know about it, and it passes through untouched into every laptop in the company.

The defence is an explicit schema. Every column is classified - anonymise, keep, or drop - and an unclassified column fails the run. Noisy by design, and the noise is a developer being asked one question at the point they add a field, which is precisely when they know the answer.

What you get

Realistic data everywhere, a one-command reset for new engineers, CI running against something resembling production, and a straight answer when a client asks whether your developers can see their customers' details.

It is a few days of work and it removes an entire category of incident. Of the things a growing team can do for its security posture, this one has the best ratio of effort to risk removed.

We have built anonymisation pipelines for databases with real regulatory exposure. If your staging environment currently holds a production restore, we can help you get off it without losing the data quality your team relies on.

contact@limitlesswealth.xyz