← Writing

Staging with a copy of production, and nothing gets out.

A staging server loaded with a real production copy, where SMS, mail, push, calls and payments are blocked by data scrubbing and a per-user firewall.

Sakshyam Baral··5 min readinfrasecuritylaayo

For a long time Laayo had two places to run code: production, and a test environment. The test environment was fine for building and for automated checks, but its data was thin: few stores, few customers, and menus and offers that looked nothing like the real ones. When I wanted to see how a new screen behaved with a real restaurant's menu, real order history and real zones, the test environment couldn't show me.

So I built a staging server that runs on a copy of the production database. The hard part was not copying the data. It was making sure that a copy of production, with real phone numbers, real push tokens and real payment settings, could not reach a single real person.

The copy is a read-only dump taken in a single transaction, at low CPU and disk priority, so it gets a consistent snapshot without locking anything production needs. I leave out the data of the big log tables (raw GPS history, notification delivery logs, job tables). They are large, nobody tests against them, and they would only slow down every refresh.

Staging runs on a separate machine from production, behind the production server's nginx over a private network, with its own domain and certificate.

Three layers that stop outgoing traffic

I didn't want one switch between staging and a customer's phone. There are three.

Scrub the data. After every import, a scrub script runs before the app is used. It nulls every push token, switches off the settings that would send messages, calls or payment callbacks, and points the realtime socket URL at staging's own server. Then it checks itself with a set of queries that must all return zero. I tested the scrub on a throwaway database first, before it ever touched the real staging copy.

Configure the app to send nothing. Staging's environment file has no real SMS, mail, AI or crash reporting keys. There are no cron jobs and no queue workers, so scheduled reminders, payout messages and delayed notifications simply never run. Sign-in uses a test mode where the one-time code is fixed and nothing is sent.

Firewall the process. The PHP worker pool runs as its own Linux user, and an owner-match firewall rule allows that user to reach the database, Redis, DNS and the map API, and nothing else. The PHP service is set to require the firewall unit, so it cannot start without it.

The firewall has an honest gap. Push notifications go to Google's servers, and so do map requests, so the firewall cannot tell them apart. That is why the scrub matters: with every push token gone, there is nobody to push to. Each layer covers something the others can't.

Because the staging user is the only one firewalled, I never run maintenance commands on staging as any other user. Everything runs as the staging user, so it stays inside the same walls.

Turning things back on, one at a time

A staging server where nothing works is not useful either. Some features need to be tested against the real thing, so each one has its own opt-in script that I run by hand:

  • Payments. The QR payment provider can be enabled with a separate firewall chain that allows only its hosts. Staging payments use an id range far away from production's and a prefix on every payment remark, so they can never be confused with real ones. A guard limits how often staging may log in to the provider and halts after repeated failures.
  • Card payments. The script accepts only test keys and refuses live ones.
  • AI. The AI features get their own firewall chain to the AI providers. I type the key in when the script asks, so it never sits in a file I copy around.
  • Realtime. Staging runs its own instance of the Go realtime server, so live order status and rider tracking work end to end.

Each script has a check mode and an undo mode.

The data copy carries production settings, and that bit me once. Production's admin settings turn the voice assistant on. On staging, before the AI script had been run, the copied setting forced voice on with no key behind it, and the app showed "Couldn't start the assistant". The fix was a guard in the settings code: no key means the feature reports itself as off, and the app falls back to voice search. That guard is useful in production too.

Pointing the apps at staging

The apps need a way to talk to staging that a customer can never trigger. On iOS, a launch argument in Xcode switches the base URL, and the choice is remembered. On Android, debug builds accept a switch through an intent extra, and release builds can be built for staging with a Gradle property. Store builds never pass it.

The remembered choice caused some confusion once. My TestFlight install stayed on staging because of a preference saved by an earlier Xcode run, while a friend's fresh TestFlight install of the same build hit production. Same build, two backends. Now TestFlight and App Store installs ignore the saved preference and read the environment from a build setting, and staging TestFlight builds are archived with that setting changed. Only debug runs from Xcode can switch at runtime.

A few smaller things came up while setting it up. Some stores were closed in the copy, so a script opens every store and makes every item available on staging, and saves the original values in backup tables so it can be undone. Store closures can also come from a per-zone setting, which the first version of that script missed.

How it fits the release flow

Every feature now goes through three places. The test environment is where code is built and proven, with automated checks, shadow comparison and write diffs. Staging is where I use the apps by hand, as a customer, a rider and a restaurant, against data that looks like the real thing. Production comes last, through a script with a check step, a backup and a rollback.

Because it is a copy of production, staging is also the first place a change meets real rows, real menus and real settings, before any customer sees it.

Get new posts by email

One email when something new goes up. No tracking pixels, unsubscribe in one click.