Building a deployment you can undo — code, configuration, database, and traffic together

Treat a deployment as ten layers, not one change

A rollback-ready deployment is not a stored copy of the previous executable. The old and new versions have to handle the same configuration, database schema, data formats, and external contracts together for a period. On top of that, feature exposure and traffic must be reducible or cuttable independently, and it must already be decided who does what when a given signal appears.

Rolling only the code back sometimes makes the incident worse. The new deployment may already have dropped a column or converted data into a lossy form. Secrets may have been revoked, or configuration may now point at a different external system. Once new-format messages sit in a queue, the old code cannot read them. In those conditions, restarting the old binary is not recovery but another incompatible change.

A deployment tool only executes a decided action. A Kubernetes Deployment revision is managed around the pod template and does not roll back external configuration, databases, or data with it.^[1] A stalled Deployment reports state; whether rollback, pause, or forward fix is right for that change is decided by the operating system above it. So what to write before deploying is not which code changed but which state and contracts this change moves.

Google SRE treats artifact reproducibility and automation, configuration ownership, versioning, and semantic validation, and gradual rollout with the ability to stop, as separate operating conditions.^[6] If configuration follows an external value that can change at runtime, returning to the same version does not reproduce the same state.

What do you lose by having only two deployment states?

If the pipeline has only "running" and "succeeded or failed", the only thing available in between is a rollback. Real operations need at least Prepared, partial exposure, and evaluate/bake, and after evaluation four branches must stay open: expand, pause, roll back, and fix forward.

Expand if the result is good. Pause if evidence is thin or the signal is contaminated. Roll back if compatibility with the earlier state still holds. If data or external state has already changed irreversibly, fixing forward with a corrected version and compensation is more appropriate. A canary is a limited deployment combined with an evaluation procedure; sending less traffic is not enough on its own.^[2]

Pause is the important state here. Automatically redeploying the old version while the signal is uncertain creates a second change unrelated to the cause and complicates the situation. Halting expansion while preserving current state keeps the evidence needed for comparison and diagnosis.

Prepared

Prepared means the artifact, configuration, secrets, schema expansion, routes, and observability signals are ready. The path back to the previous version is verified at this point too.

Partial exposure

Partial exposure means the new version is exposed only to limited replicas, a user cohort, a share of requests, or a feature flag scope.

Evaluate and bake

Evaluate and bake is not waiting for time to pass; it is observing until the candidate-to-control difference, the completion of asynchronous work, and the effects of cache and queues become visible.

The compatibility window is not an artifact retention period

The period in which you can return to the previous version is not simply how long artifacts are kept. It is the period in which the old code can read the new schema and new data, and the new code can still handle the old data and requests that remain.

In practice the window must not be shorter than the longest of seven values: the time needed to decide on a rollback; the longest-running batch or job; the maximum residency of queues and events; the lifetime of clients, sessions, and caches; backfill and consistency verification time; the overlap validity of secrets; and replication lag plus recovery verification time.

This is not any tool's official formula but a model that merges parallel change with gradual rollout into one operating measure.^[3] If old clients or queued messages remain and you remove the compatibility columns and adapters, the deployment is not finished even though all traffic moved to the new version.

Split database changes into expand, migrate, contract

Running an incompatible database change in the same moment as the code deployment closes the rollback window almost immediately. The standard way to avoid that is to split it into expand, migrate, and contract. Say you want to rename customer_name to display_name.

In expand, add the new column without dropping the old one. Build a schema on which both old and new code run, and design the read path so an empty new column raises no error. In migrate, write newly created data to both paths for the needed period, or keep one as the ledger while filling the other asynchronously. Backfill existing data in small units and record position and results; stopping and resuming must not cause double conversion or corruption. In contract, confirm the evidence that the read path moved to the new column and that old versions, batches, and queue consumers no longer use the old column, then drop it in a separate deployment. "The new code deployed fine" and "the old contract can be removed" are different decisions.^[3]

Operating conditions also differ by engine. In PostgreSQL, ALTER TABLE can take a strong lock or trigger a table rewrite depending on the sub-command.^[4][7] Adding a constant default can avoid a rewrite, but the conditions have to be checked. That a migration file runs is not the same as it being safe in production, so check the expected locks, rewrites, runtime, and replication impact. MySQL's atomic DDL is a concept that improves consistency on failure, not transactional DDL a user can undo at will.^[8] Several DDL statements cause implicit commits and are subject to metadata locks, so do not assume DDL can be cancelled the way an application transaction can.

Point-in-time recovery differs from a deployment rollback too. PostgreSQL recovers a whole cluster to a point in time using a base backup and WAL, with configuration files as a separate concern. MySQL likewise combines a full backup with binary log replay. Rewinding a production database affects valid changes made after the target point as well, so extracting the needed data from an isolated restore and reconciling it against current state is often more appropriate.^[9]

Mark irreversible data changes from the start

Seven kinds of change have no complete inverse, or one that costs too much: lossy aggregation that folds several values into one; encryption or re-encoding performed after discarding the original; merging duplicate accounts or records; external payments, emails, messages, and webhook dispatches; events and asynchronous jobs already consumed; encryption keys discarded beyond recovery; and original fields deleted after being stored only in the new format.

For these, the rollback contract's action is not redeploying the old code. It becomes: stop new writes, identify the affected range, deploy a fixed version, run compensating transactions or data repair, and re-verify consistency.

A change whose reversibility is unclear should not be hidden in the deployment note but explicitly classified as forward recovery required. That is what stops people from hunting for a recovery path that does not exist in the middle of an incident.

A feature flag and a kill switch are not the same thing

A feature flag separates binary deployment from user exposure. You can deploy new code while keeping the feature hidden, or open it only to certain users, tenants, or regions. Google SRE also describes feature flags as a way to separate binary release from launch and to selectively disable a problematic feature.^[2]

But not every flag is an operational kill switch. Using one safely takes seven contracts: a safe default for when the flag provider does not respond; the maximum time a change takes to reach every instance; the scope of application by user, tenant, and region; change permissions and an audit record; a path to operate it in an emergency without a separate deployment; a test of the off path under real production conditions; and a removal date with an owner.

OpenFeature standardizes the flag evaluation API, providers, evaluation context, hooks, and provider state events.^[5] It defines that an evaluation error returns the default the application supplied — but which default is safe, and whether a flag change can undo a data side effect, are the application design's responsibility. Once new-format data has been written or an external request sent, turning the flag off leaves that state in place. A feature flag is a control that stops exposure, not a data recovery device.

The roles of rolling, canary, blue-green, and traffic split

The four terms are not mutually exclusive choices. A canary can be built as a separate Deployment with a traffic split, and you can send only part of the traffic to the new environment between blue-green environments while evaluating.

In a Kubernetes rolling update, old and new pods exist simultaneously during replacement.^[1] Google SRE defines a canary as a partial, time-bounded deployment plus an evaluation procedure, and asks that representative traffic, measurement intervals, and the effect of shared dependencies be considered together.^[2] Blue-green makes the route switch fast but does not automatically separate the database and state contracts.

Bake time is not a fixed number of minutes

Bake time is not a rest period fixed as "wait thirty minutes after deployment". It is the span in which representative work actually passes through, enough for this change's main failures to be observable. The required length varies with request volume and the variety of user types; weekday, weekend, day, and night traffic patterns; the longest batch and asynchronous job; queue delay and retry intervals; cache and session lifetime; the call frequency of external dependencies; metric aggregation intervals and lag; and backfill and consistency verification time.

On a low-traffic service, synthetic requests or explicit verification scenarios are more useful than a short canary. Conversely, even at high volume, looking only at an overall average that does not separate candidate from control lets candidate errors hide inside normal traffic.^[2]

A rollback trigger has to be specified across eight fields: the already-defined user or service health signal; whether the scope is candidate, control, or the whole system; the tolerated candidate-to-control difference or an absolute blocking condition; the observation window that failure needs to appear in; the minimum requests, jobs, or user cohort; which action applies among pause, flag off, traffic zero, rollback, and forward fix; the decision maker and the operator; and the manual override's permissions, reason record, and expiry. Which user journeys become which indicators belongs to observability design; here we stop at connecting already-defined health signals to deployment actions.

When automatic rollback becomes the riskier move

Automation's first action is not always to redeploy the previous version. In eight situations an automatic pause or a stop to exposure is safer.

Google SRE handles candidate and control contamination, observation windows, halting a rollout, and human judgement together in canary evaluation.^[2] Kubernetes likewise reports failure to progress as a state without deciding the recovery action for you.^[1] Once database locks, rewrites, and point-in-time recovery scope are taken into account, automating the halt to traffic expansion first is often safer than automating databases, data, secrets, and restore paths outright.

Rollback, forward recovery, failover, and restore are different actions

The four actions change different things. Failover moves where execution happens; restore rebuilds data. If a software defect was deployed everywhere, failover alone does not solve it. If data is uncorrupted, a full restore instead drags valid transactions back into the past.^[10]

So during an incident, ask first what is actually broken, then confirm how far the current compatibility contract still holds, and pick one of the four actions.

Pre-deployment rollback readiness checks

For artifacts and dependencies, look at five things: is the digest and build provenance of the artifact recorded; are you promoting the same verified artifact rather than rebuilding in a development environment; is the previous artifact preserved in a genuinely runnable state; have you confirmed the lockfile, base image, runtime, and key SDK versions; and have you confirmed old-to-new compatibility of external API, message, and event contracts?

For configuration and secrets, look at six: does configuration carry a version and an owner; did you validate meaning — endpoints, ranges, units, mutual exclusion — beyond format checks; do you avoid following an external reference that changes at runtime; is there a secret rotation order and an end-of-validity condition for the old secret; have you confirmed the default behaviour of feature flag and secret lookups on provider failure; and do you refrain from using a leaked or revoked secret as a rollback target?

For database schema and data, look at eight: was an additive expand done before any destructive change; can the old version read and write on the expanded schema; is the migration stoppable, resumable, and re-runnable; are backfill position and consistency check results recorded; if dual writes are used, is there mismatch detection and reconciliation; are irreversible data changes and external effects marked separately; do changes that cannot be rolled back have a forward recovery procedure; and have you confirmed the lock, rewrite, and implicit-commit conditions for your engine and version?

For feature exposure and traffic, look at six: can binary deployment and feature exposure be separated; was the kill switch's off path tested under real production conditions; are there routes and labels that distinguish candidate from control; are there permissions and a procedure to lower traffic weight or set it to zero; were sessions, connection draining, WebSockets, and queue or batch traffic considered; and did you confirm the capacity needed to return traffic to the old environment?

For signals and decisions, look at six: can the pre-defined health signals be read on a candidate basis; does the rollback trigger include an observation window and minimum evidence; are pause, flag off, traffic zero, rollback, and forward fix distinguished; is there a cooldown and a manual override for automated actions; are the stop authority and the operator named; and is there an escalation path for nights, holidays, and absent owners? Finally, for recovery verification: was a rehearsal of actually returning to the previous artifact and configuration performed; was the restore procedure tested separately from the existence of backups; were recovery time and data loss range recorded; is there an owner who can control changes as one during an incident; and is there a place to preserve deployment, verdict, and override records?

Post-deployment compatibility removal checks

Do not remove the old contract the moment all traffic moves to the new version. First confirm that no synchronous traffic goes to the old version, that old clients, sessions, and caches no longer use the old contract, and that queue, event, and batch work in the old format has all been processed. Confirm too that backfill is complete and source-to-target consistency verified, that dual-write mismatches are absent or inside an approved tolerance, that usage of the old columns, APIs, and message schemas is zero, and that no consumer still uses the old secret.

Then comes the cleanup. Note the close of the rollback window in the change record. Revoke old secrets and withdraw permissions as a separate change. Assign an owner and a date for removing feature flags, adapters, and temporary read fallbacks. Drop old columns and tables in a deployment separate from the cutover. After the migration, confirm a new backup or recovery point, then update the runbook, schema documents, dependency records, and operational handover. The removal work itself needs its own rollback and forward recovery plan.

Compatibility code becomes technical debt. But removing it in the same deployment merely because the new version looks healthy closes your most important recovery path yourself. Splitting add, migrate, and remove into different changes is what separates the failure cause and recovery action of each stage.

A deployment you can undo is built before you deploy

The rollback button is the last execution device. For it to be safe, the old code has to handle the new schema. The old environment has to take current traffic and sessions, and the previous state of configuration and secrets has to still be valid. If a data change has no inverse, the rollback option was already gone before anyone pressed the button.

During operations the best first action is sometimes not redeploying the old version but stopping the widening of exposure. Then confirm the current compatibility contract and pick the right action among rollback, forward recovery, failover, and restore.

A server migration where DNS, cache, cron, and database sync all move at once needs the same rollback contract. The concrete preflight and cutover steps belong to a separate migration runbook.