The step after "did the tests pass"
Test results alone must not settle a release. Tests show facts observed on a specific build, environment, data set, and scenario. They do not judge whether that scope covered the actual change, whether production will behave the same, or what a failure does to users and data. Google SRE makes the same point: test and production environments are never identical and not every scenario can be exercised, so some defects only surface under real traffic.[4][6]
What the release meeting decides is the step after "did the tests pass". It has to establish what was verified and what is still uncertain, who is harmed and how if something goes wrong, and whether the team can notice the problem and stop exposure or recover. Deciding, on that evidence, whether the organization accepts the remaining risk is the release decision.
The verdict is one of READY, READY WITH CONDITIONS, or HOLD. Scores across items must not be summed. Some conditions — possible user data loss, an unverified critical path, no recovery method — cannot be offset by other results. Release readiness reads blockers and evidence gaps first, not an average.
Test results only count as evidence with their context
"The full suite passed" is not enough to judge on. At minimum the following context has to travel with it: which build, what was verified, which environment and data were used, which items are NOT RUN, BLOCKED, or excluded, and when the results were generated.[1]
Even if everything is PASS, the results mean something different if configuration changed afterwards. If a critical user path was out of scope, or external payment, authentication, and messaging systems were only exercised while healthy, "passed" covers much less than it sounds. No FAIL and having checked the conditions that could fail are not the same statement.
Evidence connected from requirement to approval — Link requirements, tests, defect actions and approval records so teams judge quality consistently.
Release Evidence Matrix
Beyond the test report, a release decision needs evidence on requirements, security and data, the production environment, user impact, recoverability, and accountability.
Functional test results cannot stand in for security readiness either. NIST SSDF defines high-level practice areas for integrating security into the SDLC, and OWASP ASVS provides verifiable requirements for technical security controls.[7][8]
The Matrix below does not compute a quality score. It checks whether the evidence needed for a judgment exists, which gaps block the release, and which risks may be accepted conditionally. Not every cell needs a long entry. What is not allowed is offsetting a blocking gap with good results in another row.
Evidence area | Question to ask | Evidence required | Blocking condition | Conditional acceptance | Owner |
|---|---|---|---|---|---|
| Requirements and change scope | Does the actual change match what this release set out to solve? | Approved release scope, change summary, acceptance criteria, exclusions | The team has not agreed on core requirements or change scope | Non-core items are explicitly excluded with a follow-up owner and deadline recorded | Product Owner |
| Test results and limits | Were the critical paths verified on a suitable environment and data? | Target build, environment, results, NOT RUN / BLOCKED / excluded items, evidence timestamp | A critical path is NOT RUN or BLOCKED, or results are not tied to the current build | The gap is limited in impact, and exposure scope, observation method, and follow-up verification are set | QA Lead / Engineering Lead |
| Open defects and residual risk | If a remaining defect fires, who is harmed and how? | Open defect list, impact analysis, reproduction conditions, workarounds, uncertainty | Harm to data, money, permissions, legal obligations, or hard-to-recover damage is uncontrolled | Impact is bounded and the response, owner, and expiry are recorded | Product / Engineering |
| Security, privacy, data | Are there unverified risks in new permissions, sensitive data handling, or data changes? | Security review, applied requirements and versions, data-flow check, open security issues | Unauthorized access, disclosure, or data corruption is unresolved, or the approver has not reviewed the risk | The gap is non-core, exposure is limited, and compensating controls and a review date are set | Security / Privacy Owner |
| Production environment and dependencies | Are production config, capacity, credentials, and external services ready under real conditions? | Config confirmation, dependency status, constraints, support contacts, on-call owner | A core dependency or production setting is unclear and nobody is on point for an incident | Exposure is limited to specific users, hours, or features and an owner can respond immediately | Operations / Service Owner |
| User-impact observation | Will the team notice a serious user failure after release? | User impact to watch, screen or signal, observation window, stop trigger | Users could be seriously harmed and the team has no way to notice | A person can check directly within a limited scope and a re-decision precedes expansion | Service Owner / Reliability Owner |
| Recoverability | If something breaks, can exposure be stopped and a known state restored or safely fixed? | Verified stop and recovery path, execution permission, owner, completion check | An irreversible high-impact change, or the recovery path was never verified | Impact is bounded and a verified alternative response or fix-forward path exists | Release / Operations Owner |
| Accountability and approval record | Who decides, who accepts the residual risk, and who acts when something goes wrong? | Release Decision Record, residual risk owner, approval authority, review date | No decision-maker or response owner, and the risk-accepting party is unclear | Conditional release only when condition, owner, approver, and expiry are all recorded | Release Authority |
How READY, READY WITH CONDITIONS, and HOLD are separated
READY means the evidence required for the current release scope is in hand and no unacceptable residual risk was found. READY does not claim "no defects" or "no incidents". Normal operational risk remains. What must exist is evidence that the important uncertainties are identified, sit inside the organization's accepted range, and can be observed and handled if they materialize.
READY WITH CONDITIONS means there is no absolute blocker, but an unverified condition or remaining risk exists. All of the following must be settled: who or which features are exposed, what is observed, under what circumstances exposure stops, how the team responds, who executes that decision, and when READY or HOLD is re-judged. "Ship and keep an eye on it" is not READY WITH CONDITIONS. A conditional release without scope, stop condition, owner, and expiry is an unrecorded READY.
HOLD is the reasonable verdict when any of the following holds. Core change scope or requirements are unclear. A critical user path was not verified. Hard-to-recover harm to data, money, permissions, or personal information is possible. The impact of remaining uncertainty cannot be bounded. There is no way to notice user harm. There is no evidence that exposure can be stopped or recovered. Nobody is authorized to accept the risk and nobody owns the response.
Recoverability is not an appendix for operational convenience after launch. Google SRE describes gradual rollout, the ability to pause, and the ability to roll back as core properties of safe change, and notes that an unrecoverable deployment prolongs user harm.[6][9] How to avoid designing irreversible changes is also a theme of Release It!.[5]
The decision runs in five steps. First, if a non-offsettable blocker exists, HOLD. Second, if the user, data, and business impact of the remaining uncertainty cannot be explained, HOLD. Third, if exposure cannot be limited, observed, stopped, and recovered, HOLD. Fourth, if the conditions, owner, stop trigger, and re-decision point are recorded, READY WITH CONDITIONS. Fifth, if the evidence required for the current scope is sufficient and no unacceptable residual risk remains, READY.
- READY
- READY means the evidence required for the current release scope is in hand and no unacceptable residual risk was found. It does not mean zero defects — it means remaining risk sits inside the accepted range.
- READY WITH CONDITIONS
- READY WITH CONDITIONS means there is no absolute blocker but an unverified condition or remaining risk exists. It only stands when exposure scope, observation signal, stop trigger, response, owner, and re-decision point are all recorded.
- HOLD
- HOLD is the verdict to withhold release because at least one non-offsettable blocker exists — an unverified critical path, hard-to-recover harm, no recovery evidence, or no accountable owner.
Residual risk must be recorded differently from a defect list
NIST defines residual risk as the risk that remains after controls or responses are applied. Acceptable risk is residual risk within the organization's risk tolerance. The definition comes from security risk management, but the distinction is just as useful for release decisions.[10]
A defect list records "what is wrong". A residual risk record has to state "under what conditions, to whom, what could happen, and why we accept it now".
Compressed labels such as "medium risk", "Severity 3", or "fix later" leave the next person unable to reproduce the judgment. The following sentence shape keeps the conditions and the accountability of a risk together.
Because of [cause or unverified condition], [failure event] may occur in [affected area], causing [user, data, or business impact]. The confirmed evidence and uncertainty are [evidence]. On release, [exposure limit or mitigation] is applied, and when [stop trigger] is observed in [signal], [response or recovery] is executed. The owner is [role], the risk acceptance authority is [role], and the review point is [date or event].
Severity and likelihood may be recorded, but they must not compute the release verdict on their own. Risks with the same label reach different conclusions depending on data recoverability, user scope, detectability, and whether an owner exists. The Register fields below are the minimum set for keeping that difference on record.
Field | What to record |
|---|---|
| Risk ID | A stable identifier |
| Risk scenario | Cause, failure event, and consequence in one sentence |
| Affected parties | User types, data, business processes, external dependencies |
| Confirmed evidence | What is already known from tests, reviews, and operational checks |
| Remaining uncertainty | Conditions not yet verified |
| Impact and reversibility | Not only the size of harm but whether it can be undone |
| Release conditions | Exposure limits by user, feature, time, or region |
| Observation and stop trigger | What to watch and when to stop |
| Response or recovery | Actions on trigger and how completion is confirmed |
| Owner | The role that watches and acts |
| Acceptance Authority | The role authorized to accept the residual risk |
| Expiry / Review | When conditional approval ends or is re-judged |
What a Release Decision Record must keep
A Release Decision Record is not a document for collecting signatures. It records which evidence was read and which uncertainty was accepted, so that the judgment can be reviewed later.
A longer record is not a better one. It only needs to make the evidence findable, expose the assumptions and accountability behind the judgment, and state when a conditional decision ends. Of the sixteen fields below, Decision, Rationale, and Expiry / Reassessment cannot be left blank.
Field | What to record |
|---|---|
| Release ID | Product or service name, version, or change identifier |
| Decision | READY / READY WITH CONDITIONS / HOLD |
| Decision Time | When decided and when it takes effect |
| Release Scope | Included features, excluded features, target users |
| Affected Assets | Users, data, permissions, external systems |
| Evidence References | Source documents per Matrix area and their timestamps |
| Known Unknowns | Unverified conditions and current assumptions |
| Hard Blocker Check | Whether a blocker exists and the reasoning |
| Residual Risks | Risk IDs and the conditional acceptance |
| Release Conditions | Exposure limits, observation window, expansion criteria |
| Stop Trigger | The event that starts a stop or re-decision |
| Recovery Evidence | Reference to the verified stop and recovery path |
| Owners | Roles for observation, response, recovery |
| Acceptance Authority | Final decision-maker and risk acceptor |
| Expiry / Reassessment | When a conditional decision expires or is reviewed |
| Rationale | Why this verdict rather than another |
Hypothetical scenario: releasing order status notifications
Suppose a B2B order management service adds order status notifications through an external messaging provider.
Functional and regression tests are all PASS. The security and data review is complete: only approved contacts are used and no new sensitive data is stored separately. If a notification fails to send, the order status itself is still saved correctly.
What was not sufficiently verified is how the messaging provider's rate limits behave at real peak hours. If they bite, some users receive notifications late or not at all. That does not lose order data, but it does delay work.
The team can expose the feature to a limited customer group first and can observe message processing delays, delivery failures, and mismatches between order status and notification status. The evidence that exposure can be stopped once a problem is confirmed has been verified. The operations owner and the re-decision point are set for after the first normal peak window.
The verdict is READY WITH CONDITIONS. The rationale has four lines. The accuracy and storage path of core order data are verified. The provider's real peak-limit behavior is still uncertain. The remaining risk is confined to delayed or missing notifications and does not affect order processing. Exposure limiting, observation, stopping, and owner assignment are all possible.
The release conditions also have four lines. Expose only to the selected customer group. The operations owner monitors processing delay and delivery failure. Stop expansion when order status and notification status disagree. Re-review the evidence after the first normal peak window. The owners are the Service Owner and the Operations Owner, and reassessment happens when the first peak-window review completes.
It is not READY because an important dependency remains unverified under real production conditions. It is not HOLD because core order data is not damaged, impact can be bounded, and the problem can be observed and exposure stopped. If that difference cannot be explained, a READY WITH CONDITIONS verdict must not be issued.
Common mistakes in release decisions
Five mistakes recur in release decisions. All share one cause: replacing a look at the kind of evidence and its reversibility with counts, labels, the presence of a document, or an average score.
- Treating test count or pass rate as overall quality
- Many tests are weak evidence if they only repeat items unrelated to the change. What matters first is whether the users, data, and dependencies that this change touches were checked.
- Reading defect severity without user impact
- The same severity label can cover a cosmetic error on a rarely seen screen and a monetary or data mismatch. Trigger conditions, impact scope, and reversibility must be recorded together.
- Assuming recoverability because a recovery document exists
- A document existing and a procedure actually working on the current build, config, and data are different things. The release gate checks verified evidence, not the presence of a document.
- Leaving a conditional release open indefinitely
- A conditional decision without an owner and expiry just pushes the risk to the next meeting. Convert to READY when conditions are met, or re-judge as HOLD or tighter limits when they are not.
- Hiding a blocker behind an aggregate score
- Missing security evidence or an irreversible data change must not be outvoted by high scores elsewhere. The release gate reads non-offsettable conditions before any average.
Release is not "are there no defects" but "what do we know and accept"
Good test results are very important evidence in a release decision. But test results never become the verdict themselves.
A releasable state is not one where every risk has vanished. It is one where the facts needed for the current scope are sufficiently verified, the remaining uncertainty and user impact can be explained, a problem can be observed and stopped or recovered, and the person accepting that risk is clear.
So the last question of the release meeting is not "how many tests passed". It asks what was verified, what is still unknown, and who accepts the consequences of shipping in that state. Only when that question is answered with evidence and a record can READY, READY WITH CONDITIONS, and HOLD be told apart.[2][3]
