Automated test failure analysis, quarantine and return-to-service worksheets

A successful retry does not remove the reason for the first failure. Execution order or data may have changed, or the first attempt may have left state that changes the result. Connect the first attempt and retries under one incident identifier, with the product and test revisions, environment, starting data and expected result. Even when the product version is unchanged, a different account or reset data is evidence worth recording.

Separate execution results, investigation progress and asset status. PASS, FAIL, BLOCKED and NOT RUN describe scenario execution; received, investigating, actioned and reviewed describe the incident's progress. Quarantine is a separate decision about where an automated test has been removed from execution. Interpret tool-specific states as defined by the tool. Playwright's retry documentation distinguishes tests passing on the first attempt from those passing after a retry. Neither state translates directly into a product defect count or release decision.

Start by comparing the agreed requirement, the test's expectation and the product's observed behavior. If the requirement is correct and the product differs, investigate a product defect. If the product meets the requirement but the expected value, locator or observation timing is wrong, the test needs correction. If shared data was overwritten or execution environments differed, control that condition and check again. A timeout at the end of a log does not establish an environmental cause.

The investigation worksheet records the incident, hypothesis, supporting or excluding evidence, next investigation, owner and review date. Changing the product, test and data together makes it difficult to identify what addressed the cause. Where practical, vary conditions separately and connect the before-and-after results. Keep the cause unresolved when evidence is insufficient. Playwright's guidance on isolation and data control helps with reviewing these preconditions.

The completed examples come from a local Node.js synthetic fixture executed by IXC for this material. Small functions and a controlled data sequence deliberately contain known causes. The fixture did not test customer software, a browser or operational CI, and it did not measure real-world failure frequency.

In F-PRODUCT, product logic incorrectly allowed a viewer to edit. The same input and expected result reproduced the failure before the product's role condition was corrected. In F-DATA, setup operations A and B used the same document key, so B overwrote A's owner. A retry with clean data passed, but repeating A-write, B-write, A-read failed again. The corrected fixture used a separate key per run and checked the same sequence. This is a controlled interleaving example, not validation of a concurrent scheduler.

In F-TEST, the requirement counted active records while the test expected the total record count. Comparing the requirement with two active records and one inactive record justified correcting only the expected value, leaving the product unchanged. Each corrected case passed three subsequent checks. That number describes this local example, not a stability threshold for every test asset. A real record must also identify the corrected code or data preparation rule and explain any change to the expectation.

Record the quarantined test and incident, the execution stage it leaves, the reason, release impact and next review date. If verification temporarily becomes manual, agree on who performs it, with which data and when. If another test layer substitutes for it, record which part of the original scenario it covers. Without a substitute, report the scope as unverified to the release owner. Do not count quarantined tests as successful verification of work that did not run.

For example, independent data can support a check of core behavior while a shared-data collision is investigated. That result does not establish stability of the entire parallel run. Preserve the substitute execution identifier and remaining conditions. A target based only on reducing the number of quarantined tests could be met by deleting tests, so review elapsed time, substitute verification and unresolved risk together. If a test leaves automation permanently, decide whether a person will continue that verification.

The return review connects evidence of cause, the corrective change, revalidation conditions, results, remaining limitations and reviewer. If the product changed, check whether the test remained the same. If the test changed, check that it still verifies the requirement. Merely changing an expected value to match the observed output can erase the signal while leaving a passing status. The F-TEST correction is justified because the agreed requirement counts active records, not simply because the product returned two.

Agree on repetition counts and environments based on product risk and variation in data and execution. Repeating a test without the original failure-inducing sequence or conditions is insufficient evidence. Hand over the owner and record location for a recurrence after restoration. Returning an automated test to use and approving a release are separate decisions; the release meeting receives what actually ran on that build and what remains unverified.

For periodic review, distinguish initial failures, retries, investigation time, script maintenance and environment or data preparation time. Align the observation period so changes in execution volume and maintenance work can be read together. An increase in the pass rate after retries does not establish better product quality or lower cost. The test automation portfolio addresses which tests remain in automation; use the release decision memo and scenario result sheet when passing verified results into a release decision.

The downloadable PDF and editable Excel worksheets are in English. Use the same incident identifier across the initial record, investigation, quarantine and substitute verification, return review and maintenance work. Representative logs, the environment and data preparation method, and the current maintenance owner make it possible to discuss a concrete investigation, correction and handover scope with IXC's test automation service.