The problem and decision criteria
A test list says, “Uploading a file larger than 10 MB displays an error.” It includes success and failure scenarios and reads naturally. Yet it does not specify the upload, the required error, or whether the file must remain unsaved. The test could pass while missing incorrect behavior.
Review AI-generated test cases first for the evidence behind the requirement, controlled conditions, observable results, and criteria for failure. If the expected result lacks a clear basis, return a clarification question instead of generating more cases. Review what is verified, not just how the sentence reads.
The upload feature and test lists here are design examples. They are not outputs from an actual model or Qoretix, customer cases, or execution results. Rather than comparing generation tools, this article explains how to decide whether to use a case as written, revise it, or clarify the requirement first.
Who determined the expected result?
A test oracle is the basis for deciding whether an actual result is correct. Approved requirements, agreed calculation rules, and verified reference results can serve that role. If the expected result cannot be determined, successful execution does not establish valid verification. ISTQB's Test Analyst guidance also calls for explicit pass/fail criteria and consideration of data and post-execution state, not only screen output. [1]
During review, write a short answer beside the expected result: why is this correct? Evidence such as inclusive upper bound in requirement R-UP-01 v1.0 allows progress. If the only answer is that the current code behaves that way or AI called it a common policy, business correctness remains unconfirmed.
This does not exclude code-based generation. Tests documenting current behavior can reveal changes. But distinguish tests preserving current behavior from tests verifying agreed requirements. Do not promote a known defect in current behavior into an approved expectation.
Suppose the agreed rule is file size ≤ limit, but the implementation uses file size < limit. Reading that implementation and expecting a file exactly at the limit to be rejected preserves the bug. Calling the same production function to calculate the expected result cannot independently reveal it either. Derive expectations from boundary values calculated using the approved rule.
Evidence connected from requirement to approval — Link requirements, tests, defect actions and approval records so teams judge quality consistently.
Record ambiguous requirements as questions, not tests
“Files up to 10 MB can be uploaded” is insufficient for an exact boundary test. Establish whether 10 MB means 10,000,000 bytes or 10 MiB, 10,485,760 bytes. Whether the limit is inclusive and applies to the file body or the complete HTTP request are separate decisions. Empty files, upload permissions, and data handling after failure also require policies.
To prevent a generator from filling gaps arbitrarily, separate requirement review into confirmed rules, assumptions awaiting approval, and unresolved questions. For example:
Requirement clarification record: design example
QuestionQ-UP-01: What is the size unit, and is the upper limit inclusive?
Decision owners: product owner and API contract owner.
Affected cases: uploads just below, exactly at, and just above the limit.
Temporary assumption: allow up to 10 MiB inclusive. Do not use as acceptance criteria before approval.
Next action: finalize the rule and applicable version, then update expected results.
A question does not require stopping all related work. Data generation and isolated environment setup that do not depend on the unit can continue. But do not pass cases dependent on an unresolved rule. Requirement ambiguity and a product defect are different states.
For the structure linking requirement and test versions, see Requirements-to-test traceability. Here the focus is the judgment within one case found through that linkage.
Turn a weak case into one that can actually decide correctness
Assuming those questions have been answered, consider the following fictional API contract. These rules and response codes are choices for the example, not universal upload-service standards.
Agreed rule R-UP-01 v1.0 in the design example
A user with edit permission can upload into a prepared folder. The file body must be at least 1 byte and no more than 10 MiB. Use an ASCII .txt file with valid metadata, eliminating other rejection reasons such as file type, user storage quota, and storage failure.
A successful upload completes storage, then returns HTTP 201, a nonempty fileId, sizeBytes equal to the actual size, and status READY. Exactly one metadata record and one stored object must exist, and downloaded bytes must match the source.
An oversized upload returns HTTP 413 and code FILE_TOO_LARGE, without issuing fileId. No file metadata or persistent object for the request remains. This is a synchronous contract with no asynchronous job that stores the file after the response; storage queries in the test must be able to observe completed writes.
The original weak case reads:
Before revision: design example
Input: a large file.
Action: upload it.
Expected result: an error displays correctly.
Revised to verify the server-side size limit and storage outcome, it becomes the following. This is a test specification, not an execution result.
| Element | Revised case TC-UP-MAX-PLUS-ONE: design example |
|---|---|
| Requirement | R-UP-01 v1.0: reject an oversized file with the specified error and retain no file data |
| Preconditions | Edit-enabled test account, prepared folder, and case-specific data area; initially zero metadata records and stored objects there. Confirm intermediary proxies allow the request size including file body and overhead, and that the test route reaches the API responsible for checking the size rule. |
| Input | limit-plus-one.txt: 10,485,761 bytes of ASCII a, without BOM or extra newline. Verify generated file bytes, not filename or string length. |
| Action | Send the file once to the real upload API in an owned or authorized test environment. Do not mock the target API response or file storage path for this verification. |
| Expected result | 413, code = FILE_TOO_LARGE, and no fileId. After synchronous processing ends, zero metadata records and persistent objects for this request. An error for another reason is not a pass. |
| Evidence | Input byte count, execution identifier, request and response, and metadata/object queries before and after execution. Do not turn query failures or authorization errors into zero counts. Exclude personal information and credentials. |
| Cleanup and isolation | Clean case-specific data after checking evidence. Restore residual data and changed test settings even on failure. Record cleanup failures separately; do not assume the next run starts clean. |
The difference is not sentence length. Incorrect results now have specific detection points: accepting the file fails the status assertion, rejecting it for the wrong reason fails the error-code assertion, and returning rejection while storing the file fails the storage assertion.
Do not trust a zero-object query alone. First confirm that the same observation mechanism can see one object from a successful upload. Querying the wrong location or lacking permissions can manufacture false absence evidence. Also separate result verification from cleanup so cleanup cannot delete the evidence first.
Adding boundary values must change both input and judgment
ISTQB CTFL describes equivalence partitioning for input groups processed similarly and boundary value analysis at ordered partition boundaries. State transition testing applies when behavior depends on state. Review whether a generated list merely varies ordinary values or actually verifies boundaries and state changes. [2]
The upper boundary of this contract is shown below. Each row is an independent run with the same initial state.
| Input file size | Expected response | Additional checks |
|---|---|---|
| 10,485,759 bytes: one below the limit | 201 | Matching sizeBytes, READY, one metadata record and one object, matching downloaded bytes |
| 10,485,760 bytes: exactly the limit | 201 | Same checks; exposes an implementation that incorrectly excludes the limit |
| 10,485,761 bytes: one above the limit | 413, FILE_TOO_LARGE | No fileId, zero metadata records and persistent objects |
This table does not complete upload verification. It omits the lower bound, empty files, permissions, interrupted transfers, and storage failure. Do not call an upper-bound example complete boundary or feature coverage.
Defects to look for in a generated list
This checklist is not a format-compliance scorecard. It is an editorial and review proposal for deciding what to fix when a problem is found. High scores elsewhere cannot compensate for unresolved requirements or expectations.
| Problem | Review question and improvement | Next action |
|---|---|---|
| No basis for expected result | Which requirement, contract, or reference result supports it? Create a question if source and version cannot be linked. | Clarify requirement |
| Repeated happy paths | Are valid/invalid inputs, upper/lower bounds, rejection, interruption, and recovery needed? For stateful features, check initial state, action, next state, and forbidden transitions. | Revise before use |
| Weak assertions | Does it check only that a response exists or a page opens? Specify the values, states, and side effects needed for the purpose; do not force all layers into one unit test. | Revise before use |
| Existing bug copied as truth | Does the expected result come from current implementation or its calculation function rather than an approved rule? Separate characterization from requirement verification. | Clarify or revise |
| Mock hides the target | Is the purpose the error-handling UI or actual server error detection? Do not claim verification beyond the replaced path. | Revise purpose or split verification |
| Semantically duplicate cases | Are requirement, initial conditions, input class, and expected result the same? Parameterize or merge common purposes while preserving different judgments such as accepting the limit and rejecting above it. | Discard or merge |
| Missing authorization or failure paths | Is only login checked? Define permitted/forbidden actions and data state after failure. | Clarify and supplement |
| Data does not meet conditions | Do byte size, encoding, time zone, role, and existing data match the specification? Separate cases rejected first by unrelated validation. | Revise before use |
| Condition-dependent instability | Does it depend on time, randomness, external responses, shared data, or order? Control conditions and record observation methods and limits for uncontrolled parts. | Revise or defer execution |
A hidden button does not prove server authorization. OWASP WSTG v4.2 WSTG-ATHZ-02 distinguishes feature and resource access by role and identity. Here, review the purpose and judgment evidence of each case; detailed multi-tenant combinations are a separate scope. Security execution must use authorized environments and synthetic accounts and data. [3]
Where a mock substitutes matters more than how many mocks exist
Playwright's official mock example returns a predefined network response and explicitly states that the real API is not called. It can verify UI behavior for that response, but does not prove that a server checked file size or prevented storage. [4]
Mocks can appropriately reproduce external timeouts or stabilize a screen state. Verifying the server size limit and storage result in this example requires separate execution of those real paths. Record replaced dependencies and actually executed paths to avoid either discarding all mocked tests or overstating them as complete integration checks.
Verify the paths beyond the happy path — Reconstruct requirements from behavior and exercise exceptions in input, permissions and external calls.
Separate human judgment from tool-based checks
People need not rewrite every case. Filter repetitive structural, reference, and code errors with tools first, leaving reviewers time for whether the expectation is correct and the observation method is sufficient.
Static rules can find missing fields, nonexistent references, type or syntax errors, unintended skip markers, and missing asynchronous waits. But having an assertion is different from asserting the right requirement. Automatic semantic grouping can find merge candidates; consolidate only after confirming they verify the same thing.
Execution checks establish whether the test is discovered and starts, setup and cleanup work, and the intended path and assertions are reached. Repetition or reordered runs can uncover dependencies, but several passes do not prove the absence of nondeterminism.
Also distinguish tool safeguards by scope. Fixtures prepare environments and resources, but their scope matters. Playwright Test's default page and context are test-scoped, while worker-scoped fixtures also exist. Separate browser contexts do not automatically reset external databases or file storage. Design shared-account and shared-data isolation separately. [5][6]
Wait for the intended condition rather than only sleeping a fixed duration. Playwright Test's web assertions, such as await expect(locator).toHaveText(...), repeatedly check conditions. Not every expect automatically retries, and automatic waiting cannot correct a wrong expected result. [7]
Verify that the test fails when behavior is wrong
For important cases, add a small falsification check. In an isolated local copy of this example, temporarily removing the size check should make the oversized case fail. An incorrect boundary comparison should make the exactly-at-limit success case fail. Revert the experiment afterward; do not ship it in production code.
This supplementary check establishes detection of the intended fault. Catching an injected error does not prove detection of every other defect. An experiment that failed to alter the target or judgment path should not count as a falsification check.
NIST SSDF v1.1 PW.8.2 also connects security test scope, design, execution, result recording, and finding triage. Drawing on that guidance, this article proposes separate records for generated-case review, execution results, and follow-up. This is not AI test-generation certification or complete SSDF compliance. [8]
To manage failures or instability after execution, use the Automation failure analysis, quarantine, and reinstatement log. Design review and post-execution failure analysis are different judgments.
End the review with one of four next actions
These are test design review outcomes. Do not confuse them with execution PASS/FAIL or release approval.
| Review outcome | Decision condition | Next action |
|---|---|---|
| Use as written | Requirement basis, inputs, judgment, evidence, and isolation fit the purpose without revision | Finalize the reviewed version and validate execution in the agreed environment; keep execution status unrun if not yet executed |
| Revise before use | Requirement is settled, but data, assertions, mocks, isolation, or other details need changes | Owner revises; review changed parts before execution |
| Clarify requirement | No basis for deciding correctness, or documents, code, and policies conflict | Record question, decision owner, and affected cases; defer judgments requiring the answer |
| Discard or merge | Duplicate verification, no purpose, or adequate replacement by another case | Record the retained case and rationale; do not discard merely because execution is difficult or reveals a defect |
People should first review cases introducing new expectations or risky data changes. Shared specifications and automated checks can reduce repeated work for inputs governed by already reviewed rules. Do not batch-approve unresolved requirements or unverified authorization conditions to save review time.
This workflow can begin with an existing repository and issue tracker. If someone owns requirement decisions, data is controlled, and outcomes are observable, a new generator is not a prerequisite. If teams use conflicting expectations or cannot observe stored state after failure, organize contracts and observation before increasing generation volume.
In the next review, choose one case and ask: “If we introduce a specific wrong behavior, which check in this case should fail?” A clear answer determines the verification target. Then supply data, environment, and execution conditions. An unclear answer calls for better evidence rather than more prose.
If several teams need to agree on review criteria and roles, review IXC's QA process consulting scope.
