Automation priority is not determined by scriptability
Test automation priorities should not depend only on whether a script can be written. Automate first the checks that run frequently, have substantial user or business impact if they fail, have results a machine can judge clearly, and allow environments and data to be reproduced repeatedly. The value of fast feedback must also exceed implementation, maintenance, and failure-analysis costs.
Conversely, retain human judgment when interpreting meaning, usability, or the context of exceptions is the essence of verification. Even if automation detects a rule violation or a visual difference, automation + human review is appropriate when a person must decide whether users actually face a problem. If unstable environments and data make automation results untrustworthy, classify the test as unready for automation, not unimportant. ISTQB likewise considers repeat-execution value, ROI, repeatability, maintainability, and frequently used business flows when selecting candidates.[6]
This article does not decide which requirements to test, which regressions belong in the current change scope, or whether test results justify release. It addresses only whether to automate a check or assign it to people after its purpose and target have been defined.
Manage a test portfolio, not an automation rate
Automated tests are code too. They cost money to build, require changes as products and environments change, and demand investigation to distinguish product defects from problems in test code, data, or environments. Automation runs more tests quickly and consistently and brings feedback forward, but it also brings initial investment, ongoing maintenance, and rigidity in the face of change.[7]
A useful investment relationship is: repeated feedback value + risk-reduction value > implementation cost + environment and data cost + maintenance cost + failure-analysis cost. This does not require converting every item into a number. It is closer to a ledger for identifying overlooked costs and benefits than a target for increasing test counts.
For example, automating a manual test that takes 30 minutes each time may offer little value if it checks a feature used once and then discarded. Conversely, scenarios such as authorization bypass or data corruption must not be deferred on frequency alone even if they are rare. Risk combines the likelihood of an event with its impact.[8]
Maintenance that keeps results trustworthy — Run with stable selectors, classify failures, repair their causes and verify the result again.
There is no reason to start with GUI automation
Treating test automation as replaying screen clicks can lead teams to accumulate expensive UI tests first. Automation also applies to systems without a UI, network connections, components, APIs, and integration points. Contract testing is a form of integration testing that checks whether different services honor agreed interactions and data rules.[7]
Choose the automation layer where the same risk can be observed most quickly and reliably, rather than fitting a fashionable diagram.[1] Check monetary calculations and state transitions directly at component level. Check authentication, authorization, and data invariants through API responses and stored state. Check compatibility between services with contract or integration tests. Keep UI journeys for risks that emerge only when users move through several actual screens. Combine machine-collected evidence with human judgment for visual differences, accessibility meaning, and usability.
A user journey does not have to be verified by one enormous UI script. Calculations, permissions, contracts, and data state can be checked at more stable points, leaving the UI to verify actual connections and essential behavior.
Twelve dimensions for prioritizing automation
The table is an IXC Insights portfolio framework assembled from ISTQB criteria for automation suitability, cost, environments, and oracles, together with W3C accessibility evaluation principles. No single official document presents these exact twelve dimensions; the framework combines several sources for use in one decision.[2][6][9]
No individual dimension decides whether to automate. High business risk raises testing priority, but weak oracles and unstable environments may make mixed verification or testability improvements more appropriate before full automation.
Dimension | First question | Signals favoring automation | Signals favoring mixed, human-led, or deferred testing |
|---|---|---|---|
| Execution Frequency | What triggers this check, and how often does it recur? | Runs on every commit, PR, build, or deployment, or across many environments and combinations | One-off checks; low frequency alone does not exclude high-risk cases |
| Business / User Risk | What harm would failure cause users and the business? | High impact, including authorization, payments, data corruption, or interruption of critical work | Limited impact or easy detection and recovery; high risk does not itself create an automatic oracle |
| Deterministic Oracle | Can rules, states, or tolerances determine the expected result? | Objective values, state transitions, invariants, schemas, and response codes | Contextual judgments such as looking good, feeling convenient, or being appropriate to the situation |
| Environment Stability | Can versions, time, networks, and external dependencies be controlled? | Reproducible environments and dependencies controlled through mocks, stubs, or test interfaces | Unreliable external sandboxes or uncontrolled time, devices, and networks |
| Test Data Cost | What does creating, isolating, and resetting data cost? | Seeds and resets are available; independent accounts and data can be created repeatedly | Manual approval, paid external data, personal information, or shared state that is hard to restore |
| Maintenance Cost | How often do product changes break the automation code? | Stable observation points, reusable common behavior, and a clear owner | Frequent UI, contract, or environment changes; no maintenance owner; excessive coupling to implementation details |
| Speed / Feedback Value | How soon must a result arrive to change the next action? | Results immediately inform merging, fixes, or further investigation after a change | Results arrive after the decision, or failure analysis takes longer than execution |
| Human Judgment Requirement | Does a pass decision need interpretation, policy, context, or consultation? | Clear rules and structured exceptions | User meaning, regulatory interpretation, UX, or complex exceptions require judgment |
| Exploratory Value | Is the goal to confirm known results or discover new problems? | Known risks and repeatable expected results | New features, incomplete requirements, or unexpected interactions are the focus |
| Accessibility / Visual Judgment | Can rule violations be separated from actual usability? | Machine-checkable attributes, contrast, structure, or image differences | Meaningful alternative text, natural keyboard flow, information hierarchy, or actual usability |
| Rare but High-risk Scenario | Is the scenario rare but costly when it fails? | Stable invariants, preconditions, and recovery results can be checked automatically | Coordination, recovery decisions, and emergency communication depend on practice and human decisions |
| Flakiness Risk | Do identical conditions produce identical results? | Stable repeated results and reproducible, diagnosable failures | Race conditions, shared data, networks, and timing dependencies make results vary |
Classify through three gates: value, trust, and responsibility
There is no need to score all twelve dimensions precisely. In practice, apply three gates in order: Is repetition valuable? Can the result be trusted? Who will maintain and interpret it?
The results connect as follows. High value, reliable results, and little need for human judgment favor automation first. High value with some interpretation required favors automation + human review. When exploration, usability, or contextual judgment is the essence of verification, use human-led testing. When a valuable check cannot currently be trusted because of its environment, data, maintenance, or flakiness, do not automate it yet.
- Gate 1. Is repetition valuable?
- Consider execution frequency, user and business risk, and the value of fast feedback. Frequent repetition or a need for immediate results increases automation value. Even infrequent checks proceed to the next gate when potential harm is high. Do not exclude them on frequency alone.
- Gate 2. Can the result be trusted?
- Check the automatic oracle, environment stability, test data, and flakiness. Fast execution is not useful when no machine-checkable expected result exists or identical conditions keep producing different outcomes. First improve requirements, oracles, environments, and data, or isolate the check.
- Gate 3. Who will maintain and interpret it?
- Compare implementation and maintenance costs with the need for human judgment and exploration. Keep tests human-led when interpreting meaning or discovering new problems is their main value. Machines can prepare repetitive work and collect evidence while people reach conclusions.
Test Portfolio Matrix
The table sets out entry conditions, execution methods, typical examples, and next management actions for the four portfolios. Human-led testing and not automating yet are distinct categories.
In human-led testing, people's observation and interpretation create quality. Exploratory testing, usability reviews, and clarification of ambiguous expectations belong here.[5] Do not regard this as failed automation.
A deferred test may have an objective result, but its environment, data, or maintenance structure is not ready. Improving testability allows it to move into another portfolio later.
Portfolio | Entry conditions | Execution | Typical examples | Next management action |
|---|---|---|---|---|
| Automation first | High repeat value and risk, stable oracle, environment and data, and manageable maintenance | Pipeline or scheduled runs with machine PASS/FAIL decisions | Unit tests for calculations and state transitions; API authentication and authorization; data invariants; contract compatibility; controlled integration scenarios; machine-measurable performance conditions | Keep execution fast, assign an owner, and track failure causes and maintenance time |
| Automation + human review | Machines can collect some evidence, but meaning, context, or exceptions still need judgment | Automatic execution, analysis, and diffs followed by human review of results and residual risk | Critical user journeys; automated accessibility checks with manual usability assessment; visual regression diffs with design review; recovery simulations with human-led drills | Define the boundary between machine and human decisions and retain review records |
| Human-led | Exploration, learning, usability, policy, or business context is the purpose of the check | Exploratory testing and expert review with a charter, timebox, and records | Exploring new features; edge cases in ambiguous requirements; user acceptance; qualitative UX; complex business exceptions; observation of real users | Record objectives, scope, findings, and follow-up tests instead of leaving testing unstructured |
| Do not automate yet | Oracle, environment, or data is unready, or maintenance and failure analysis cost more than repeated feedback is worth | Manual compensating controls and a testability backlog | Unstable external sandboxes; data requiring manual approval each time; shared state that cannot be reset; unexplained flaky tests; temporary flows nearing retirement | Record why automation is deferred and when to reconsider; improve environment, data, and observability |
The same test type can belong to different portfolios at different layers
Even API tests or UI tests can belong to different portfolios depending on their layers and conditions. The table gives default portfolios and reasoning for eleven test targets.
These classifications are not permanent labels attached to test types. An API test heavily dependent on shared data and an external sandbox may belong in the deferred portfolio, while a stable critical UI journey may be a high automation priority.
Test target | Default portfolio | Reason |
|---|---|---|
| Price, tax, and points calculations | Automation first | Deterministic inputs and outputs allow fast repetition at component level |
| API authentication and authorization | Automation first | Expected responses and data-access conditions can be checked objectively for each user role |
| API and event contracts | Automation first | Allowed request/response interactions, as well as schemas, can be checked repeatedly |
| Order creation, retries, and duplicate-prevention integration flow | Automation first, conditionally | High value when state reset and external dependencies can be controlled |
| Critical UI journey from login to completed payment | Automation + human review | Repeat checks of essential connections; separate human review for usability and exception exploration |
| Layout and design consistency | Automation + human review | Machines find diffs; people decide whether changes are intended or defective |
| Accessibility conformance | Automation + human review | Automatic rules find some barriers but cannot determine overall accessibility or user experience |
| New features with unclear requirements | Human-led | The team is learning what to verify and discovering new risks |
| Disaster recovery and failover | Automation + human review | Check state and recovery results automatically; verify decisions, roles, and communication through drills |
| External approval and unusually unstable sandboxes | Do not automate yet | Waiting for environments and investigating false positives may cost more than automatic execution saves |
| UI and network tests that vary under identical conditions | Do not automate yet | Results are not reliable quality signals until their causes are understood |
Evidence connected from requirement to approval — Link requirements, tests, defect actions and approval records so teams judge quality consistently.
Accessibility and visual judgment usually call for partial automation
Accessibility tools help scan many pages quickly and detect violations a machine can judge. W3C explains, however, that not all aspects of accessibility can be checked automatically and that tool results may be inaccurate or misleading. Tools must not replace human judgment and actual user experience.[9]
Divide the work into three parts. Machines collect structural rule violations, missing attributes, repeated problems, and differences between pages. People judge the meaning of alternative descriptions, natural keyboard movement, information order, and whether work can be completed. Users or specialists assess whether tasks are possible with real assistive technology in context.
Visual regression works the same way. Image-difference detection can be automated, but a separate oracle is needed to distinguish intended design changes from defects. Official automation strategy material also states that subjective qualities such as look and feel need human review.[6]
Rare but dangerous scenarios are exceptions to the frequency rule
It is easy to exclude password resets, bulk deletion, privilege escalation, failover, and backup recovery because they run infrequently. Risk, however, combines likelihood and impact. Rare scenarios still need appropriate verification and controls when failure would cause substantial harm.[8]
The entire process need not become one automation script. Automatically check referential integrity after deletion, critical data and service state after recovery, and required preconditions before failover. Verify human roles, approvals, emergency communication, and recovery decisions through regular drills. Automate stable invariants while leaving contextual decisions and collaboration to people.
Flaky tests are trust debt, not automation assets
A test that alternates between passing and failing with the same code and configuration is not a reliable signal, however quickly it runs. Every failure demands another investigation into product defects, test problems, environments, or networks. Eventually the team starts ignoring failure alerts.[3][4]
ISTQB recommends separating tests with unreliable repeated results from active automation suites and analyzing their causes independently. Unexpected intermittent failures can arise in test code, the system under test, the automation framework, hardware, or networks.[7] Counting a rerun pass as success leaves the cause intact. A rerun may provide reproduction evidence, but it is not a decision rule that resolves unreliable results.
Handle a flaky test in five steps. Isolate it from active gates. Investigate race conditions, shared data, time, asynchronous processing, and network dependencies. Include logs that distinguish environment errors from product failures. Fix the cause, verify repeated stability, and reinstate the test. If resolution costs exceed the value of repetition, reclassify it as not ready for automation. Google Research likewise describes how flaky tests undermine confidence in test results and disrupt development.[10]
Track portfolio health instead of automation counts
A healthy automation portfolio is not one whose test count keeps growing. It delivers the required signals on time at a cost that can be trusted. The eight indicators in the table describe that health.
ISTQB automation metrics also include average script maintenance time, execution time, failed scripts, and execution counts.[6] There is no need to impose identical targets on every organization. Compare trends and bottlenecks with the team's release cadence, system risks, and environment costs.
Indicator | Question |
|---|---|
| Change-to-Signal Time | How long after a change does the team receive a result it can act on? |
| Failure Analysis Time | Does investigating failures take longer than running tests? |
| Maintenance Time | Does fixing automation after each product change cost more than its repeated feedback is worth? |
| Environment / Data Blockage | Are environments and data the main causes of blocked or inconclusive runs? |
| Flaky / Inconclusive Trend | Are unreliable results increasing? |
| Human Review Yield | Does human review keep finding meaning, usability issues, and new risks that automation misses? |
| Portfolio Movement | Are deferred tests neglected, or do testability improvements move them into other portfolios? |
| Ownership | Is responsibility for failures and maintenance clear? |
Automation handles repetition; people handle meaning and discovery
Test automation does not simply replicate human clicks. It is an investment in turning repeatable checks into faster, more consistent evidence.
Good automation candidates run frequently, offer substantial risk reduction, and allow results, environments, and data to be reproduced reliably. A test that is difficult to automate is not necessarily less important. Human contextual judgment may be essential, or the current system may not support trustworthy automation.
A portfolio needs all four states. Automate repeatable decisions. Combine approaches when machines collect evidence and people interpret its meaning. Keep exploration and usability judgments human-led. Improve readiness conditions instead of forcing unreliable automation into a suite.
These classifications are not permanent. As a product matures and requirements and interfaces stabilize, human-led testing produces repeatable checks. Conversely, automation with growing maintenance costs and flakiness may return to mixed review or deferred status. Recent ISTQB guidance points in the same direction: exploration offers greater information value early in a product, while maintainable automation becomes a larger investment as it matures.[5][11]
The purpose of automation is not to eliminate people. It reduces time spent repeating executions so people can examine risks and meanings that machines cannot judge more carefully.
