Turn visitor counts into a request mix
The visitor count a 2 vCPU, 4 GB server can take cannot be computed from a spec sheet. The same monthly visitors produce entirely different load for a service where most traffic hits cached public pages and one that repeats login, search, checkout, and file upload. Server capacity is more accurately defined as the maximum load sustained under a defined request mix while every latency, error, resource pressure, and queue criterion holds. Google Cloud's load testing guidance likewise directs you to find acceptable thresholds across latency, throughput, and resource use rather than one absolute maximum.[1] Google SRE similarly asks for traffic, latency, errors, and saturation to be read together.[3]
The label 2 vCPU does not guarantee equal compute either. On some VMs a vCPU is a hardware thread; on other families it maps to a physical core. Shared CPU makes the usable cycles depend on other workloads on the same host. CPU generation, SMT layout, storage type and limits, and network throughput all belong in the record.[2]
So the result is written not as "2 vCPU, 4 GB equals N visitors" but in this form. Under workload profile [version] and environment [fingerprint], completed throughput of [value] held for [window]. Per-route P95 latency was [value], error rate [value], maximum queue [value], and the first limiting signal was [CPU, memory, database, I/O, network, external dependency]. Validated headroom against expected peak is [value], and this result holds only under [cache state, data size, background work, provider conditions].
Monthly visitors say nothing about how load distributes over time. The same hundred thousand visits can spread evenly across a month or arrive within minutes. Nor does one visit mean one server request. What the server actually receives usually mixes eight strands: public reads served from CDN or page cache; authenticated reads that pass through both application and database; writes that require data change and transactions; search, filter, and aggregation requests with heavy CPU and database cost; image conversion, file upload, and report generation; requests waiting on an external API; jobs processed asynchronously from a queue; and background work such as cron, backup, and indexing.[3]
Load testing has to separate at least three numbers. When offered RPS rises while completed RPS stops rising, work is piling up or being dropped somewhere ahead of or inside the server. In a closed test using a fixed virtual user count, a slower server also delays the start of the next request. As a result, offered load reduces itself just as the real processing limit is reached. Validating a target RPS therefore calls for an open arrival-rate model that starts requests independently of response time.[7]
Concurrent work matters more than concurrent users. A thousand people with a browser tab open does not mean a thousand jobs running on the server. Conversely, a small user count with slow database queries or external APIs leaves server-side work resident for longer and raises concurrency. In a stable system, average concurrent work L equals completion rate λ times average residence time W — Little's Law. It is a cross-check for windows near steady state with finite means, and P95 or P99 must never be substituted for average residence time.[5] In practice, track in-flight HTTP requests, busy application workers, running database queries, connection pool waiters, jobs queued in messaging, and work waiting on external API responses separately.
- Offered RPS
- The request rate the load generator attempted to send.
- Completed RPS
- The throughput for which the server finished a response.
- Successful RPS
- Completed throughput that also succeeded functionally.
Define capacity before testing it
Raising load and then eyeballing a graph to decide "this looks fine" lets the criterion drift every time. Pass/fail criteria are declared before the test. The three concepts mean the following.
Grafana k6 thresholds are likewise defined as pass/fail criteria against test metrics. The documentation's P95 of 200 ms and error rate under 1 percent illustrate usage; they are not recommended values for every service. Real criteria come from the service's user expectations and operating goals.[4]
Headroom is not one CPU margin figure. The calculation means something only when request mix, data size, cache state, code, and dependencies are the same. Nor is there a universal answer for the required margin. It is set by forecast error at peak, the size and duration of short spikes, the lead time for provisioning or autoscaling, overlap with batch, backup, and indexing work, the capacity left when one instance or an external dependency fails, and the latency and errors the service tolerates. Google Cloud likewise explains that optimal resource utilisation differs per application and that performance degrades before a system reaches 100 percent.[1]
Cross-check once more with CPU time. With containers or systemd cgroups, usage_usec in cpu.stat yields the CPU time over the window.[8] Where CPU is the main bottleneck and per-request CPU cost holds roughly steady with load, available CPU seconds times the target utilisation, divided by CPU time per request, serves as a secondary check. Think of 2 vCPU as offering up to roughly two CPU seconds per second when scheduling conditions allow. Shared CPU, CPU quota, throttling, SMT, single-thread bottlenecks, and database or network waits mean the expression does not state real throughput. The final judgment always comes from the load test.
- Validated sustainable capacity
- The maximum load that, under a defined request mix and over a stated measurement window, meets the throughput target, holds the latency and error criteria, keeps queues from growing, produces no resource pressure, OOM, or abnormal throttling, and returns to a normal state once load is removed.
- Capacity headroom
- Validated sustainable completed throughput divided by expected peak completed throughput, minus one.
- CPU time per request
- CPU seconds used over the measurement window divided by completed requests; watch whether it changes as load rises.
From production traffic to improvement priorities — Model the request mix, investigate bottlenecks and retest the effect of each improvement.
Record the server's environment fingerprint first
Comparing another provider's 2 vCPU, 4 GB server, or reproducing the same test, requires these conditions to be recorded alongside the numbers.
Google Compute Engine maps a vCPU to a core on some machine families and defaults to two vCPUs per core on others. On many AWS EC2 instances one CPU thread is presented as one vCPU. DigitalOcean likewise separates shared and dedicated CPU by the level of access guaranteed, along with CPU generation, NVMe, and network performance. Four numbers reading 2 vCPU, 4 GB are therefore not enough to compare two different VMs as equals.[2]
Area | What to record |
|---|---|
| Compute | Provider, region and zone, plan name, vCPU count, CPU model and generation, architecture |
| CPU allocation | Shared or dedicated, SMT, vCPU to core mapping, CPU quota |
| CPU anomaly signals | Throttling time, steal time, skew onto a single core |
| Memory | Actually usable memory, container or cgroup limit, swap size and policy |
| Storage | Local, block, or network storage, SSD or NVMe, volume size, IOPS and throughput limits |
| Network | Instance network limits, packet and connection caps, path to the load generator |
| Topology | Whether app, database, cache, and queue share a server or not |
| Software | OS, kernel, runtime, web server, database, cache versions |
| Process model | Worker, thread, and process counts, connection pool and queue limits |
| Data | Data size, index size, value distribution, number of test accounts |
| Cache | Cold or warm, TTL, warm-up method |
| Build | Commit, image digest, configuration version, feature flags |
| Dependencies | External APIs, messaging, storage, authentication, and what is mocked |
| Load generator | Tool and version, location, its own CPU, memory, and network spec |
What must be measured alongside
The four Google SRE signals are a starting point, but a capacity judgment needs internal metrics that separate the cause. Latency of successful and failed requests has to be separated too.[3]
P50 is the median, and P95 means 95 percent of observations fall at or below that value. When several instances or windows must be combined, the instrumentation method and histogram bucket design need review as well.[9]
On Linux, PSI shows the time work stalled on CPU, memory, or I/O contention. Cgroup v2 exposes CPU usage time and throttling, total memory usage, OOM events, and read and write bytes and I/O counts. Read cpu.stat, memory.current, memory.stat, memory.events, and io.stat under the workload cgroup path, and read cpu, memory, and io under /proc/pressure alongside them.[6]
memory.current is the cgroup's total memory usage, not a computed working set. Estimate the operational working set from the usage that stabilises under sustained load together with refault, major faults, swap, PSI, and OOM events. On PostgreSQL, use numbackends, blks_read, blks_hit, I/O time, active time, and wait events.[10] On another DBMS, pick the equivalents that answer the same questions. With Redis as a cache, record keyspace_hits, keyspace_misses, and evicted_keys, and read miss and eviction growth per stage rather than only the overall hit ratio.[11]
Metric | What to record | The question it answers |
|---|---|---|
| Request mix | Share by route and operation, read or write, auth, payload, cache state, database and external call counts | Does it carry the same cost distribution as production traffic? |
| Requests per second | Offered, started, completed, and successful RPS per route | Does completed throughput keep rising as input rises? |
| Concurrent work | In-flight requests, active workers, active database queries, pool waiters, queue depth | Does work in progress stabilise, or keep piling up? |
| CPU utilisation | Host, cgroup, process, and per-core usage, user and system, steal, throttling | Is the CPU really busy, or capped by one core or a quota? |
| CPU time | CPU seconds per window, CPU ms per request or job | Does the per-unit CPU cost change as throughput grows? |
| Memory working set | Post warm-up cgroup usage, RSS, heap, page cache, peak, refault | Does required memory reach equilibrium, or keep climbing? |
| Swap and OOM | Swap in and out, major faults, reclaim, memory.events high, max, oom, oom_kill | Is memory shortage being paid for in swap latency or OOM? |
| Database queries | QPS and TPS, queries per route, query P50 and P95, slow queries, locks and wait events | Does the database saturate before the application? |
| Database I/O | Read and write IOPS, bytes, latency, fsync, temp writes | Does I/O wait rise before disk throughput does? |
| Database connections | Active, idle, waiting, maximum connections, pool utilisation and wait time | Do connection limits or pool waits create the request queue? |
| Cache hit | Hit and miss and eviction per layer — CDN, reverse proxy, application, Redis, database buffer | Did the test measure only one cache layer? |
| Network | In and out bytes, packets, new connection rate, connection errors and retransmits, response size | Is network or connection handling capped while CPU remains? |
| Queue depth | Web and app worker queue, database pool wait, message queue lag, oldest age | Does the queue stabilise, or grow with time? |
| P50 latency | Median of successful and failed requests per route | Is the ordinary request experience holding? |
| P95 latency | 95th percentile of successful and failed requests per route | Do the slow upper requests breach the criterion? |
| P99 latency | Measured with enough samples on routes where tail risk matters | Do rare delays threaten payment, authentication, or persistence? |
| Error rate | 5xx, timeouts, connection errors, rejections, functional failures, wrong results | Is fast failure making average latency look better? |
| Saturation | CPU, memory, and I/O PSI, throttling, queue, lock, pool wait | Does real work stall before utilisation reaches 100 percent? |
| Headroom | Expected peak, validated sustainable capacity, first breached criterion, run-to-run variance | Is enough margin left for peak, burst, and recovery? |
Benchmark protocol
First, write down the question the test must answer. "How many users can this server take?" is a poor goal. "Under the same request mix as the production peak, does it hold completed throughput of [target] while meeting per-route latency, error, queue, and resource pressure criteria?" is a good one. Distinguish also whether the subject is a normal peak, a momentary spike, or sustained load over a long window.
Second, fix pass/fail and abort conditions first. Before any load is sent, settle per-route P50, P95, and where needed P99 targets, tolerances for functional and system errors, the completed throughput target, acceptable queue depth and pool wait, CPU throttling and resource pressure criteria, the prohibition on swap and OOM, database connection, query, and I/O criteria, the abort conditions, and the recovery criterion after load stops. Without service-specific targets, set interim operating criteria first and do not present them as an industry standard.
Third, freeze the environment and store the fingerprint. Record the application build and configuration, worker counts, database and cache versions, data size, VM plan, and CPU model. Prevent auto-updates, deployments, backups, and data migrations from running during the test. If backups or batch jobs really do overlap the peak in production, run a separate overlap scenario.
Fourth, reconstruct the real request mix. Derive the share per route and operation from production logs or analytics. Strip personal data and credentials, and reproduce the distribution so the test data does not collapse onto one or two cache keys. Without production data, build a hypothetical profile from the product flow and label the result a synthetic workload.
Fifth, separate cold from warm. Never average an empty cache and a warmed cache into one result. Among cold start, warm steady state, post-eviction, immediately after deployment, and immediately after restart, choose only the profiles tied to real operational risk and record each separately.
Sixth, confirm the load generator is not the bottleneck. Watch the generator's CPU, memory, network, open file count, and dropped requests. A saturated generator measures the limits of the test rig rather than the SUT. The k6 guidance for large tests warns that fully saturating generator CPU degrades results and gives keeping CPU under 80 percent as an example. That is a check on the generator, not a recommended CPU threshold for the server under test.[12]
Seventh, run an idle baseline and a smoke test. Record CPU, memory, database, and queue with no load. Then send load small enough to confirm functional success and spot missing instrumentation. If the smoke stage shows functional errors or observability gaps, do not proceed to the capacity test.
Eighth, raise load in stages and hold each stage. Raise the target RPS or job rate from small steps. Do not take a momentary peak from each stage; hold it long enough to reveal whether completed throughput stabilises, whether P95 keeps climbing, whether the working set reaches equilibrium, whether the queue settles at a level, and whether database or external dependency latency accumulates. Use an open arrival-rate model to validate a target RPS and a closed model to validate a fixed concurrent session count. Do not read the two results as meaning the same thing.[4]
Ninth, record the first breached criterion. There is no need to wait for the server to stop entirely. Record the point at which one of these first appears persistently: a per-route latency breach, an error rate breach, completed throughput flattening, a continuously growing queue, rising CPU throttling or PSI, memory reclaim, swap, or OOM risk, rising database pool, lock, or I/O wait, collapsing cache hit with rising eviction, or external dependency timeouts. The last stage just below that point where every criterion held becomes the candidate for validated sustainable capacity.
Tenth, confirm controlled overload behaviour. In a non-production environment, set safe abort conditions and observe how the system fails past capacity. Do all requests slow together? Are some requests explicitly rejected? Do timeouts accumulate? Does the queue consume memory? Does it recover once load drops? Google Cloud recommends load shedding — rejecting some requests to preserve performance for the rest — over slowing every request under overload. The actual policy still has to be designed around the service's business characteristics and retry safety.[1]
Eleventh, check soak and recovery. Sustain expected peak long enough to surface time-dependent problems: memory leaks, connection leaks, growing queue backlog, cache eviction, database temp file, WAL, and checkpoint effects, log and disk growth, and interference between periodic GC and batch work. After load is removed, confirm that queues drain within the stated time, that latency and errors return to baseline, that memory returns to its stable band, that connections and workers normalise, and that no OOM or process restart occurred.
Twelfth, repeat under identical conditions. Do not pick a single best run. Repeat under the same conditions and record the median and the run-to-run range. If variance is large on a shared-CPU VM, present the lowest, median, and highest runs rather than an average, and read CPU steal, throttling, and storage latency alongside them.
Capacity test worksheet
The blanks in the form below are filled with real measurements. The worksheet has five blocks.
The mix in block B must total 100 percent within the chosen profile. Different profiles such as normal peak, write-heavy, cold cache, and batch overlap are managed as separate versions.
D separates throughput, latency, and stability in D-1 from resources and bottlenecks in D-2. Fill the field-level blanks below. Both forms retain Baseline, Warm-up, L1, L2, L3, Overload, and Recovery.
Block E fits on one page. It records the test ID, workload profile version, application build, environment fingerprint, expected peak demand, validated sustainable completed and successful throughput, P50, P95, P99, error rate, maximum queue, CPU utilisation and CPU time per request, working set estimate, swap and OOM, database query, I/O and connection figures, cache hit and eviction, network, PSI and saturation, the first breached criterion, the likely bottleneck, headroom, the recovery result, and run-to-run variance. It closes with the conditions under which the result holds and the changes that force a retest.
- A. Test identity and environment
- This block identifies exactly what was tested. Enter actual environment values in the A fields below.
- B. Request mix profile
- Records, per scenario, the user operation and route, mix percentage, arrival model, payload, authentication, read or write, cache state, database queries per request, and external calls.
- C. Pass / fail criteria
- Fixes the criteria, measurement window, and scope for throughput, latency, errors, CPU, memory, database, cache, network, queue, recovery, and headroom before any load is sent.
- D. Stage result
- Two blocks that split service metrics from resource and dependency metrics per stage. The first holds RPS, latency, errors, and queue; the second holds CPU, memory, database, I/O, cache, network, PSI, and the first breach.
- E. Final capacity statement
- One page carrying the validated sustainable throughput, latency, errors and queue, the first limiting signal, headroom, the conditions under which the result holds, and the changes that force a retest.
- A / Identity / Test ID
- [enter value]
- A / Identity / Purpose and decision question
- [enter value]
- A / Identity / Run date and owner
- [enter value]
- A / SUT / Commit / Image Digest
- [enter value]
- A / SUT / Configuration and Feature Flag version
- [enter value]
- A / Compute / Provider / Region / Zone / Plan
- [enter value]
- A / Compute / vCPU / RAM
- [enter value]
- A / Compute / CPU model, generation, and architecture
- [enter value]
- A / Compute / Shared / Dedicated / SMT / Core Mapping
- [enter value]
- A / Limits / cgroup·Container CPU quota
- [enter value]
- A / Limits / Memory limit / Swap / PID·File limit
- [enter value]
- A / Storage / Storage type, size, IOPS, and throughput limits
- [enter value]
- A / Network / Instance limit and test path
- [enter value]
- A / Software / OS / Kernel / Runtime / Web Server
- [enter value]
- A / Process / Worker, thread, and queue configuration
- [enter value]
- A / Database / Location, version, size, and connection pool
- [enter value]
- A / Cache / Location, version, capacity, TTL, and eviction
- [enter value]
- A / Data / Record/index size and distribution
- [enter value]
- A / Dependencies / External API, storage, authentication, and mock scope
- [enter value]
- A / Generator / Tool, version, specification, and location
- [enter value]
- A / Telemetry / Metric·Log·Trace Dashboard
- [enter value]
- A / Safety / Execution window, stop conditions, and recovery owner
- [enter value]
- B / [S-01]
- User task: [enter value] · Route·Protocol: [enter value] · Mix %: [enter value] · Arrival Model: [enter value] · Payload: [enter value] · Auth: [enter value] · Read/Write: [enter value] · Cache State: [enter value] · DB Query/Request: [enter value] · External Call: [enter value] · Notes: [enter value]
- B / [S-02]
- User task: [enter value] · Route·Protocol: [enter value] · Mix %: [enter value] · Arrival Model: [enter value] · Payload: [enter value] · Auth: [enter value] · Read/Write: [enter value] · Cache State: [enter value] · DB Query/Request: [enter value] · External Call: [enter value] · Notes: [enter value]
- B / [S-03]
- User task: [enter value] · Route·Protocol: [enter value] · Mix %: [enter value] · Arrival Model: [enter value] · Payload: [enter value] · Auth: [enter value] · Read/Write: [enter value] · Cache State: [enter value] · DB Query/Request: [enter value] · External Call: [enter value] · Notes: [enter value]
- D-1 / Baseline
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-1 / Warm-up
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-1 / L1
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-1 / L2
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-1 / L3
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-1 / Overload
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-1 / Recovery
- Offered RPS: [enter value] · Completed RPS: [enter value] · Successful RPS: [enter value] · Concurrent Work: [enter value] · P50: [enter value] · P95: [enter value] · P99: [enter value] · Error Rate: [enter value] · Queue Depth: [enter value] · Stable: [enter value]
- D-2 / Baseline
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
- D-2 / Warm-up
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
- D-2 / L1
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
- D-2 / L2
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
- D-2 / L3
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
- D-2 / Overload
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
- D-2 / Recovery
- CPU %: [enter value] · CPU ms/req: [enter value] · Throttle·Steal: [enter value] · Working Set: [enter value] · Swap·OOM: [enter value] · DB QPS·P95: [enter value] · DB Conn·Wait: [enter value] · IOPS·Latency: [enter value] · Cache Hit·Evict: [enter value] · Network: [enter value] · PSI: [enter value] · First breach: [enter value]
Area | Metric | Criterion | Measurement window | Scope |
|---|---|---|---|---|
| Throughput | Completed / successful RPS | To be filled | To be filled | Overall and per route |
| Latency | P50 | To be filled | To be filled | Per route |
| Latency | P95 | To be filled | To be filled | Per route |
| Latency | P99 | Fill only for routes that need it | To be filled | Per route |
| Error | System error rate | To be filled | To be filled | Overall and per route |
| Error | Functional failure rate | To be filled | To be filled | Core operations |
| CPU | Utilisation / PSI / throttling | To be filled | To be filled | Host and cgroup |
| CPU | CPU ms per request | To be filled | To be filled | Overall and per route |
| Memory | Working-set estimate / PSI | To be filled | To be filled | Host and cgroup |
| Memory | Swap / OOM | To be filled | To be filled | Host and cgroup |
| Database | Query P95 / connection wait | To be filled | To be filled | Database and pool |
| Database | IOPS / I/O latency / lock | To be filled | To be filled | Database and storage |
| Cache | Hit / miss / eviction | To be filled | To be filled | Per layer |
| Network | Throughput / error / retransmit | To be filled | To be filled | App and database paths |
| Queue | Depth / oldest age / pool wait | To be filled | To be filled | Per queue |
| Recovery | Time to return to baseline | To be filled | After load stops | Overall |
| Headroom | Margin against expected peak | To be filled | Final judgment | Workload profile |
From service architecture to operating standards — Design networks and servers together with deployment paths, access controls and backup policies.
Result interpretation table
A bottleneck in the table below is a hypothesis for the next round of verification, not a settled cause. Several resources can saturate at once.
Linux PSI shows the time real work stalled on CPU, memory, or I/O contention, which makes it useful for separating utilisation from saturation. PostgreSQL connection, I/O, and wait event data and Redis hit, miss, and eviction counters likewise separate bottlenecks outside the app server.[6]
Observed pattern | Leading hypothesis | What to check next | Conclusion to avoid |
|---|---|---|---|
| Offered RPS rises, completed RPS flattens, queue keeps growing | Sustained processing limit reached | Where the wait sits: worker, database pool, message queue | Errors are low, so there is still headroom |
| CPU, CPU PSI, and throttling rise while CPU time per request stays flat | Compute saturation | Per-core usage, single thread, quota, steal | Reading only average CPU and ignoring database and queue |
| Total CPU is low but one core saturates and latency rises | Single thread, lock, or serialised section | Runtime profile, lock wait, event loop lag | More vCPUs will improve it linearly |
| Working set keeps growing with GC, refault, swap, and memory PSI | Memory leak or shortage | Heap, page cache, connection and buffer leaks | Adding RAM solved the leak |
| App CPU is low but database query P95, pool wait, lock, and I/O rise | Database bottleneck | Top SQL, execution plans, connections, wait events | Adding app instances will fix it |
| Cache hit falls while database QPS and I/O rise together | Cache churn, key distribution, or warm-up problem | Miss per layer, TTL, eviction, test data | Concluding only that the database is slow |
| P50 stays flat while P95 and P99 climb sharply | Tail contention | GC, locks, connection wait, external APIs, slow queries | Passing on average response time alone |
| Error rate rises while average latency falls | Fast failure, rejection, connection errors | Split success and failure latency, status codes, functional success | Concluding that latency improved |
| CPU and database have room but network throughput and connection errors near a limit | Network or connection handling bottleneck | Packets, retransmits, connection rate, payload | Declaring a vCPU shortage |
| Adding app instances barely raises total throughput | Shared database, cache, queue, or external dependency bottleneck | Saturation of each subsystem | Continuing to add more app replicas |
| Same image and load, but performance varies run to run | Shared CPU, storage, or network variability | CPU steal, throttling, disk latency, host class | Publishing one best run as the server's performance |
| CPU utilisation is high but latency, errors, queue, and PSI are stable | Current load is being served; margin still needs review | Spike, recovery, run-to-run variance | Failing it on high CPU alone |
| After load stops, queue, memory, and connections do not normalise | Recovery failure, leak, or retry storm | Queue drain, retry, connection lifecycle | Passing on the load-phase P95 alone |
Scale-up and scale-out trigger template
Scale-up and scale-out are not decided from one CPU graph. Which of the three routes becomes the leading candidate depends on what the first bottleneck is.
The decision is written down. A trigger record carries the decision ID, measurement date, owner, workload profile version, application build, and environment fingerprint. It then records the expected peak's offered and completed rates and request mix, and the validated capacity's rate, duration, repeat count, and run-to-run variance. Headroom carries the formula, the result, and why that margin is required — forecast error, spike size, scaling lead time, background work overlap, and capacity left after a failure.
The first breached criterion records the metric, threshold, observed value, window, and scenario. The bottleneck class picks one of CPU, memory, database, storage, cache, network, queue, or dependency, with both supporting and contradictory evidence recorded. The scale-up and scale-out triggers each carry a condition, a candidate action, the expected effect, an invalidation condition, and whether a retest is required. The scale-out trigger adds prerequisites: load distribution, health checks, state and session handling, downstream headroom, and deployment and rollback all verified. It closes with rollback conditions and the post-change validation plan.
- Scale up
- The conditions that make it one: When the required memory working set exceeds the instance's practical limit and it is not a leak. When single-instance CPU or I/O is clearly the first bottleneck and a higher tier could improve it. When state or local data makes horizontal distribution impractical for now. When shared-CPU variability is the problem and a dedicated CPU or another performance tier is needed
- Scale out
- The conditions that make it one: When the per-instance bottleneck reproduces and requests distribute independently across instances. When session, file, and job state is externalised or consistently shared. When the load balancer and health checks have been verified under real load. When the database, cache, queue, and external dependencies have room for the added app load. When target throughput must hold while instances drop out during deployment or failure
- Optimise or restructure first
- The conditions that make it one: When slow queries, locks, or connection pool waits are the first bottleneck. When cache hit collapse or eviction is the cause. When there is a memory or connection leak. When queues are unbounded and retries storm. When an external API rate limit or timeout is hit. When a single global lock or serialised section exists. When the load generator itself is saturated
Conclusion
A 2 vCPU, 4 GB server is enough for some services and short from the start for others. What makes the difference is not monthly visitors but the cost and distribution of the requests.
A good capacity judgment carries nine things: which requests were sent in what proportion; what offered, completed, and successful RPS each were; how much work was concurrently processed or waiting; whether per-route P50, P95, and where needed P99 held their criteria; how CPU time per request and throttling behaved, not only utilisation; whether memory working set, swap, OOM, and pressure stayed stable; which of database, cache, storage, network, and queue saturated first; whether the required headroom above expected peak remained; and whether the system returned to a normal state once load disappeared.
The closing sentence is not "it handles N users" but this. This server has been validated to this throughput under this workload profile and environment, the first limiting resource is this, and this much margin remains against the expected peak. Change the code, the data, the request mix, the cache policy, the provider, the CPU tier, storage, or network, and that sentence has to be measured again.
