When should an alert wake someone up? Designing actionable alerts

What does a page ask someone to do?

A page at 3 a.m. is not merely a notification that the system detected an anomaly. It should request intervention because user harm will grow without it and the recipient can actually improve the situation.

This is why designing pages from CPU usage at 80% or memory at 90% creates problems. CPU at 90% may coexist with ample spare capacity, working automatic scaling, and no user problems. Conversely, important flows such as payments or login may be failing while CPU looks ordinary.

Before asking how anomalous a signal is, ask three questions: Are users actually being harmed? How quickly is that harm growing? Can a person do something now? Google SRE likewise treats pagers as an expensive use of human time and asks whether a page is urgent, actionable, and connected to current or imminent user impact.^[1]

Are Page, Ticket, and Dashboard priorities?

Labels such as critical, warning, and info do not determine how operations will respond. A more useful distinction is when people must act. Google SRE makes a similar distinction: some situations need immediate action, some need action with days of latitude, and others only need recording for later diagnosis.^[1] The table expresses these as three operational routes.

Distinguish severity from route. A problem that looks technically serious does not automatically justify a page when nobody can do anything now. Conversely, a currently small problem may justify paging if it will quickly become irrecoverable and mitigation is available only now.

What does the Decision Matrix examine?

Choose whether to page by considering the conditions together, rather than using one metric. In this matrix, paging is close to an AND condition: actual or imminent user harm, short Time to Harm, insufficient automatic recovery, a human action available now, a clear owner, and evidence reliable enough to justify the interruption. If several conditions are absent, a ticket or dashboard is likely more appropriate.

Consider User Impact and Time to Harm together

User impact does not make every problem equally urgent. A certificate expiring next week clearly needs attention, but if sufficient time remains and renewal can be checked, it can generally be handled through the work queue.

Conversely, suppose storage is rapidly running out, writes are expected to stop before the next on-call handover at the current growth rate, and scaling is unavailable. Paging may be justified even before users see errors. Google SRE considers clearly imminent problems as well as user symptoms that have already occurred.^[6] It also stresses that paging rules should remain simple and understandable.

Instead of disk > 80%, ask: Given current growth and remaining headroom, how long before the service reaches a dangerous state? Can automatic or scheduled action happen first? This is the distinction between a fixed threshold and an actionable alert.

Why is a person needed if the problem fixes itself?

Suppose a server becomes unhealthy, the load balancer removes it, and a new instance replaces it successfully. If users were unaffected and no further action is required, the event may be valuable observation data without being a good reason to wake someone.

Recurrence changes the discussion. If instances die and restart several times daily while the service recovers automatically, a ticket to remove the cause may be more appropriate than a page. If a person runs the same command every time they are paged, the team is operating manual automation rather than a well-designed alert. Google SRE recommends considering automation when repeated pages have mechanically predetermined responses.^[1]

Automatic recovery does not eliminate every reason to page. Recovery itself may fail, users may continue failing during repeated recovery, or automatic actions may risk a larger incident and require a person to decide whether to stop them. The question is not whether recovery exists, but whether automatic recovery alone sufficiently controls the current risk.

What is the purpose of a page if nobody can act?

Suppose an external service fails. Paging is justified if our service can switch providers, limit functionality, preserve work in a queue, reroute traffic, or reduce the scope of customer impact.

If the recipient can only watch an external status page, review the paging policy. Immediate operational decisions such as incident communication, suspending transactions, or approving business actions still count as valid human actions, even without a technical fix. Actionability is therefore broader than an engineer being able to change code. It includes rollback, failover, load shedding, disabling features, traffic switching, and supplier escalation when acting now reduces harm.

The detailed response sequence and runbook structure belong to incident response. Alert policy operates earlier and only needs to decide whether to call a person now.

Why is an unowned page an ownership problem?

Putting alerts in a shared channel such as #alerts and hoping someone notices is hardly paging. At minimum, a page needs a service or feature owner, and the system must know the current on-call recipient. For external dependencies, distinguish the team responsible for user outcomes from the team responsible for the dependency itself.

An urgent service without settled ownership can temporarily be protected by a central on-call route. Do not leave that as its normal operating model. An unowned page is ownership debt, not alerting debt.

Why are false positives not free?

Sensitive alert conditions may reduce missed incidents but wake people more often during normal operation. Insensitive conditions reduce page counts but detect real incidents later.

The Google SRE Workbook describes this through precision, recall, detection time, and reset time.^[2] Assess how accurately alerts identify meaningful events, which events they miss, how long detection takes, and how quickly alerts clear after recovery.

Precision has a particularly high cost for pages. Repeated false positives do more than interrupt someone once: they undermine trust in the pager itself. This does not mean every page must wait for perfect certainty. Very short Time to Harm and severe consequences from missing an event may justify some uncertainty to reduce delay.

Set confidence criteria through a trade-off rather than a fixed number: Which costs more, waking someone unnecessarily or discovering this problem a few minutes later?

Should one dependency trigger dozens of pages?

One cause can trigger alerts in many components simultaneously. A database connection problem can cause API errors, queue delays, worker failures, failed health checks, and increased user error rates. If each alert creates a separate page, people appear to have five problems.

Prometheus Alertmanager provides deduplication and grouping for this situation, plus inhibition to suppress related lower-level notifications while a higher-level alert is active.^[3] An overly clever dependency graph is also risky. Google SRE explains that complex dependency-based alert hierarchies are hard to maintain as system structures change.^[1]

A more realistic goal than having the alert system identify the exact root cause is to present one incident as one representative page and keep other component signals as investigation context. The alerting system does not have to replace the entire diagnostic system.

What should repeated pages prompt before threshold changes?

When alerts recur, teams often raise thresholds first: CPU at 80% is noisy, so change it to 90%; if 90% is still noisy, extend the duration.

Requiring a condition to persist before firing, or keeping it firing briefly after recovery, is useful for reducing transient spikes and flapping. Prometheus provides for and keep_firing_for for these purposes.^[3] These are tuning methods for a page whose underlying need is already justified.

If people never act when paged, the threshold is not the problem. Repeated identical actions may indicate an automation problem. Ten pages from one outage indicate grouping problems. Another team already responding indicates routing or inhibition problems. Review the page’s reason for existing before refining its expression.

Why is fixed CPU 80% a poor starting point?

CPU and memory usage still matter. Resource metrics help explain causes and judge imminent saturation. The problem is that resource usage does not map one to one to user harm. The two table examples show how the same CPU number produces different routing decisions.

Set thresholds after verifying when a value leads to real user harm and what people must do beforehand, rather than asking which percentage is dangerous. Prometheus likewise recommends alerting at higher layers on symptoms connected to user pain, instead of turning every individual cause into an alert, and allowing room for small fluctuations.^[3]

How do actual scenarios classify?

The six examples show that routing depends on the time relationship between harm and action, not the metric type. Failed purchases and imminent capacity exhaustion become pages. Early certificate-expiry detection and daily worker restarts become tickets. An automatically replaced instance with no user impact belongs on a dashboard.

What must a page contain?

If recipients must interpret the situation from scratch, the alerting system has passed its work to people. A useful page includes the affected service or user function, when and where the problem began, the current owner, a runbook or initial check point, relevant dashboards, and identifiers that group the incident. Prometheus alerting rules also support descriptions and runbook links in annotations.^[3]

The alert need not establish a root cause. Attempting to infer both a rising payment failure rate and the specific database causing it can make rules excessively complex. A page should state something simpler with confidence: what is affected, why a person is needed now, and who should respond.

Should existing alerts keep being reclassified?

A page that was valuable initially may become unnecessary. Automatic recovery may be added, architecture may change, or a recurring problem may acquire a fully standardized response. Conversely, repeated evidence that a dashboard-only signal precedes incidents may justify a higher route.

Review actual event history rather than only whether the threshold is correct. Did someone act when paged? Was user harm present or truly imminent? How often had recovery already completed before anyone checked? Did one incident create multiple pages? Was the same manual response repeated? Do recipients trust the page and respond immediately?

Google SRE also tracks paging frequency as a sustainable on-call workload, rather than viewing each alert in isolation.^[7] Excessive pages take away time to investigate and improve real problems.

What is the final question before waking someone?

Alerting does not exist to forward every system anomaly to people. Operations can collect much more data: metrics, logs, traces, and dependency states are useful for diagnosis. But not every signal deserves to interrupt sleep or concentration.

Before creating a page, one question should remain: What should the person receiving this alert do now, and what user harm would grow if that action waited until morning? A clear answer makes it a paging candidate. Necessary action that can wait belongs in a ticket. Signals requiring no action belong on a dashboard or in a record.

Alert quality shows in the ability to call people when they are truly needed, more than in the number of anomalies detected.