Alerting & on-call — Subnomic Docs

Alerting watches your fleet against rules you define and pages the right people when something crosses a line — then, for the serious ones, opens an incident and escalates to whoever is on call. It's the loop that turns a metric into a response.

Alert rules

Under Alerts → Rules, define a threshold rule:

Source — what to watch: cpu, memory, disk (server metrics) or unhealthy_resources (a count of failing workloads).
Comparator + threshold — e.g. disk ≥ 90, unhealthy > 0.
For-window — how long the condition must hold before firing, so a brief spike doesn't page anyone (capped to the metric retention window).
Scope — a single server, or the whole fleet when no server is set.
Severity (low / medium / high) and an optional description, plus an enabled toggle.

How alerts fire & resolve

A background evaluator samples metrics and resource health each tick and compares them to your enabled rules — it is the sole writer of alerts. When a rule's condition holds for its for-window an alert opens and the tenant's owners/admins are notified; when the condition clears, the alert auto-resolves. You can also resolve a firing alert by hand.

Alerts lists what's firing now (with severity); the dashboard shows an active-alerts widget.
Alert notifications are on by default — no extra wiring.

On-call & escalation

High-severity alerts (and detections) open an incident automatically. Set up an on-call rotation under On-call so there's always a current responder. If an incident stays open and unacknowledged for too long, a background escalator pages the current on-call — exactly once. Acknowledge an incident to stop escalation.

The whole loop is safe across replicas — alerts and escalations are claimed one at a time, so an alert pages and an incident escalates exactly once. See Incidents & autonomous SRE for what happens after an incident opens.

What changed? (correlation)

Most incidents trace back to a change. The Recent changes panel on an incident surfaces the tenant's recent mutating activity — task runs, deploys, workflow runs, resource and server changes — within a window, ranked so the most recent rank highest. It's the first place to look for "what did we just do?".

Permissions

alert.read alert.write oncall.manage incident.read

alert.read to view rules & alerts, alert.write to manage rules and resolve alerts, oncall.manage to edit rotations (incident.read to view them). The evaluator and escalator are background jobs and need no user permission. Hidden until an administrator enables it for your workspace.