Monitoring & Alerting Setup Checklist Template

The scope answer on the first task decides whether the build phases or the review phase appear.

• Open the record, name the service and choose the scope — new service or quarterly alert review, with the service owner and on-call lead picked from your members
• List every component the service depends on — hosts, containers, databases, queues, load balancers, DNS, certificates and third-party APIs
• Write down the user journeys that define “working” — usually two or three, such as sign in, place an order or call the API
• Check which components already report to monitoring — agents installed, exporters running, logs shipped; anything missing goes on a gap list
• Confirm an on-call rotation and escalation policy cover this service — an alert with no rotation behind it pages nobody
• Note the monitoring and paging tools in use and any end-of-life dates — a tool migration changes every route and needs the same testing

Tasks appear only when the scope is a new service.

• Instrument the four golden signals at the service edge — latency, traffic, errors and saturation, measured at the load balancer or API gateway
• Add RED metrics for each internal component — request rate, errors and duration per endpoint or queue consumer
• Apply USE to each resource — utilisation, saturation and errors for CPU, memory, disk, network and connection pools
• Define an SLI for each user journey — for example, the proportion of sign-in requests served successfully in under 500 ms
• Agree the SLO and error budget with the service owner — 99.9% over 30 days allows about 43 minutes of total failure
• Add synthetic checks for each journey from outside your network — they catch DNS, certificate and CDN failures internal metrics never see

Tasks appear only when the scope is a new service. The alert table lives in this phase.

• Agree what each severity level does — for example, page now, ticket for the next working day, or dashboard only
• Base pages on symptoms wherever you can — error rate against the SLO rather than CPU above 80%; keep cause alerts for definite causes such as a disk filling within hours
• Add burn-rate alerts to each SLO — the SRE Workbook pages on a 14.4× burn over one hour or 6× over six hours, and raises a ticket at 1× over three days
• Give every threshold a duration — a for: 5m clause in Prometheus, or the equivalent elsewhere, so one spike does not wake anyone
• Record every alert in the alert table — name, condition and threshold, severity, route and runbook link, one row each
• Configure grouping and inhibition — related alerts arrive as one notification, and warnings stay quiet while the matching critical is firing
• Route pages to the rotation, not to named people — with escalation to the secondary if a page is not acknowledged in time

Tasks appear only when the scope is a new service.

• Write a runbook for every paging alert — what it means, who is affected, the first three checks, how to mitigate and when to escalate
• Put the runbook and dashboard links in the notification itself — at 03:00 nobody searches the wiki
• Build the service dashboard from the top down — golden signals and SLO first, error budget remaining next, resources last
• Add deployment and change markers to the dashboard — most incidents follow a change, and the marker shows which one
• Demote any alert whose runbook says “wait and see” — if there is nothing to do, it is a ticket or a panel, not a page

A Fail result shows the fix task. The on-call lead’s approval on the last task halts the checklist until they decide.

• Validate rules and routing before deploying them — promtool check rules, amtool check-config, and amtool config routes test with each alert’s labels
• Fire each paging alert once under control — stop a test instance, break a synthetic check or inject errors in staging, then confirm the resolve notification too
• Send a test notification to everyone on the rotation — to the phone, not just a chat channel, including any SMS or voice fallback
• Add a watchdog alert that always fires to an external heartbeat service — if the monitoring pipeline dies, the heartbeat stops and something else pages
• Record the end-to-end test result — pass, or fail with the alerts that did not arrive and where they went instead
• Fix the failed routes and test them again — go-live waits until every paging alert reaches the rotation
• Approve the alert set — the on-call lead records approved or not approved, with the reason

Tasks appear only when the scope is the quarterly alert review.

• Export every alert that fired this quarter — count, time to acknowledge and what the responder did
• Tune, demote or delete alerts that fired with no action taken — a page nobody acted on is noise, and noise trains people to ignore pages
• Count pages per on-call shift against a target — the SRE book sets a maximum of two incidents per 12-hour shift
• Find incidents no alert caught — from postmortems and customer-reported tickets; add the missing symptom alert
• Check every alert still has an owner, a working route and a runbook that opens — people leave and teams reorganise
• Remove silences and maintenance windows that outlived their purpose — an open-ended silence is an alert switched off

• Update the alert table and the service catalogue entry — the table on this run becomes the baseline for the next review
• Brief the rotation on new, changed and deleted alerts — at the next on-call handover, so nobody meets a new page cold
• Raise tickets for every gap found — missing instrumentation, runbooks to write, dashboards to fix, each with an owner and a date
• Confirm the date of the next quarterly review — the recurring schedule opens it automatically
• Sign off and close — the service owner confirms the alert set, with the routing changes and test evidence attached