Alert straight from Grafana — define alert rules on your panels, send notifications through contact points, and route them with notification policies.
Why: you can alert from Prometheus (rules → Alertmanager, covered earlier) OR from Grafana, which has its own alerting engine. Grafana alerting is panel-centric and unifies alerts across many data sources in one UI. Many teams use Prometheus rules for infra and Grafana alerts for dashboard-driven ones — both are valid; know they coexist.
Prometheus rules ─▶ Alertmanager ─▶ notify (covered in earlier lessons)
Grafana alert rules ─▶ contact points ─▶ notify (this lesson)
(both can run; pick per use case)Why: a Grafana alert rule evaluates a query on a schedule and fires when a condition holds. You build it in Alerting → Alert rules: a PromQL query, a threshold condition, an evaluation interval, and a "for" duration (the pending period before firing) — the same anti-flapping idea as Prometheus rules.
# Alert rule (built in the UI):
# Query A: sum(rate(http_requests_total{status=~"5.."}[5m]))
# / sum(rate(http_requests_total[5m]))
# Condition: WHEN last(A) IS ABOVE 0.05
# Evaluate: every 1m for 10m (pending before firing)Why: a contact point is where a notification goes — Slack, email, PagerDuty, a webhook. A notification policy is the routing tree that matches alerts (by label) to contact points, just like Alertmanager's routes. Set up the contact point first, then the policy that sends critical alerts to it.
Alerting → Contact points define WHERE (Slack webhook, email, ...)
Alerting → Notification policies match alert labels → route to a contact point
Alerting → Silences mute matching alerts during maintenanceNote: an alert you have never seen fire is an alert you cannot trust. After wiring a rule to a contact point, force the condition (or use Grafana's test notification) and confirm the message actually arrives. Verify the full path — rule fires → policy routes → contact point delivers — before relying on it in production.
1. trigger the condition (or send a test notification)
2. rule goes pending → firing
3. notification policy routes it
4. contact point delivers → you receive the message
(if any step is silent, fix it now — not during an incident)