TOPIC DEVOPS
Monitoring & Observability knowing what your system is doing at all times
IN 10 SECONDS
Monitoring collects metrics (CPU, latency, error rates) and triggers alerts when thresholds are breached. Observability goes further — with traces (distributed request paths), structured logs, and rich metrics — so you can ask arbitrary questions about system behavior without pre-defining every dashboard.
GOTCHA Alert fatigue is real. If an alert fires but no one acts on it, it's noise — delete or tune it. Every alert should be actionable, urgent, and specific. A good rule: no dashboard is complete without a 'RED' panel (Rate, Errors, Duration).
HOW A METRIC BECOMES AN ALERT
01 Exporter/node agent runs on every machine, collecting CPU, memory, disk, and custom app metrics and exposing them at `/metrics`.
02 Prometheus server scrapes all targets at a configured interval (e.g., 15s) and stores metrics in a time-series DB.
03 Alertmanager evaluates PromQL rules — if `node_cpu_seconds_total > 0.9` for 5 minutes, fires an alert.
04 Alertmanager deduplicates, groups, and routes the alert to PagerDuty, Slack, or email based on severity and team.
05 On-call engineer receives the notification, opens Grafana, and investigates the dashboard or traces to find the root cause.
POKE IT YOURSELF
promtool check rules alerts.yml — validate Prometheus alerting rules
curl localhost:9090/api/v1/query?query=up — query Prometheus API for target status
Drill this topic →
~90 sec read