supraj.dev

Interview questions & answers

Prometheus

Metrics collection, PromQL, alerting, and observability.

  • 21 questions
  • 2 easy
  • 12 medium
  • 7 hard
  • 4 related articles

21 questions

Question 1Easy

What is Prometheus and how does it collect metrics?

#
Show answer

Prometheus is an open-source monitoring and alerting system. It collects metrics via a pull model — scraping HTTP endpoints (typically /metrics) on targets at configured intervals. Targets are discovered via service discovery (Kubernetes, Consul, EC2), static configs, or file-based SD. Each scrape stores samples with labels (key-value pairs) and a timestamp. Data is stored locally on disk with a custom time-series database using the TSDB format.

Question 2Easy

What metric types does Prometheus support?

#
Show answer

Four core metric types: (1) Counter — cumulative value that only increases (total requests, errors). (2) Gauge — snapshot value that can go up or down (CPU, memory, queue size). (3) Histogram — samples observations in configurable buckets, also provides sum and count of observations (request latency). (4) Summary — similar to Histogram but calculates configurable quantiles on the client side. Each metric has a name and a set of labels identifying its dimensions.

Question 3Medium

What is the difference between rate() and increase() in PromQL?

#
Show answer

Both calculate per-second rate of change for Counter metrics over a time range. rate(metric[5m]) returns the per-second average rate of increase. increase(metric[5m]) returns the total increase over the time range. Relationship: increase = rate * time_range_seconds. Both handle counter resets (restarts) correctly. Prefer rate() for alerting and dashboards because it produces consistent units (per second), and increase() for understanding total volume in a window.

Question 4Medium

How do relabel_configs differ from metric_relabel_configs?

#
Show answer

relabel_configs runs before ingestion — modifies target labels (which target to scrape, how to identify it). Used for: filtering targets, adding/replacing instance labels, setting honor_labels. metric_relabel_configs runs after ingestion — modifies individual metric labels within scraped samples. Used for: dropping high-cardinality labels, renaming metrics, aggregating similar metrics. Misusing metric_relabel_configs can silently drop metrics — always test with a debug endpoint first.

Question 5Medium

What are recording rules and alerting rules in Prometheus?

#
Show answer

Recording rules precompute frequently-used or expensive PromQL expressions and store them as new time series. Syntax: groups: [- name: node_rules, rules: [record: job:node_memory_util:avg, expr: avg(node_memory_MemTotal_bytes - node_memory_MemFree_bytes) / node_memory_MemTotal_bytes]]. Alerting rules define conditions that trigger alerts: alert: HighMemoryUsage, expr: job:node_memory_util:avg > 0.8, for: 5m, labels: {severity: warning}, annotations: {summary: "Memory usage above 80%"}. The for clause prevents flapping — alert fires only after sustained violation.

Question 6Hard

How does Prometheus handle data storage, retention, and compaction?

#
Show answer

Prometheus TSDB stores data in 2-hour blocks on disk under --storage.tsdb.path (default data/). Each block contains chunk files (compressed samples), an index (label → series mapping), and metadata. Retention: --storage.tsdb.retention.time=15d deletes blocks older than 15 days. Compaction merges smaller blocks into larger ones, deduplicates, and applies downsampling (5m resolution after 40 days, 1h after 10d with flags). Write-ahead log (WAL) in data/wal/ ensures durability — on crash, Prometheus replays the WAL. For high ingestion rates, tune --storage.tsdb.retention.size to limit total bytes instead of time.

Question 7Hard

How does Thanos extend Prometheus for global querying and long-term storage?

#
Show answer

Thanos adds: (1) Thanos Sidecar — ships Prometheus TSDB blocks to object storage (S3, GCS) and serves store API queries. (2) Thanos Store Gateway — serves historical data from object storage. (3) Thanos Query — provides a global Prometheus API endpoint that queries multiple Prometheus instances + Store Gateway simultaneously. (4) Thanos Compactor — deduplicates, downsampling, and retention enforcement on object storage. (5) Thanos Ruler — evaluates recording/alerting rules without scraping. Architecture: each Prometheus runs a sidecar; Thanos Query federates across all of them. Deduplication via --query.replica-label flag handles HA Prometheus pairs.

Question 8Hard

How do you handle high cardinality in Prometheus and prevent it from causing performance issues?

#
Show answer

High cardinality (millions of unique label combinations) causes high memory usage, slow queries, and OOM kills. Prevention: (1) Use metric_relabel_configs to drop high-churn labels (e.g., pod IDs, request IDs, user IDs). (2) Aggregate before storing — pre-compute with recording rules. (3) Enable --storage.tsdb.max-exemplars carefully. (4) Use --storage.tsdb.sample-count-per-block limits. (5) Monitor cardinality with prometheus_tsdb_series_count. (6) Set --storage.tsdb.retention.time strictly. (7) Use promtool tsdb analyze to identify top label pair cardinality. Remediation: drop offending labels via relabeling or aggregate unwanted series. For user-facing label dimensions, use a separate Prometheus with different retention.

Question 9Medium

How does Prometheus service discovery work in Kubernetes?

#
Show answer

The Prometheus Operator or kubernetes_sd_configs discovers targets via the Kubernetes API. Four roles: (1) node — discovers all nodes, grabs node-exporter metrics. (2) service — discovers Services, useful for Service-level metrics. (3) pod — discovers all pods, anno-based scrape config. (4) endpoints — discovers Endpoints (the default for kubelet). Relabeling filters targets: __meta_kubernetes_pod_annotation_prometheus_io_scrape = "true". The __metrics_path__ label sets the scrape path. __meta_kubernetes_namespace identifies the namespace.

Question 10Medium

How does the Prometheus Operator simplify monitoring in Kubernetes?

#
Show answer

The Prometheus Operator introduces CRDs: Prometheus (deploy/manage Prometheus instances), ServiceMonitor (define scrape targets declaratively), PodMonitor (scrape pods directly), PrometheusRule (alerting and recording rules), Alertmanager (manage Alertmanager instances). Benefits: (1) Auto-discovers targets via label selectors. (2) Manages Prometheus upgrades, scaling, and configuration. (3) Provides kube-prometheus-stack (kube-prometheus) — a complete monitoring stack with Grafana dashboards, node-exporter, kube-state-metrics, and default alerting rules.

Question 11MediumIncident diagnosis

How would you combine metrics, logs and traces during a latency incident?

#
Show answer

Use metrics to establish the affected time window, service and error/latency pattern. Inspect traces to identify which request stage or dependency consumes time, then correlate structured logs with the relevant trace or request identifier. Keep service, environment and release labels consistent across signals.

Prometheus stores metrics; showing a Grafana dashboard does not mean Prometheus also stores logs or traces. Those need suitable backends and instrumentation. Avoid request IDs as Prometheus labels because they create unbounded cardinality. Correlation narrows a hypothesis; verify it before declaring a root cause.

Question 12HardProduction latency & bottleneck isolation

Production API p99 latency jumps from 250 ms to 4 seconds, but CPU and memory stay normal. As the on-call senior DevOps engineer, how do you isolate the bottleneck, restore user experience, and prove the fix?

#

A retail checkout API slows during peak traffic. Dashboards show 35% CPU, stable RSS, few HTTP 5xx, and green Kubernetes readiness. Customers report spinning checkout pages; the average response time looks much better than p99.

Show answer

Triage: Confirm user impact by route, region, dependency and deployment version. Compare p50, p95, p99, request rate, error rate and queue depth against the last healthy window. Follow a slow request across the edge/load balancer, ingress, application, cache and database with trace spans and correlation IDs. Check connection-pool wait, thread/worker limits, DB lock waits and slow queries, storage latency, network retransmits, downstream timeouts, garbage-collection pauses and cache misses. A low CPU reading is compatible with hundreds of requests blocked on I/O or queued for a small connection pool.

Containment: Declare and coordinate the incident, identify any recent rollout or configuration change, and make the smallest reversible mitigation consistent with evidence: roll back a bad release, disable an expensive feature, restore a healthy dependency, throttle requests, or expand the actual saturated pool within safe backend limits. Do not blindly scale application pods when the database connection cap or a downstream service is already overwhelmed. Monitor the affected route's tail latency and successful transactions from the customer edge while the mitigation takes effect.

Prevention: Add trace coverage and histograms at meaningful boundaries, track queue time alongside execution time, and make user-journey SLOs part of release gates. Recreate the failure with realistic concurrency and data, document the causal bottleneck and establish a runbook. Recovery is proven when latency and success rate remain healthy under normal traffic, not when CPU stays low.

Trade-offs

Increasing worker/pool concurrency may reduce queue time or merely shift overload to a database. Sampling reduces trace cost but can hide rare tail failures; use targeted capture during incidents.

Common mistake

Concluding that 35% CPU and healthy pods mean the application is fine, or treating average latency as the customer experience.

Follow-up

If p99 climbs while p50 stays flat and database CPU is only 20%, which connection-pool and lock-wait metrics would you inspect first?

Follow the request and record queue + service time
  1. Customer p99, error rate and affected cohort
  2. Edge and ingress timing
  3. Application worker/queue wait
  4. Cache, database and downstream spans
  5. Reversible mitigation and external SLO verification

Check your reasoning

  • Name two latency bottlenecks that can occur with low CPU usage.
  • Explain the difference between request queue time and actual execution time.
  • Describe a user-facing metric that proves recovery.
Question 13HardSLOs, blind spots & external monitoring

Prometheus dashboards and Kubernetes health checks are green, but customers in one region say the app is unavailable. How do you prove impact, identify the monitoring blind spot and prevent future false-green incidents?

#

The dashboard shows up=1 and no pod restarts. Sales users in another country receive TLS or login failures from the public hostname. Internal requests are successful, and the central alert rule has not fired.

Show answer

Triage: Treat customer reports as incident evidence and reproduce from affected locations, networks and accounts. Check the complete path: public DNS and resolver, CDN/WAF, TLS certificate and handshake, regional load balancing, ingress, authentication and critical backend functionality. Compare edge error rates, access logs, traces, synthetic availability and actual successful transactions by cohort. Determine what the green panel represents: up=1 only means Prometheus scraped a target; a 200 from /health may not prove login or checkout works. Inspect scrape freshness, missing series, label/filter gaps, aggregation that hides a regional outage and alert routing/silences.

Containment: Escalate and mitigate the proven failing boundary: revert a WAF or CDN change, repair the certificate or routing record, fail over a failing region where tested, or disable a broken auth dependency. Verify from the previously affected client vantage point; success from a cluster debug pod is insufficient. Preserve request IDs and exact locations for incident review, and update affected users with measured rather than assumed status.

Prevention: Monitor user-journey SLIs (successful login/transaction, availability, latency) with external synthetic probes and, where appropriate, real-user monitoring. Use Prometheus blackbox probes for HTTP/TLS/DNS reachability and trace actual critical transactions where synthetic checks alone are too shallow. Segment SLO alerts by important region/version/cohort, handle missing telemetry explicitly, and test alerting by deliberately breaking a safe synthetic route. Monitoring is useful only when it measures the failure customers would notice.

Trade-offs

External checks cost money and can generate false alerts when the probe provider fails; use multiple vantage points and distinguish probe failure from a true service outage.

Common mistake

Dismissing real reports because Prometheus up is green, or using one internal readiness endpoint as the entire availability SLO.

Follow-up

A public TLS certificate expires but your internal HTTP health endpoint remains healthy. Which metrics or probes should have alerted?

Turn internal green indicators into customer evidence
  1. Reproduce from affected customer region/account
  2. Trace DNS → CDN/WAF → TLS → edge → auth → backend
  3. Compare actual errors with internal scrape status
  4. Mitigate boundary and re-probe externally
  5. Add segmented user-journey SLIs and alert tests

Check your reasoning

  • Explain what the up metric does and does not measure.
  • Name two external synthetic checks that detect failures internal pods cannot.
  • Describe how a global average can hide a regional outage.
Question 14HardIncident command & stakeholder communication

A critical outage is ongoing and leadership demands an exact restoration ETA, but the root cause is still unknown. How do you communicate responsibly, direct the incident and build confidence without promising a fabricated timeline?

#

Checkout has failed for 18 minutes. Senior leadership asks in a large incident channel: 'When will we be back?' Multiple engineers have competing theories, and customers want immediate updates. A tempting response is to promise a fix in 15 minutes without evidence.

Show answer

First, state verified facts: which customer actions are affected, known scope, incident start time, current severity and whether a safe mitigation is in progress. Be explicit that restoration time is not yet estimable because the failing component or recovery duration is unconfirmed. Offer a fixed next-update time or cadence, an incident owner and concrete checkpoints such as 'failover underway' or 'external checkout probe recovering'. Do not turn an internal hypothesis into an official root cause or an unsupported ETA. If you have a measured mitigation duration and prerequisites, communicate an appropriately qualified range and its uncertainty.

During response, establish an incident commander, a technical lead and a communications owner as team size warrants. Maintain one incident timeline and a decision log; separate confirmed signals from theories, assign investigation and mitigation owners, freeze unrelated production changes, and escalate to the service/provider owner when needed. Prioritize reversible actions that restore customers over perfect root-cause certainty. Translate technical status into business impact and identify what must become true to declare recovery.

After mitigation, externally verify the previously broken customer journey, monitor for relapse, explain residual risk and schedule the post-incident review. Report actual detection and recovery times with evidence, document the unknowns and close with prevention owners. A senior answer does not invent confidence; it demonstrates reliable communication and decision-making in uncertainty.

Trade-offs

Frequent updates reduce uncertainty for stakeholders but distract responders if the same person handles debugging and communications. Assign roles and use predictable, concise updates.

Common mistake

Saying '15 minutes' without evidence, withholding all information until root cause is known, or sending contradictory updates from multiple engineers.

Follow-up

Two potential fixes have different risks and unknown completion times. What would your next leadership update contain, and who decides whether to deploy a mitigation?

A useful incident communication rhythm
  1. Confirm impact and assign incident roles
  2. State facts, unknowns and immediate mitigation
  3. Commit to next update, not an invented restoration ETA
  4. Track decisions, blockers and customer recovery signals
  5. Communicate verified restoration and follow-up ownership

Check your reasoning

  • Give an example of a useful status update without an ETA.
  • Explain why communications and technical response should have separate owners.
  • State which signals justify saying the incident is resolved.
Question 15MediumOpenTelemetry

What does Prometheus 3 change for teams sending OpenTelemetry metrics?

#

Services emit OTel metrics like http.server.request.duration through a collector that rewrites names.

Show answer

Prometheus 3 can act as a native OTLP receiver at /api/v1/otlp/v1/metrics once you enable it, so OpenTelemetry SDKs or collectors can push metrics directly. It also allows UTF-8 metric and label names by default, so dotted OTel names can be stored without rewriting dots to underscores.

In PromQL, such names use the quoted syntax, for example {"http.server.request.duration"}. Decide on a translation strategy and which resource attributes become labels, because promoting too many attributes creates cardinality problems.

Trade-offs

Native OTLP ingestion removes a hop, but push-based ingestion means you lose scrape-based up/down signals unless you add them.

Common mistake

Promoting every resource attribute to a label.

Follow-up

How would your existing dashboards cope if metric names stop being rewritten?

Question 16MediumHistograms

Why does p99 latency differ between dashboards, and how do you make it trustworthy?

#

One panel says p99 is 180 ms, another says 450 ms, for the same service.

Show answer

histogram_quantile interpolates inside bucket boundaries, so with classic histograms the accuracy depends on where the buckets are; a wide bucket around your SLO threshold gives vague numbers. Panels also differ when they aggregate differently: sum the bucket rates by le first, then compute the quantile. Averaging per-instance quantiles or summary quantiles is not valid.

Native histograms use many automatically sized buckets, giving better resolution at lower cost; check that your Prometheus and client versions support scraping them. For SLOs, count requests under the threshold directly rather than relying on an estimated quantile.

Trade-offs

More buckets improve accuracy but add series; native histograms reduce that trade-off.

Common mistake

Averaging p99 values across pods.

Follow-up

How would you set bucket boundaries for a service with a 300 ms SLO?

Question 17MediumReliability

How do you know your monitoring itself is working?

#

During an outage, nobody was paged because Alertmanager had been crash-looping for two days.

Show answer

Add an always-firing "watchdog" alert routed to an external heartbeat service; if the heartbeat stops, that service pages you, which catches a dead Prometheus or Alertmanager. Alert on scrape failures with up == 0, and on targets that vanish entirely with absent(up{job="api"}).

Watch Prometheus's own metrics: rule evaluation failures, dropped notifications, scrape durations close to the timeout and storage problems. Monitor each Prometheus from a second one, and test the paging path regularly.

Trade-offs

An external heartbeat adds a dependency, but it is the only way to catch a monitoring system that cannot alert about itself.

Common mistake

Relying on Prometheus to alert that Prometheus is down.

Follow-up

How would you test end to end that a page actually reaches the on-call phone?

Question 18MediumAlerting

How would you write an SLO burn-rate alert instead of a raw error-rate threshold?

#

The 1% error alert pages at night for brief blips and missed a slow, steady failure last month.

Show answer

Burn rate is the observed error rate divided by the rate your SLO allows. For a 99.9% SLO the budget is 0.1%; a burn rate of 14.4 sustained for one hour uses about 2% of a 30-day budget, which is worth a page.

Use multiwindow alerts: require the high burn rate over a long window (1 hour) and a short window (5 minutes), so the page is significant and stops soon after recovery. Add a slower, lower burn-rate alert over longer windows that opens a ticket. Precompute the ratios with recording rules.

Trade-offs

Burn-rate alerts page less and catch slow failures, but they need a clear SLI and enough traffic to be stable.

Common mistake

Using one short window, which reacts to every blip.

Follow-up

How would you handle a low-traffic service where one error is 5% of requests?

Fast-burn condition for a 99.9% SLO · promql
(
  sum(rate(http_requests_total{job="api",code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m]))
) > (14.4 * 0.001)
Question 19MediumInstrumentation

When do you use the Prometheus Pushgateway and what are its pitfalls?

#

A nightly batch job finishes in 30 seconds and is scraped every 60 seconds, so most runs are never seen.

Show answer

Prometheus normally pulls metrics from a live target. The Pushgateway is for components that cannot be scraped, chiefly short-lived service-level batch jobs that disappear before the next scrape. The job pushes its metrics to the gateway, which Prometheus scrapes instead.\n\nIt should be a narrow exception. Pushed series stay in the gateway until you delete them through its API, so labels identifying a particular machine or run leave stale metrics behind forever. Prometheus also loses its up-based health signal: it can see the last pushed metrics but cannot tell whether the original job succeeded or is still alive. The gateway is a single point of failure and a potential bottleneck.\n\nDo not put an instance label on service-level batch metrics. For a job behind a firewall, put Prometheus on the near side or use PushProx. For machine-level batch results, the Node Exporter textfile collector is a better fit.

Trade-offs

It makes otherwise invisible jobs measurable, at the cost of stale data and no automatic lifecycle management.

Common mistake

Treating the Pushgateway as a general push-based replacement for scraping or as an instance health check.

Follow-up

How would you clean up metrics from batch jobs that change their instance label on every run?

Question 20HardAlerting

What are multi-window, multi-burn-rate alerts?

#

An alert on a 99.9% availability SLO fires for a one-minute blip and keeps paging long after traffic recovers.

Show answer

Burn rate is how quickly a service consumes its error budget relative to the SLO. A burn rate of 1 means the budget is exactly used up at the end of the SLO window: for 99.9% over 30 days, a constant 0.1% error rate.\n\nA multi-window, multi-burn-rate alert requires the threshold to be exceeded on both a long window and a short one, with the short window commonly one twelfth of the long one. The long window decides whether the burn is significant; the short window makes the alert stop soon after the error stops. For a 99.9% SLO, the SRE workbook suggests paging on 14.4x burn over 1 hour and 5 minutes (about 2% of the budget), paging on 6x over 6 hours and 30 minutes, and ticketing on 1x over 3 days and 6 hours. Alert severity should match response urgency.\n\nLow-traffic services need care: one failed request out of ten is a 10% error rate, which looks like a huge burn but may be meaningless. Longer windows, a lower SLO or synthetic traffic can help.

Trade-offs

Two conditions reduce noise and speed recovery, but add recording-rule and query complexity.

Common mistake

Using a single fixed burn rate for every severity, or paging on a tiny request count.

Follow-up

Why does the short window make a page resolve in minutes rather than an hour?

Question 21MediumPromQL

Why does rate() return unexpected values after a counter reset or restart?

#

A pod restarts and rate(http_requests_total[5m]) briefly shows a large spike rather than a flat line.

Show answer

rate() computes the per-second average increase of a counter over a range and does account for resets: a drop between samples is treated as a reset and the increase after it is added to the pre-reset remainder. It also extrapolates to the range boundaries to compensate for scrape misalignment or a missed scrape. A too-short range, such as twice the scrape interval, makes that extrapolation volatile and can produce odd spikes after a restart.\n\nThe more common bug is aggregation order. Apply rate() to each individual counter first, then sum() or wrap it in _over_time. Summing counters before applying rate() hides the individual reset, so the drop looks like negative traffic or a reset of the sum.\n\nThere are data-type caveats too. irate() uses only the last two samples and is meant for graphing volatile counters, not SLO alerts. A classic histogram _sum can decrease when observations are negative, which rate() mistakes for a reset; native histograms are rated as a whole and handle resets correctly.

Trade-offs

Longer ranges smooth reset artefacts but delay detection of a real change.

Common mistake

Assuming rate() cannot misbehave after aggregation, or using irate() as the default for alerting.

Follow-up

Why should rate() come before sum() in a PromQL query over many pods?