Prometheus is an open-source monitoring and alerting system. It collects metrics via a pull model — scraping HTTP endpoints (typically /metrics) on targets at configured intervals. Targets are discovered via service discovery (Kubernetes, Consul, EC2), static configs, or file-based SD. Each scrape stores samples with labels (key-value pairs) and a timestamp. Data is stored locally on disk with a custom time-series database using the TSDB format.
Four core metric types: (1) Counter — cumulative value that only increases (total requests, errors). (2) Gauge — snapshot value that can go up or down (CPU, memory, queue size). (3) Histogram — samples observations in configurable buckets, also provides sum and count of observations (request latency). (4) Summary — similar to Histogram but calculates configurable quantiles on the client side. Each metric has a name and a set of labels identifying its dimensions.
03
Question 3Medium
What is the difference between rate() and increase() in PromQL?
Both calculate per-second rate of change for Counter metrics over a time range. rate(metric[5m]) returns the per-second average rate of increase. increase(metric[5m]) returns the total increase over the time range. Relationship: increase = rate * time_range_seconds. Both handle counter resets (restarts) correctly. Prefer rate() for alerting and dashboards because it produces consistent units (per second), and increase() for understanding total volume in a window.
04
Question 4Medium
How do relabel_configs differ from metric_relabel_configs?
relabel_configs runs before ingestion — modifies target labels (which target to scrape, how to identify it). Used for: filtering targets, adding/replacing instance labels, setting honor_labels. metric_relabel_configs runs after ingestion — modifies individual metric labels within scraped samples. Used for: dropping high-cardinality labels, renaming metrics, aggregating similar metrics. Misusing metric_relabel_configs can silently drop metrics — always test with a debug endpoint first.
05
Question 5Medium
What are recording rules and alerting rules in Prometheus?
Prometheus TSDB stores data in 2-hour blocks on disk under --storage.tsdb.path (default data/). Each block contains chunk files (compressed samples), an index (label → series mapping), and metadata. Retention: --storage.tsdb.retention.time=15d deletes blocks older than 15 days. Compaction merges smaller blocks into larger ones, deduplicates, and applies downsampling (5m resolution after 40 days, 1h after 10d with flags). Write-ahead log (WAL) in data/wal/ ensures durability — on crash, Prometheus replays the WAL. For high ingestion rates, tune --storage.tsdb.retention.size to limit total bytes instead of time.
07
Question 7Hard
How does Thanos extend Prometheus for global querying and long-term storage?
Thanos adds: (1) Thanos Sidecar — ships Prometheus TSDB blocks to object storage (S3, GCS) and serves store API queries. (2) Thanos Store Gateway — serves historical data from object storage. (3) Thanos Query — provides a global Prometheus API endpoint that queries multiple Prometheus instances + Store Gateway simultaneously. (4) Thanos Compactor — deduplicates, downsampling, and retention enforcement on object storage. (5) Thanos Ruler — evaluates recording/alerting rules without scraping. Architecture: each Prometheus runs a sidecar; Thanos Query federates across all of them. Deduplication via --query.replica-label flag handles HA Prometheus pairs.
08
Question 8Hard
How do you handle high cardinality in Prometheus and prevent it from causing performance issues?
High cardinality (millions of unique label combinations) causes high memory usage, slow queries, and OOM kills. Prevention: (1) Use metric_relabel_configs to drop high-churn labels (e.g., pod IDs, request IDs, user IDs). (2) Aggregate before storing — pre-compute with recording rules. (3) Enable --storage.tsdb.max-exemplars carefully. (4) Use --storage.tsdb.sample-count-per-block limits. (5) Monitor cardinality with prometheus_tsdb_series_count. (6) Set --storage.tsdb.retention.time strictly. (7) Use promtool tsdb analyze to identify top label pair cardinality. Remediation: drop offending labels via relabeling or aggregate unwanted series. For user-facing label dimensions, use a separate Prometheus with different retention.
09
Question 9Medium
How does Prometheus service discovery work in Kubernetes?
The Prometheus Operator or kubernetes_sd_configs discovers targets via the Kubernetes API. Four roles: (1) node — discovers all nodes, grabs node-exporter metrics. (2) service — discovers Services, useful for Service-level metrics. (3) pod — discovers all pods, anno-based scrape config. (4) endpoints — discovers Endpoints (the default for kubelet). Relabeling filters targets: __meta_kubernetes_pod_annotation_prometheus_io_scrape = "true". The __metrics_path__ label sets the scrape path. __meta_kubernetes_namespace identifies the namespace.
10
Question 10Medium
How does the Prometheus Operator simplify monitoring in Kubernetes?
Use metrics to establish the affected time window, service and error/latency pattern. Inspect traces to identify which request stage or dependency consumes time, then correlate structured logs with the relevant trace or request identifier. Keep service, environment and release labels consistent across signals.
Prometheus stores metrics; showing a Grafana dashboard does not mean Prometheus also stores logs or traces. Those need suitable backends and instrumentation. Avoid request IDs as Prometheus labels because they create unbounded cardinality. Correlation narrows a hypothesis; verify it before declaring a root cause.
Production API p99 latency jumps from 250 ms to 4 seconds, but CPU and memory stay normal. As the on-call senior DevOps engineer, how do you isolate the bottleneck, restore user experience, and prove the fix?
A retail checkout API slows during peak traffic. Dashboards show 35% CPU, stable RSS, few HTTP 5xx, and green Kubernetes readiness. Customers report spinning checkout pages; the average response time looks much better than p99.
Show answer+
Triage: Confirm user impact by route, region, dependency and deployment version. Compare p50, p95, p99, request rate, error rate and queue depth against the last healthy window. Follow a slow request across the edge/load balancer, ingress, application, cache and database with trace spans and correlation IDs. Check connection-pool wait, thread/worker limits, DB lock waits and slow queries, storage latency, network retransmits, downstream timeouts, garbage-collection pauses and cache misses. A low CPU reading is compatible with hundreds of requests blocked on I/O or queued for a small connection pool.
Containment: Declare and coordinate the incident, identify any recent rollout or configuration change, and make the smallest reversible mitigation consistent with evidence: roll back a bad release, disable an expensive feature, restore a healthy dependency, throttle requests, or expand the actual saturated pool within safe backend limits. Do not blindly scale application pods when the database connection cap or a downstream service is already overwhelmed. Monitor the affected route's tail latency and successful transactions from the customer edge while the mitigation takes effect.
Prevention: Add trace coverage and histograms at meaningful boundaries, track queue time alongside execution time, and make user-journey SLOs part of release gates. Recreate the failure with realistic concurrency and data, document the causal bottleneck and establish a runbook. Recovery is proven when latency and success rate remain healthy under normal traffic, not when CPU stays low.
Trade-offs
Increasing worker/pool concurrency may reduce queue time or merely shift overload to a database. Sampling reduces trace cost but can hide rare tail failures; use targeted capture during incidents.
Common mistake
Concluding that 35% CPU and healthy pods mean the application is fine, or treating average latency as the customer experience.
Follow-up
If p99 climbs while p50 stays flat and database CPU is only 20%, which connection-pool and lock-wait metrics would you inspect first?
Follow the request and record queue + service time
Customer p99, error rate and affected cohort
Edge and ingress timing
Application worker/queue wait
Cache, database and downstream spans
Reversible mitigation and external SLO verification
Check your reasoning
Name two latency bottlenecks that can occur with low CPU usage.
Explain the difference between request queue time and actual execution time.
Describe a user-facing metric that proves recovery.
Prometheus dashboards and Kubernetes health checks are green, but customers in one region say the app is unavailable. How do you prove impact, identify the monitoring blind spot and prevent future false-green incidents?
The dashboard shows up=1 and no pod restarts. Sales users in another country receive TLS or login failures from the public hostname. Internal requests are successful, and the central alert rule has not fired.
Show answer+
Triage: Treat customer reports as incident evidence and reproduce from affected locations, networks and accounts. Check the complete path: public DNS and resolver, CDN/WAF, TLS certificate and handshake, regional load balancing, ingress, authentication and critical backend functionality. Compare edge error rates, access logs, traces, synthetic availability and actual successful transactions by cohort. Determine what the green panel represents: up=1 only means Prometheus scraped a target; a 200 from /health may not prove login or checkout works. Inspect scrape freshness, missing series, label/filter gaps, aggregation that hides a regional outage and alert routing/silences.
Containment: Escalate and mitigate the proven failing boundary: revert a WAF or CDN change, repair the certificate or routing record, fail over a failing region where tested, or disable a broken auth dependency. Verify from the previously affected client vantage point; success from a cluster debug pod is insufficient. Preserve request IDs and exact locations for incident review, and update affected users with measured rather than assumed status.
Prevention: Monitor user-journey SLIs (successful login/transaction, availability, latency) with external synthetic probes and, where appropriate, real-user monitoring. Use Prometheus blackbox probes for HTTP/TLS/DNS reachability and trace actual critical transactions where synthetic checks alone are too shallow. Segment SLO alerts by important region/version/cohort, handle missing telemetry explicitly, and test alerting by deliberately breaking a safe synthetic route. Monitoring is useful only when it measures the failure customers would notice.
Trade-offs
External checks cost money and can generate false alerts when the probe provider fails; use multiple vantage points and distinguish probe failure from a true service outage.
Common mistake
Dismissing real reports because Prometheus up is green, or using one internal readiness endpoint as the entire availability SLO.
Follow-up
A public TLS certificate expires but your internal HTTP health endpoint remains healthy. Which metrics or probes should have alerted?
Turn internal green indicators into customer evidence
Reproduce from affected customer region/account
Trace DNS → CDN/WAF → TLS → edge → auth → backend
Compare actual errors with internal scrape status
Mitigate boundary and re-probe externally
Add segmented user-journey SLIs and alert tests
Check your reasoning
Explain what the up metric does and does not measure.
Name two external synthetic checks that detect failures internal pods cannot.
Describe how a global average can hide a regional outage.
14
Question 14HardIncident command & stakeholder communication
A critical outage is ongoing and leadership demands an exact restoration ETA, but the root cause is still unknown. How do you communicate responsibly, direct the incident and build confidence without promising a fabricated timeline?
Checkout has failed for 18 minutes. Senior leadership asks in a large incident channel: 'When will we be back?' Multiple engineers have competing theories, and customers want immediate updates. A tempting response is to promise a fix in 15 minutes without evidence.
Show answer+
First, state verified facts: which customer actions are affected, known scope, incident start time, current severity and whether a safe mitigation is in progress. Be explicit that restoration time is not yet estimable because the failing component or recovery duration is unconfirmed. Offer a fixed next-update time or cadence, an incident owner and concrete checkpoints such as 'failover underway' or 'external checkout probe recovering'. Do not turn an internal hypothesis into an official root cause or an unsupported ETA. If you have a measured mitigation duration and prerequisites, communicate an appropriately qualified range and its uncertainty.
During response, establish an incident commander, a technical lead and a communications owner as team size warrants. Maintain one incident timeline and a decision log; separate confirmed signals from theories, assign investigation and mitigation owners, freeze unrelated production changes, and escalate to the service/provider owner when needed. Prioritize reversible actions that restore customers over perfect root-cause certainty. Translate technical status into business impact and identify what must become true to declare recovery.
After mitigation, externally verify the previously broken customer journey, monitor for relapse, explain residual risk and schedule the post-incident review. Report actual detection and recovery times with evidence, document the unknowns and close with prevention owners. A senior answer does not invent confidence; it demonstrates reliable communication and decision-making in uncertainty.
Trade-offs
Frequent updates reduce uncertainty for stakeholders but distract responders if the same person handles debugging and communications. Assign roles and use predictable, concise updates.
Common mistake
Saying '15 minutes' without evidence, withholding all information until root cause is known, or sending contradictory updates from multiple engineers.
Follow-up
Two potential fixes have different risks and unknown completion times. What would your next leadership update contain, and who decides whether to deploy a mitigation?
A useful incident communication rhythm
Confirm impact and assign incident roles
State facts, unknowns and immediate mitigation
Commit to next update, not an invented restoration ETA
Track decisions, blockers and customer recovery signals
Communicate verified restoration and follow-up ownership
Check your reasoning
Give an example of a useful status update without an ETA.
Explain why communications and technical response should have separate owners.
State which signals justify saying the incident is resolved.
15
Question 15MediumOpenTelemetry
What does Prometheus 3 change for teams sending OpenTelemetry metrics?
Services emit OTel metrics like http.server.request.duration through a collector that rewrites names.
Show answer+
Prometheus 3 can act as a native OTLP receiver at /api/v1/otlp/v1/metrics once you enable it, so OpenTelemetry SDKs or collectors can push metrics directly. It also allows UTF-8 metric and label names by default, so dotted OTel names can be stored without rewriting dots to underscores.
In PromQL, such names use the quoted syntax, for example {"http.server.request.duration"}. Decide on a translation strategy and which resource attributes become labels, because promoting too many attributes creates cardinality problems.
Trade-offs
Native OTLP ingestion removes a hop, but push-based ingestion means you lose scrape-based up/down signals unless you add them.
Common mistake
Promoting every resource attribute to a label.
Follow-up
How would your existing dashboards cope if metric names stop being rewritten?
16
Question 16MediumHistograms
Why does p99 latency differ between dashboards, and how do you make it trustworthy?
One panel says p99 is 180 ms, another says 450 ms, for the same service.
Show answer+
histogram_quantile interpolates inside bucket boundaries, so with classic histograms the accuracy depends on where the buckets are; a wide bucket around your SLO threshold gives vague numbers. Panels also differ when they aggregate differently: sum the bucket rates by le first, then compute the quantile. Averaging per-instance quantiles or summary quantiles is not valid.
Native histograms use many automatically sized buckets, giving better resolution at lower cost; check that your Prometheus and client versions support scraping them. For SLOs, count requests under the threshold directly rather than relying on an estimated quantile.
Trade-offs
More buckets improve accuracy but add series; native histograms reduce that trade-off.
Common mistake
Averaging p99 values across pods.
Follow-up
How would you set bucket boundaries for a service with a 300 ms SLO?
17
Question 17MediumReliability
How do you know your monitoring itself is working?
During an outage, nobody was paged because Alertmanager had been crash-looping for two days.
Show answer+
Add an always-firing "watchdog" alert routed to an external heartbeat service; if the heartbeat stops, that service pages you, which catches a dead Prometheus or Alertmanager. Alert on scrape failures with up == 0, and on targets that vanish entirely with absent(up{job="api"}).
Watch Prometheus's own metrics: rule evaluation failures, dropped notifications, scrape durations close to the timeout and storage problems. Monitor each Prometheus from a second one, and test the paging path regularly.
Trade-offs
An external heartbeat adds a dependency, but it is the only way to catch a monitoring system that cannot alert about itself.
Common mistake
Relying on Prometheus to alert that Prometheus is down.
Follow-up
How would you test end to end that a page actually reaches the on-call phone?
18
Question 18MediumAlerting
How would you write an SLO burn-rate alert instead of a raw error-rate threshold?
The 1% error alert pages at night for brief blips and missed a slow, steady failure last month.
Show answer+
Burn rate is the observed error rate divided by the rate your SLO allows. For a 99.9% SLO the budget is 0.1%; a burn rate of 14.4 sustained for one hour uses about 2% of a 30-day budget, which is worth a page.
Use multiwindow alerts: require the high burn rate over a long window (1 hour) and a short window (5 minutes), so the page is significant and stops soon after recovery. Add a slower, lower burn-rate alert over longer windows that opens a ticket. Precompute the ratios with recording rules.
Trade-offs
Burn-rate alerts page less and catch slow failures, but they need a clear SLI and enough traffic to be stable.
Common mistake
Using one short window, which reacts to every blip.
Follow-up
How would you handle a low-traffic service where one error is 5% of requests?
A nightly batch job finishes in 30 seconds and is scraped every 60 seconds, so most runs are never seen.
Show answer+
Prometheus normally pulls metrics from a live target. The Pushgateway is for components that cannot be scraped, chiefly short-lived service-level batch jobs that disappear before the next scrape. The job pushes its metrics to the gateway, which Prometheus scrapes instead.\n\nIt should be a narrow exception. Pushed series stay in the gateway until you delete them through its API, so labels identifying a particular machine or run leave stale metrics behind forever. Prometheus also loses its up-based health signal: it can see the last pushed metrics but cannot tell whether the original job succeeded or is still alive. The gateway is a single point of failure and a potential bottleneck.\n\nDo not put an instance label on service-level batch metrics. For a job behind a firewall, put Prometheus on the near side or use PushProx. For machine-level batch results, the Node Exporter textfile collector is a better fit.
Trade-offs
It makes otherwise invisible jobs measurable, at the cost of stale data and no automatic lifecycle management.
Common mistake
Treating the Pushgateway as a general push-based replacement for scraping or as an instance health check.
Follow-up
How would you clean up metrics from batch jobs that change their instance label on every run?
An alert on a 99.9% availability SLO fires for a one-minute blip and keeps paging long after traffic recovers.
Show answer+
Burn rate is how quickly a service consumes its error budget relative to the SLO. A burn rate of 1 means the budget is exactly used up at the end of the SLO window: for 99.9% over 30 days, a constant 0.1% error rate.\n\nA multi-window, multi-burn-rate alert requires the threshold to be exceeded on both a long window and a short one, with the short window commonly one twelfth of the long one. The long window decides whether the burn is significant; the short window makes the alert stop soon after the error stops. For a 99.9% SLO, the SRE workbook suggests paging on 14.4x burn over 1 hour and 5 minutes (about 2% of the budget), paging on 6x over 6 hours and 30 minutes, and ticketing on 1x over 3 days and 6 hours. Alert severity should match response urgency.\n\nLow-traffic services need care: one failed request out of ten is a 10% error rate, which looks like a huge burn but may be meaningless. Longer windows, a lower SLO or synthetic traffic can help.
Trade-offs
Two conditions reduce noise and speed recovery, but add recording-rule and query complexity.
Common mistake
Using a single fixed burn rate for every severity, or paging on a tiny request count.
Follow-up
Why does the short window make a page resolve in minutes rather than an hour?
21
Question 21MediumPromQL
Why does rate() return unexpected values after a counter reset or restart?
A pod restarts and rate(http_requests_total[5m]) briefly shows a large spike rather than a flat line.
Show answer+
rate() computes the per-second average increase of a counter over a range and does account for resets: a drop between samples is treated as a reset and the increase after it is added to the pre-reset remainder. It also extrapolates to the range boundaries to compensate for scrape misalignment or a missed scrape. A too-short range, such as twice the scrape interval, makes that extrapolation volatile and can produce odd spikes after a restart.\n\nThe more common bug is aggregation order. Apply rate() to each individual counter first, then sum() or wrap it in _over_time. Summing counters before applying rate() hides the individual reset, so the drop looks like negative traffic or a reset of the sum.\n\nThere are data-type caveats too. irate() uses only the last two samples and is meant for graphing volatile counters, not SLO alerts. A classic histogram _sum can decrease when observations are negative, which rate() mistakes for a reset; native histograms are rated as a whole and handle resets correctly.
Trade-offs
Longer ranges smooth reset artefacts but delay detection of a real change.
Common mistake
Assuming rate() cannot misbehave after aggregation, or using irate() as the default for alerting.
Follow-up
Why should rate() come before sum() in a PromQL query over many pods?