supraj.dev

Interview questions & answers

Istio

Service mesh, traffic management, security, and observability.

  • 14 questions
  • 2 easy
  • 7 medium
  • 5 hard
  • 1 related article

14 questions

Question 1Easy

What is Istio and what is a service mesh?

#
Show answer

Istio is a service mesh — a dedicated infrastructure layer for handling service-to-service communication. It adds a sidecar Envoy proxy to each pod, intercepting all traffic. Features: traffic routing (canary, blue-green), security (mTLS, auth policies), observability (metrics, traces, logs), and resilience (retries, circuit breakers, fault injection). The control plane (istiod) manages proxy configuration. The data plane (Envoy proxies) handles all traffic.

Question 2Easy

What is a VirtualService in Istio?

#
Show answer

A VirtualService defines routing rules for traffic to a destination. It specifies: (1) which hosts to match (service name, FQDN, wildcard), (2) route rules based on headers, URI, or weight, (3) destination subsets (via DestinationRule subsets), and (4) advanced features like retries, timeouts, and fault injection. Example: route 90% traffic to v1 and 10% to v2 for canary deployments. VirtualServices are the core traffic management primitive.

Question 3Medium

What is a DestinationRule in Istio?

#
Show answer

A DestinationRule defines policies that apply after routing: (1) Subsets — label-based groups of pods (e.g., v1, v2). (2) Traffic policy — connection pool settings, outlier detection (circuit breakers), load balancer algorithm (ROUND_ROBIN, LEAST_REQUEST, RANDOM, PASSTHROUGH). (3) TLS settings — mTLS mode (DISABLE, SIMPLE, MUTUAL, ISTIO_MUTUAL). Example: trafficPolicy: loadBalancer: simple: LEAST_REQUEST; connectionPool: tcp: maxConnections: 100. DestinationRules are always paired with VirtualServices.

Question 4Medium

How does Istio handle mTLS between services?

#
Show answer

Istio can enforce mTLS automatically between all sidecars. Modes: (1) DISABLE — plaintext. (2) ISTIO_MUTUAL — Istio certs (SPIFFE format), auto-rotated every 24h. (3) MUTUAL — custom CA certs. Enable globally: PeerAuthentication: mtls: mode: STRICT. Per-namespace: PeerAuthentication: metadata: namespace: prod. Per-port: portLevelMtls: 8080: mode: STRICT. The Citadel agent in istiod issues certificates via the Kubernetes CSR API. mTLS provides identity-based encryption between services — no app changes needed.

Question 5Medium

How does Istio integrate with Prometheus for observability?

#
Show answer

Istio automatically generates detailed metrics for all service-to-service traffic: request count, duration (latency), request size, response size, TCP byte counts, and gRPC streams. Metrics are enriched with labels: source/destination service, namespace, response code, protocol. Prometheus scrapes Envoy proxies via Istio's telemetry v2 (in-envoy stats). Key dashboards: (1) Istio Service Dashboard — per-service RPS, latency, errors. (2) Istio Workload Dashboard — per-workload metrics. (3) Istio Mesh Dashboard — aggregate mesh health. Istio also supports tracing (Jaeger/Zipkin) and access logging.

Question 6Hard

How does Istio's authorization policy (AuthorizationPolicy) work for access control?

#
Show answer

AuthorizationPolicy provides service-level access control (layer 7). Syntax: spec: selector: matchLabels: app: httpbin; rules: - from: - source: principals: ["cluster.local/ns/default/sa/sleep"]; to: - operation: methods: ["GET"] paths: ["/public/*"]. Actions: ALLOW, DENY, CUSTOM, AUDIT. Authorization policies are evaluated in order: CUSTOM → DENY → ALLOW. If no policy matches, the default action depends on the mesh config — typically ALLOW. For zero-trust: create a DENY-ALL policy first, then explicitly ALLOW needed traffic. Layer 7 awareness means you can restrict by JWT claims, IP blocks, namespaces, and service accounts.

Question 7Hard

How does Istio implement canary deployments and traffic splitting?

#
Show answer

Use VirtualService + DestinationRule: (1) DestinationRule defines subsets: subsets: - name: stable; labels: version: stable; - name: canary; labels: version: canary. (2) VirtualService routes traffic by weight: http: - route: - destination: host: myapp subset: stable weight: 90; - destination: host: myapp subset: canary weight: 10. (3) For HTTP header-based routing: match: - headers: version: exact: canary. (4) Gradual promotion: change weight from 10 → 25 → 50 → 75 → 100. No DNS changes needed. Combine with Flagger or Argo Rollouts for automated progressive delivery based on metrics and analysis.

Question 8Hard

What is the Istio ingress gateway and how does it differ from Kubernetes Ingress?

#
Show answer

The Istio ingress gateway is a dedicated Envoy proxy at the mesh edge — it runs as a separate Deployment (istio-ingressgateway), not as a sidecar. Unlike Kubernetes Ingress (which is a platform-agnostic API), the Istio Gateway allows: (1) Layer 4-7 routing with full Envoy capabilities, (2) TLS termination with mTLS passthrough, (3) VirtualService-based routing with match conditions (headers, URI, methods), (4) weight-based traffic splitting, and (5) integration with the Istio authorization/mTLS model. Define in Gateway CRD: spec: selector: istio: ingressgateway; servers: - port: number: 443; hosts: ["*.example.com"]; tls: mode: SIMPLE. The Gateway + VirtualService replaces the Ingress resource for mesh-aware ingress.

Question 9Hard

How do you handle Istio upgrades in production with zero downtime?

#
Show answer

Strategy: (1) Canary upgrade — deploy new istiod version alongside old: istioctl install --set revision=canary. (2) Relabel namespaces to the new revision: istioctl proxy-status to verify proxies are connected to the correct istiod. (3) Gradually restart sidecars by rolling deployments: kubectl rollout restart deploy -n default. (4) Monitor with istioctl proxy-status and dashboard metrics during transition. (5) After all proxies are on the new revision, remove the old revision. For data plane upgrades: enable ambient mesh for gradual sidecar migration, or use istioctl upgrade (in-place). Key risks: CRD schema changes between versions must be validated first. Test with istioctl experimental precheck.

Question 10Medium

What is the difference between Istio and Linkerd?

#
Show answer

Istio: Envoy-based, more features (auth policies, fault injection, rich telemetry), but higher resource usage and complexity. Linkerd: Rust-based (linkerd2-proxy), lighter (~10MB memory per sidecar vs Istio's ~50MB), simpler to operate, supports TCP and HTTP/gRPC out of the box, but fewer features (no layer 7 auth policies, no fault injection, simpler traffic splitting). Istio is better for enterprise with complex security/traffic needs; Linkerd is better for teams wanting a simple, lightweight mesh with essential features. Both support mTLS, telemetry, and traffic splitting.

Question 11MediumArchitecture

Ambient mode or sidecars — how would you choose for a new Istio mesh?

#

You are rolling out a mesh to 400 services and want mTLS everywhere without restarting every Pod.

Show answer

Ambient mode, generally available since Istio 1.24, moves the proxy out of the Pod. A per-node ztunnel handles L4: mTLS, identity and L4 authorization for every Pod in an enrolled namespace, with no sidecar injection or restarts. Waypoint proxies add L7 features such as HTTP routing and L7 policies only where you deploy them.

That lowers resource use and makes onboarding easier. Sidecars remain a fit where you need per-Pod L7 behaviour or features your ambient version does not support yet. You can start with ambient for mTLS and add waypoints gradually.

Trade-offs

Ambient reduces per-Pod overhead but concentrates L4 traffic in a node-level component you must keep healthy.

Common mistake

Expecting L7 policies to apply in ambient without deploying a waypoint.

Follow-up

Which services would need a waypoint first, and why?

Question 12MediumTraffic management

Should a new Istio setup use the Kubernetes Gateway API or Istio's own Gateway and VirtualService?

#

You are replacing a retired ingress controller and adding Istio at the same time.

Show answer

Istio supports the Kubernetes Gateway API for ingress and for routing inside the mesh, and the project intends it to become the default API for traffic management. Gateway and HTTPRoute are portable across implementations and separate the platform team's gateway from app teams' routes. Ambient mode's waypoint proxies are also configured with Gateway API resources.

For a new setup I would start with Gateway API and use Istio-specific resources only for features it does not cover yet. Existing VirtualService setups keep working, so migrate them gradually.

Trade-offs

Gateway API improves portability, but some advanced Istio features may still need Istio-specific APIs.

Common mistake

Mixing VirtualServices and HTTPRoutes for the same host without understanding which takes effect.

Follow-up

How would you migrate one service from a VirtualService to an HTTPRoute without downtime?

Question 13HardTroubleshooting

Requests through the mesh fail with 503 and flags like UF or UH. How do you debug them?

#

The service works when you call the Pod IP directly, but fails through the mesh.

Show answer

Read the response flags in the Envoy access log. UH means no healthy upstream, for example every endpoint ejected or none registered. UF means the upstream connection failed, often a TLS mode mismatch or a port problem. UO means a circuit breaker limit was hit; NR means no route matched.

Then inspect the proxy: istioctl proxy-config clusters, endpoints and routes for the client Pod, and istioctl analyze for configuration mistakes. Check DestinationRule TLS settings, outlier detection ejections, Service port names or appProtocol, and whether the destination is in the mesh.

Trade-offs

Access logs at full volume are expensive; enable them for the affected workload while debugging.

Common mistake

Restarting Pods before reading the response flags.

Follow-up

What would you check if only requests above a certain size fail?

Question 14MediumResilience

How do you stop one slow dependency from exhausting a whole service?

#

A recommendations API slows to 20 seconds and the checkout service runs out of connections.

Show answer

Set timeouts and bounded retries in the VirtualService so callers do not wait forever or retry a struggling service into the ground. In the DestinationRule, set connection pool limits so a slow dependency can hold only a fixed number of connections and pending requests; extra requests fail fast.

Add outlier detection to eject endpoints that return errors repeatedly. On the client, decide on a fallback such as returning checkout without recommendations, and alert on circuit-breaker overflow counts.

Trade-offs

Tight limits protect callers but reject requests during legitimate spikes; size them from real traffic.

Common mistake

Adding retries without timeouts, which multiplies load on the slow service.

Follow-up

How would you test these limits before a real incident?