Helm
Think it through.

helm install fails if a release already exists; helm upgrade --install is idempotent — it installs on first run and upgrades on subsequent runs. In CI/CD pipelines you always prefer helm upgrade --install because the pipeline doesn't know whether this is the first deploy or a re-deploy. Pair it with --atomic (rolls back automatically on failure) and --timeout 5m to get safe, self-healing deploys. Example: helm upgrade --install my-app ./chart -f values-prod.yaml --atomic --timeout 5m --namespace prod --create-namespace.

helm template renders chart manifests locally without contacting the Kubernetes API server, so it never applies anything. It's invaluable for: (1) diffing rendered YAML before a deploy (helm template . -f values.yaml | kubectl diff -f -); (2) catching template syntax errors in CI without a live cluster; (3) feeding rendered manifests into policy engines like OPA/Conftest. Flags to know: --debug prints the full template context on error; --set key=val overrides values inline; --show-only templates/deployment.yaml filters a single file.

Parent charts override subchart values using the subchart's alias or name as a top-level key in the parent values.yaml. If your subchart is named redis in Chart.yaml dependencies, overrides must live under a redis: key. Common mistakes: (1) wrong key name — check helm dependency list for exact alias; (2) subchart has global: values that take precedence; (3) key hierarchy mismatch. Debug with helm template . --debug 2>&1 | grep -A5 redis to see merged values. Fix: run helm show values ./charts/redis to see the subchart's full schema, then mirror the key hierarchy exactly in your parent values file.

lookup queries the live Kubernetes API during render: {{ (lookup "v1" "Secret" "default" "my-secret").data.password | b64dec }}. Security implications: (1) Secret leakage in CI — any runner with helm upgrade permissions can exfiltrate any secret the service account can read; (2) Non-hermetic renders — helm template behaves differently with/without cluster access, breaking GitOps diff tools; (3) Race conditions — the looked-up resource may change between render and apply; (4) Permission escalation — the Helm SA needs read access to arbitrary resources, widening blast radius. Best practice: avoid lookup for secrets — use External Secrets Operator or Vault Agent injection instead.

This happens when a previous helm upgrade process was killed mid-operation (OOM, SIGKILL, CI timeout) leaving a lock in the release's Kubernetes Secret. Recovery: (1) helm history my-release -n my-ns — identify the last successful revision. (2) kubectl get secrets -n my-ns -l owner=helm,name=my-release — find the sh.helm.release.v1.my-release.vN secret in pending-upgrade status. (3) Force rollback: helm rollback my-release <last-good-revision> -n my-ns --force. If rollback also hangs: (4) Delete the pending secret directly: kubectl delete secret sh.helm.release.v1.my-release.vN -n my-ns. Prevention: always set --timeout and --atomic so Helm auto-rolls back and clears the lock on failure.

Architecture: maintain two Deployments (app-blue, app-green) and a single Service. The active color is stored in a ConfigMap. Hook sequence: (1) pre-upgrade hook — a Job reads the ConfigMap, determines inactive color, scales up the inactive Deployment to full replica count, waits for rollout status, exits 0. (2) Main chart templates update the inactive Deployment's image tag. (3) post-upgrade hook — a Job runs smoke tests, then patches the Service selector (kubectl patch svc app -p '{"spec":{"selector":{"color":"green"}}}'), finally updates the ConfigMap. Hook annotations: helm.sh/hook: pre-upgrade, helm.sh/hook-weight: "-5", helm.sh/hook-delete-policy: before-hook-creation,hook-failed. Failure recovery: if pre-upgrade fails, Helm aborts before touching the active color — zero impact. If post-upgrade fails after selector switch, operator must manually revert the Service selector. Prevention: set helm.sh/hook-delete-policy: hook-failed to keep failed Job pods for forensics.

helm upgrade lifecycle: (1) Fetches last deployed release from the Kubernetes Secret store (base64+gzip+base64 encoded). (2) Renders new templates with updated values. (3) Three-way strategic merge: computes a patch using (A) last deployed manifest, (B) current live state from API server, (C) desired new manifest. This preserves out-of-band changes (e.g., HPA-modified replica counts) while applying yours. Out-of-band problems: (a) If an operator manually patched resource limits, the three-way merge preserves those — but if the chart also updates resource limits, Helm's desired state wins, silently reverting the manual patch. (b) CRDs modified outside Helm cause unexpected diffs. (c) Immutable fields (e.g., Service clusterIP) conflict with chart changes — Helm errors: "field is immutable". Fix: helm upgrade --force (deletes/recreates) or use helm.sh/resource-policy: keep annotation to exclude the resource.

A post-renderer is an executable that Helm pipes rendered YAML to on stdin and reads modified YAML from stdout. Implementation: (1) Create a wrapper script (post-render.sh): #!/bin/bash; cat > /tmp/helm-output.yaml; kustomize build /tmp/kustomize-overlay. (2) Kustomize overlay at /tmp/kustomize-overlay/kustomization.yaml: resources: [/tmp/helm-output.yaml]; patches: [{path: add-dd-sidecar.yaml, target: {kind: Deployment}}, {path: add-dd-sidecar.yaml, target: {kind: StatefulSet}}]. (3) Sidecar patch (add-dd-sidecar.yaml): a strategic merge patch adding the Datadog container with DD_AGENT_HOST: $(HOST_IP) via fieldRef. (4) Invoke: helm upgrade --install my-release third-party/chart -f values.yaml --post-renderer ./post-render.sh. Caveats: kustomize strategic merge patches for sidecar injection must match the container name to avoid duplicates on re-runs; test with helm template ... --post-renderer ./post-render.sh | kubectl diff -f - before live apply.

The stable/redis chart repository was deprecated and removed from the stable Helm repo in late 2023. This is a common breakage as the old stable and incubator repos are being phased out. Diagnosis: helm repo list to see if the repo is still configured; helm search repo redis to check availability. Fixes: (1) Migrate to the new Bitnami repo: helm repo add bitnami https://charts.bitnami.com/bitnami; helm repo update. (2) Update Chart.yaml to use bitnami/redis instead of stable/redis. Better approach: use OCI-based registries — dependencies: [name: redis, version: "~10.5", repository: "oci://registry-1.docker.io/bitnamicharts"]. (3) Run helm dependency update to download the new chart. For existing deployments, helm upgrade --install with --reuse-values may fail — force re-render with --reset-values. Prevent recurrence: use helm repo add with explicit URL, pin subchart versions, and use OCI registries for immutability and no repo-dependency.

Directory structure: charts/microservice/ (shared chart), services/service-a/values.yaml, services/service-b/values.yaml, etc. CI/CD approach: (1) Use a single pipeline with a matrix: strategy: matrix: service: [service-a, service-b, ...]. Each matrix entry runs helm upgrade --install ${{ matrix.service }} ./charts/microservice -f services/${{ matrix.service }}/values.yaml. (2) Use Helmfile for multi-release management: helmfile.yaml with releases: [name: service-a, chart: ./charts/microservice, values: [./services/service-a/values.yaml], ...]. Helmfile supports environments, templates, and hooks. (3) For version bumps across all 20: update the chart version once, then run helmfile apply — Helmfile detects which releases have changed (by diffing the rendered output against the deployed release). (4) For per-service image tags: pass --set image.tag=${{ matrix.service }}-${{ github.sha }}. (5) Use a values schema (values.schema.json) to validate per-service values at CI time. (6) For secrets: use external-secrets or SealedSecrets per namespace. Managing 20 services individually without a shared chart leads to massive config drift — always share the base chart and layer per-service values on top.

CRDs are handled specially: Helm installs CRDs from the crds/ directory before rendering templates, but does NOT manage CRDs after install — helm upgrade never touches them, and helm uninstall does NOT delete CRDs or their instances. This means CRD instances (custom resources) survive chart uninstallation, leaving orphaned objects. Safely managing CRDs: (1) For CRDs shared across charts (e.g., cert-manager, istio), install them as a separate chart or kubectl apply — never bundle them with your application chart. (2) For CRDs specific to one chart, keep them in crds/ but document manual cleanup steps. (3) Use helm.sh/resource-policy: keep on CRDs in templates (not crds/ — those are hooks) to prevent deletion on uninstall. (4) For lifecycle management, use Helm's pre-delete hook to clean up CRD instances: a Job that runs kubectl delete crd myresources.example.com --ignore-not-found, but this is risky. (5) Alternative: use a dedicated tool (Kapp, ArgoCD with sync wave -20) to manage CRDs independently of Helm releases. The Helm docs recommend: "CRDs should be managed by a separate process outside of the Helm release lifecycle."

A Helm chart contains: Chart.yaml (metadata), values.yaml (default config values), templates/ (Go template YAML files), charts/ (subchart dependencies), crds/ (CustomResourceDefinitions), README.md, and optionally templates/tests/ for test pods. helm create <name> scaffolds this structure.

Chart.yaml is the metadata file for a Helm chart. Required fields: apiVersion (v2 for Helm 3+), name, version (chart version, semver 2). Optional but common: description, type (application or library), dependencies, appVersion (version of the application packaged), keywords, maintainers, icon, kubeVersion, and annotations.

helm search hub <keyword> searches the Helm Hub (aggregated repos). helm search repo <keyword> searches repos you've added locally. helm repo add <name> <url> adds a repo, then helm repo update refreshes the cache. helm search repo bitnami/nginx --versions lists all versions of a specific chart.

Files prefixed with _ (e.g., _helpers.tpl) are not rendered as Kubernetes resources but are available as named templates. Common uses: generating resource names with {{ include "myapp.fullname" . }}, defining labels/selectors, and reusable snippets. They are defined with {{- define "name" -}}...{{- end -}} and invoked with {{ include "name" . }} or {{ template "name" . }}. Include is preferred because it pipelines output.

Three ways: (1) --values my-values.yaml (or -f) — merges with values.yaml. (2) --set key=value — inline overrides for simple values. (3) --set-string key=value — force string typing. --set-file key=./path reads a file as the value. Precedence (highest to lowest): --set, -f, values.yaml defaults.

helm create <chart-name> scaffolds a complete chart with example templates: a Deployment, Service, Ingress, HPA, ServiceAccount, and tests. It also generates a values.yaml with documented defaults, _helpers.tpl, and Chart.yaml. Edit these to match your application. Delete unneeded templates — you don't have to use them all.

Named templates (partials) are defined with {{ define "my-chart.labels" }}...{{ end }} in _helpers.tpl. They are reusable across templates. Two invocation syntaxes: {{ include "my-chart.labels" . }} (preferred — pipelines output, can be used in toYaml), and {{ template "my-chart.labels" . }} (older — outputs inline, cannot pipe). The dot (.) passes the full scope context.

{{ required "A valid .Values.foo is required!" .Values.foo }} validates that a value is present at render time. If the value is empty or missing, template rendering fails with the given error message. Use it for mandatory configuration that has no sensible default — like API keys, ingress hostnames, or environment-specific settings. This catches misconfiguration early.

Dependencies are declared in Chart.yaml under dependencies: with name, version (range like ~1.2, >=1.0), repository (URL or @repo-alias), and optional condition (enable/disable based on a value) and tags. Run helm dependency update to download them into charts/. For OCI registries, use repository: oci://registry.example.com/repo. Dependencies can be in a subcharts/ directory locally as well.

helm lint <chart-dir> validates a chart for common issues: malformed YAML, missing required fields in Chart.yaml, invalid template syntax, duplicate resource names, and Kubernetes API compatibility (if kubeVersion is set). It does NOT connect to a cluster. Run helm lint in CI before packaging to catch errors early. --strict exits with non-zero on warnings too.

helm rollback <release> <revision> reverts to the specified revision. helm rollback <release> 0 rolls back to the previous revision (Helm 3 shorthand). --wait waits for all resources to become ready. --timeout sets how long to wait. --force forces recreation of resources (deletes then recreates). Check revision history with helm history <release> and helm list --revisions.

Helm deletes all resources that it created during install (tracked via the release secret in the namespace), in reverse dependency order. It does NOT delete CRDs or their instances. Resources annotated with helm.sh/resource-policy: keep are preserved. Namespaces created by --create-namespace are NOT deleted. Release history (secrets) can be kept with --keep-history.

Each release revision is stored as a Kubernetes Secret in the release's namespace, labeled with owner: helm, name: <release>. By default, Helm keeps the last 10 revisions. Override with --history-max on upgrade: helm upgrade --install --history-max 20. You can configure the default in Helm's environment variable HELM_MAX_HISTORY. Old revisions can be manually deleted: kubectl delete secret sh.helm.release.v1.<release>.v<N>.

include is a Helm template function (not native Go) that returns the output as a string that can be piped: {{ include "mychart.labels" . | indent 4 }}. template is a native Go template action that outputs inline but cannot be piped. Always prefer include for Helm templates because it enables pipeline operations like indent, nindent, and toYaml. The required function also works with include but not template.

Global values are defined under global: in values.yaml and are accessible from the parent chart and all subcharts as .Values.global.<key>. They are the cleanest mechanism for sharing configuration across subcharts (e.g., global image registry, global domain). Subcharts cannot override global values from their own values.yaml. Passing --set global.key=val overrides for all subcharts at once.

tpl renders a string as a Go template at runtime: {{ tpl .Values.templateString . }}. Use cases: (1) allowing users to provide template expressions in values.yaml (e.g., imageTag: "{{ .Release.Name }}-tag"); (2) dynamically constructing template names; (3) reading templates from ConfigMaps and rendering them. The second argument (.) provides the template context.

Tiller was an in-cluster component in Helm 2 that ran as a pod with full cluster admin permissions. The Helm client sent release data to Tiller, which rendered templates and installed them. Security concerns: Tiller had excessive RBAC permissions (cluster-admin), its removal was a primary motivation for Helm 3. Helm 3 removed Tiller entirely — the client communicates directly with the Kubernetes API server using the user's kubeconfig context. This means: (1) no in-cluster component to secure; (2) RBAC applies directly to the user's credentials; (3) the Helm binary is the only needed client; (4) no server-side secret management.

Helm tests are pods defined under templates/tests/ with the annotation helm.sh/hook: test. They typically run assertion commands (e.g., curl a service endpoint, check database connectivity) and exit 0 on success. Run them with helm test <release>. Tests are not part of the normal install lifecycle — they must be explicitly invoked. Use test containers with restartPolicy: Never and cleanup with helm.sh/hook-delete-policy: hook-succeeded. For complex tests, use a dedicated test image with your own test framework.

Chart signing: helm package --sign --key <key-name> --keyring <keyring-file> ./chart produces a .prov provenance file alongside the .tgz. Verification: helm verify ./chart.tgz --keyring <keyring-file>. For OCI registries, Helm uses notation or cosign for signing: helm push --sign (Helm 3.12+). Note that helm install does NOT automatically verify — you must explicitly run helm verify or use OCI verification in CI/CD. The provenance file contains the chart hash, signing algorithm, and a PGP signature.

Classic repos: use an index.yaml file hosted on a web server (HTTP/S). Clients download index.yaml to discover available charts. Index files can become stale, and there's no built-in access control. OCI registries: store charts as OCI artifacts in container registries (Docker Hub, ECR, GCR, ACR, Harbor, GHCR). Benefits: (1) native auth (docker login); (2) immutable tags with digest pinning; (3) geo-replication; (4) artifact deletion policies; (5) no index.yaml staleness. Cons: no native helm search, slower discovery (can't list all charts without registry API). OCI is the future — it is now stable since Helm v3.8.

Use the {{ if .Values.featureX.enabled }}...{{ end }} pattern. For subchart conditionals, use condition in dependencies: dependencies: [- name: redis, condition: redis.enabled]. More advanced: use a range with {{ if eq .Values.environment "prod" }} for environment-specific resources. default and hasKey functions handle missing values. Use {{- with .Values.config }} for scoped conditionals. For complex logic, consider a library chart with reusable template functions.

Each release revision is stored as a Kubernetes Secret in the target namespace. The secret name follows sh.helm.release.v1.<release-name>.v<revision> and is labeled with owner: helm, name: <release name>, version: <revision number>. Data field: release contains the base64-encoded, gzipped protobuf serialization of a rls.Release struct (name, info, chart, config, manifest). Helm 2 also supported ConfigMap storage, but Helm 3 uses Secrets exclusively for better security. View with kubectl get secrets -l owner=helm. The release history size grows with each upgrade — manage with --history-max.

Use a Helm pre-install/pre-upgrade hook Job that runs migration commands (e.g., alembic upgrade head, rails db:migrate). Annotate with helm.sh/hook: pre-upgrade, helm.sh/hook-weight: "-5" (run before app deploy), helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded. Risk: if two revisions are deployed simultaneously, migrations can conflict. Safer approach: use a sidecar init container or a separate migration pipeline (Argo Workflows, Jenkins) that runs migrations before Helm deploys the new version. For zero-downtime, run backward-compatible migrations before the deploy, then cleanup after.

Set kubeVersion in Chart.yaml: kubeVersion: ">=1.21.0-0 <1.28.0-0". Helm validates the cluster version during helm install/upgrade and fails if it doesn't match. For API deprecation awareness, use api-versions flag: helm template --api-versions networking.k8s.io/v1 or the HELM_API_VERSIONS environment variable. In values.schema.json, you can further validate input. For capability checks in templates, use {{ if semverCompare ">=1.21-0" .Capabilities.KubeVersion.Version }}.

A JSON Schema file that validates user-provided values during helm install, helm upgrade, helm template, and helm lint. It checks types, required fields, enums, ranges, patterns, and nested structure. Example: "image.tag": {"type": "string", "pattern": "^v?[0-9]+\.[0-9]+"}. Schema validation fails with a clear message if values violate constraints. Essential for charts with complex configuration.

{{ (lookup "v1" "Namespace" "" "my-ns").metadata.name }} queries the live Kubernetes API during template rendering. Use cases: (1) skipping resource creation if it already exists; (2) reading existing TLS secrets for cert management; (3) checking node labels for scheduling config. Warning: lookup makes helm template cluster-dependent — it only works during helm install/upgrade, not in offline rendering.

In Chart.yaml dependencies, set condition: subchart.enabled to enable/disable a subchart based on a parent value. Tags group multiple subcharts: tags: ["database"] on multiple dependencies, then --set tags.database=false disables all tagged subcharts. Conditions take precedence over tags. Example: deploy Redis and MySQL only when database=true tag is set, regardless of individual flags.

Helm provides the date function: {{ now | date "2006-01-02" }}. Go uses the reference time Mon Jan 2 15:04:05 MST 2006 (01/02 03:04:05 PM '06 -0700). Common formats: "2006-01-02" (date), "15:04:05" (time), "2006-01-02T15:04:05Z07:00" (ISO 8601). dateModify adjusts by duration: {{ now | dateModify "-24h" | date "2006-01-02" }} for yesterday.

Helm stores the entire rendered chart in the release Secret, including any plaintext secrets from values.yaml. Options: (1) External Secrets Operator: store placeholders in the chart, create actual secrets via ESO. (2) SealedSecrets: commit encrypted SealedSecret resources to Git, Helm installs them unmodified. (3) Vault Agent Injector: inject secrets from HashiCorp Vault via annotations. (4) Avoid --set with raw secret values — use --set-file or a secrets manager. (5) Set --history-max 2 to limit release secret exposure. Never commit real secrets to values files.

Library charts (type: library in Chart.yaml) contain reusable template helpers and functions but no resource templates. They cannot be installed — only used as dependencies by other charts. Application charts (type: application, default) contain actual Kubernetes resource templates and can be installed. Library charts are ideal for sharing common labels, ingress templates, and validation functions across multiple application charts.

--atomic ensures that if the upgrade fails (resources don't become ready within --timeout), Helm automatically rolls back to the previous revision. Without --atomic, a failed upgrade leaves the release in a broken failed state. --atomic is the recommended production flag: it combines --wait (wait for all resources to be ready) with auto-rollback on failure. Use it together with --timeout 10m for safe deployments.

Nested keys use dot notation: --set parent.child.key=value. Lists use brackets: --set servers[0].port=80. JSON strings: --set-string 'annotations."kubernetes.io/ingress.class"=nginx'. For complex structures, always prefer -f values.yaml over --set — it's cleaner, version-controllable, and avoids shell escaping issues. Use --set-json (Helm 3.10+) for JSON values: --set-json 'resources={"limits":{"cpu":"500m"}}'.

List all Helm release secrets: kubectl get secrets -l owner=helm. Inspect a specific revision: kubectl get secret sh.helm.release.v1.my-release.v3 -o jsonpath="{.data.release}" | base64 -d | base64 -d | gunzip (double base64 + gzip). Delete old revisions: kubectl delete secret sh.helm.release.v1.my-release.v<N>. WARNING: deleting the only revision of a release makes it irrecoverable. Use --history-max instead for automatic cleanup.

helm test runs test pods defined in templates/tests/. For CI/CD, use the chart-testing (ct) tool: ct lint-and-install --chart-dirs charts/. ct performs: (1) helm lint on changed charts; (2) install with a default values file; (3) helm test to run verification; (4) cleanup. It's the standard for Helm chart CI (GitHub Actions, GitLab CI). Test pods should include assertions: until curl -f http://my-service:8080/health; do sleep 1; done. Use helm.sh/hook-delete-policy: hook-succeeded to clean up test pods on success.

A post-renderer is an executable that receives rendered YAML on stdin and outputs modified YAML on stdout. Typical pipeline: helm template . | kustomize build | kubectl apply. With --post-renderer, you can inject sidecars, add annotations, or modify resources without changing the chart. Implementation: write a script post-render.sh that pipes through kustomize or yq. Example: #!/bin/bash; kustomize build <(cat /dev/stdin). The post-renderer runs during helm install and helm upgrade, but NOT during helm template or helm lint.

Helm merges multiple values files in order: first file is the base, subsequent files override. Deep merge (not full replacement): nested maps are merged recursively, not replaced. Arrays are replaced wholesale (not appended). --set values override all -f files. Precedence (highest to lowest): --set > --values (later files override earlier) > values.yaml. Use --set-json for programmatic overrides. For composite charts, use helm show values <chart> to see combined default values.

Chart version in Chart.yaml follows SemVer 2: MAJOR.MINOR.PATCH. Convention: bump PATCH for bug fixes, MINOR for new features (backward-compatible), MAJOR for breaking changes. Use appVersion to track the packaged application version independently. For multi-environment: use pre-release identifiers: 1.2.3-dev, 1.2.3-rc.1, 1.2.3-staging. Link to Git tags: helm package --version "$(git describe --tags)" ./chart. Use --dependency-update with helm dependency build in CI to ensure fresh subcharts.

In air-gapped environments: (1) Download all chart sources and images on a connected machine: helm dependency update ./chart && helm package ./chart. (2) Copy chart tarballs and images to the air-gapped registry. (3) Use helm repo index . to create a local repo. (4) Configure the air-gapped cluster to use the internal registry: helm repo add internal http://internal-repo/charts. (5) Push container images referenced in templates to the air-gapped registry. (6) For OCI charts: helm chart save and helm chart export to transfer offline.

Include RBAC templates in your chart: templates/serviceaccount.yaml, templates/role.yaml, templates/rolebinding.yaml. Use conditionals to create them only when needed: {{- if .Values.serviceAccount.create }}. Best practice: create a dedicated ServiceAccount per release, bind only required permissions, and use automountServiceAccountToken: false unless the app needs API access. For cluster-scoped resources, use ClusterRole/ClusterRoleBinding with caution — they span namespaces.