Harness
Think it through.

A Harness Delegate is a lightweight agent that runs in your infrastructure (Kubernetes, Docker, or bare metal) and connects to the Harness Manager (SaaS or On-Prem) over an outbound-only HTTPS/WebSocket connection (port 443). It executes all deployment tasks, connects to your cloud providers, artifact repositories, and verification tools. It never opens inbound ports. Architecture: The Delegate is typically a Docker container (or K8s pod) running Java. It polls the Harness Manager for tasks, executes them, and streams logs/status back. It includes various connectors and SDKs for cloud providers, CI/CD tools, and verification services. Types: (1) Kubernetes Delegate: Recommended for Kubernetes deployments, runs as a Deployment (or StatefulSet for specific use cases like GitOps with local Git repo clone). It leverages K8s RBAC for permissions. (2) Docker Delegate: Runs as a Docker container on any host, suitable for Docker-based deployments or on hosts that can reach target environments. (3) Shell Script Delegate: For bare metal/VMs, runs as a systemd service (or similar), often used for legacy infrastructure or specialized tasks. High Availability (HA): For Kubernetes Delegates, deploy with at least 2 replicas (e.g., replicas: 2 in the Delegate YAML) across different Availability Zones. The Harness Manager automatically load-balances tasks across healthy Delegates. For Docker/Shell Delegates, deploy multiple instances on different hosts and use a load balancer or ensure redundancy in task assignment. Secure Networking: (1) Outbound-only: Delegates only initiate connections to the Harness Manager, eliminating the need for inbound firewall rules. (2) Least Privilege: Delegate's IAM role (AWS), Service Account (K8s), or local user should have only the minimum necessary permissions to perform its tasks (e.g., deploy to a specific namespace, access a specific S3 bucket). (3) Network Policies: In Kubernetes, apply Network Policies to restrict the Delegate pod's communication to only the Harness Manager endpoint, artifact registries, cloud provider APIs, and target application namespaces. (4) Private Link/Service Endpoints: For enhanced security, configure AWS PrivateLink or GCP Private Service Connect for the Harness Manager connection, ensuring traffic never traverses the public internet. (5) Delegate Isolation: Use separate Delegates or Delegate groups for different environments (dev/staging/prod) or teams to enforce blast radius isolation. (6) Secrets Management: Delegates retrieve secrets from configured Secret Managers (Vault, KMS) at runtime; they do not store long-lived credentials directly.

Harness's "Git Experience" (also known as GitOps) allows users to define and manage their Harness Pipelines, Services, Environments, and other core entities directly as YAML files in a Git repository. This extends the GitOps philosophy beyond just application deployments to the entire CI/CD process. How it works: (1) Declarative Configuration: All Harness entities (pipelines, services, environments, infrastructure definitions, connectors) are represented as YAML. (2) Git as Source of Truth: These YAML files are stored in a Git repository, which becomes the single source of truth for your CI/CD setup. (3) Synchronization: Harness continuously monitors the Git repository for changes. When a change is pushed to Git, Harness automatically synchronizes these changes to update or create entities in the platform. This can be configured for automatic or manual sync. (4) Bidirectional Sync: Changes made in the Harness UI can also be pushed back to Git, though the primary flow for GitOps is Git-first. Advantages over UI-only management: (1) Version Control: All pipeline definitions and configurations are versioned in Git, allowing for easy tracking of changes, rollbacks, and auditing. (2) Collaboration: Teams can collaborate on pipeline definitions using standard Git workflows (pull requests, code reviews, branching), improving consistency and quality. (3) Auditability: Every change to a pipeline or configuration is associated with a Git commit, providing a clear audit trail. (4) Disaster Recovery: In case of a Harness outage or configuration corruption, the entire CI/CD setup can be recreated from the Git repository. (5) Environment Parity: Easily replicate environments by copying Git-managed configurations, ensuring consistency between dev, staging, and production. (6) Automation: Programmatic updates to CI/CD pipelines can be made via Git commits, enabling "pipeline as code" automation. (7) Security: Git access controls (branch protection, code owner reviews) can be leveraged to secure pipeline changes. For Application Configurations: Harness integrates with Git for application manifest management (e.g., Kubernetes YAML, Helm charts, Kustomize overlays). This means your application configs also live in Git, and Harness pulls them at deployment time, aligning with pure GitOps principles for application delivery.

Harness integrates with OPA to provide "Policy as Code" enforcement at various stages of the CI/CD pipeline, ensuring compliance and governance. This is typically done using OPA policies (written in Rego) and evaluated by OPA or an OPA-compatible engine like OPA Gatekeeper. Integration Points: (1) Harness Policy Enforcement (Native): Harness has a built-in Policy Enforcement feature that allows you to upload Rego policies directly into Harness. These policies can be applied to specific entities (Pipelines, Stages, Services, Environments) or events (e.g., before execution, after deployment). (2) Custom OPA Steps: You can also use a custom "Shell Script" or "OPA" step in a Harness pipeline to invoke an external OPA server or Gatekeeper instance with the relevant data (e.g., rendered Kubernetes manifests). Scenario Design: Goal: Prevent deployments of latest image tags OR deployments requesting >2 CPUs in non-production environments (dev, staging). OPA Policy (Rego): package kubernetes.admission deny[msg] { input.request.kind.kind == "Deployment" image := input.request.object.spec.template.spec.containers[_].image endswith(image, ":latest") msg := "Deployment denied: Cannot use 'latest' image tag." } deny[msg] { input.request.kind.kind == "Deployment" cpu_limit := input.request.object.spec.template.spec.containers[_].resources.limits.cpu environment := input.request.namespace # Assuming environment name is the Kubernetes namespace environment != "production" to_number(cpu_limit) > 2 # Convert "2000m" to number 2 etc. or use a helper msg := sprintf("Deployment denied in %v: CPU limit (%v) exceeds 2 for non-production environments.", [environment, cpu_limit]) } *Note: The to_number conversion for CPU limits in Rego can be complex (e.g., "200m" vs "2"). A more robust policy would handle different CPU string formats or convert to millicores.* Harness Implementation Steps: (1) Define OPA Policy in Harness: Go to Harness Governance -> Policies. Create a new Policy Set. Upload the Rego policy above. Configure the Policy Set to apply to "Deployments" (or "Kubernetes Resources") and specify the evaluation stage (e.g., "Before Deployment"). (2) Policy Evaluation in Pipeline: When a Harness pipeline executes a Kubernetes deployment, before applying the manifests to the cluster, Harness will extract the rendered Kubernetes Deployment manifest. It then sends this manifest as input.request.object to the internal OPA engine (or the configured external OPA endpoint). The OPA engine evaluates the Rego policy against the manifest. If any deny rule in the Rego policy matches, the OPA engine returns a denial message. (3) Pipeline Reaction: Harness receives the denial from OPA. The deployment stage in the pipeline fails, preventing the non-compliant deployment from reaching the Kubernetes cluster. The pipeline execution will show the OPA policy violation message, providing immediate feedback to the developer. Alternative (OPA Gatekeeper): For real-time, admission control at the Kubernetes cluster level, you could deploy OPA Gatekeeper to your GKE/EKS cluster. The Harness pipeline would still deploy, but Gatekeeper would intercept the API call and reject it, effectively achieving the same outcome but at a lower level in the stack. Harness Policy Enforcement is preferred for shifting left, catching issues before they even attempt to hit the cluster API.

Harness "Services" and "Environments" are fundamental abstractions designed for reusability and standardization in complex CI/CD scenarios. Service: A Service in Harness represents your application or microservice artifact (e.g., a Docker image, a Helm chart, a serverless function package). The key is that a Service definition focuses on *what* is being deployed, abstracting away *how* or *where* it's deployed. Reusability: A single Service definition (e.g., my-frontend-service) can be used across multiple pipelines and deployed to various environments without modification. It holds common attributes like artifact source, manifest definitions (Kubernetes YAML, Helm values), and configuration files. Standardization: All deployments of my-frontend-service will use the same base manifests and artifact definitions, ensuring consistency. Environment: An Environment represents a deployment target (e.g., Development, Staging, Production). It defines the infrastructure (Kubernetes clusters, VMs, cloud accounts), runtime settings, and any environment-specific overrides. Reusability: Environments are typically defined once and then referenced by multiple pipelines and services. Standardization: Each Environment enforces consistent infrastructure and access controls for all services deployed into it. Promoting Reusability & Standardization: The power comes from their decoupled nature and the ability to inject environment-specific configurations at deployment time. Configuration Overrides: Harness allows you to define configurations at the Service level (common defaults) and then override them at the Environment level or Infrastructure Definition level. This is done through "Service Overrides" and "Environment Overrides". Template Library: Harness's template library allows you to parameterize common steps, stages, or entire pipelines, further enhancing reusability. Example: Single Service, Multiple Environments: Let's say you have a frontend-service that uses a Helm chart. (1) Service Definition: frontend-service is defined once in Harness. It points to a Docker image my-registry/frontend and a Helm chart charts/frontend. Its base values.yaml (in the Service definition) might define default replica count replicas: 2, and a generic hostname: frontend.local. (2) Environment Definitions: Dev Environment: Points to a dev-k8s-cluster Infrastructure Definition. Prod Environment: Points to a prod-k8s-cluster Infrastructure Definition. (3) Environment Overrides: For the Dev Environment, create a Service Override for frontend-service: replicas: 1 hostname: dev.example.com. For the Prod Environment, create a Service Override for frontend-service: replicas: 5 hostname: app.example.com resources: limits: cpu: "2" memory: "4Gi". (4) Deployment: When you run a pipeline that deploys frontend-service to the Dev Environment, Harness will merge the base Service values with the dev-values.yaml override. When deploying to Prod, it uses prod-values.yaml. This enables a single pipeline and service definition to deploy to diverse environments with distinct resource requirements, domain names, or replica counts, all managed in a standardized, version-controlled manner.

Harness Chaos Engineering (CE) integrates into CI/CD pipelines as a dedicated stage or step to proactively identify weaknesses and validate resilience. It leverages open-source chaos tools like LitmusChaos and Chaos Mesh, orchestrating experiments and analyzing their impact. Integration into CI/CD: (1) Automated Execution: CE experiments can be triggered automatically after a deployment (e.g., a canary or blue-green) or as part of a pre-production gate. (2) Contextual Fault Injection: Experiments target specific services, pods, or nodes based on the deployment context, allowing for precise testing of the newly deployed version. (3) Verification & Gates: The results of chaos experiments (e.g., application performance, error rates, SLO compliance) can be used as a pass/fail criterion for pipeline progression, ensuring that only resilient services are promoted. (4) Observability Integration: Harness CE integrates with Continuous Verification (CV) tools (Prometheus, Datadog, etc.) to monitor the impact of chaos experiments in real-time and automatically assess health. Pipeline Stage Design: Network Latency Fault Injection during Canary: Scenario: During a 10% canary deployment of my-api-service, inject 200ms network latency to the canary pods and verify that p99_latency for the service doesn't exceed 500ms and error_rate remains <1%. Harness Pipeline Structure: (1) Stage 1: Build & Deploy Canary (10%): Type: Deploy Stage (Kubernetes, Canary Strategy). Steps: Deploy my-api-service (new version) to 10% traffic. Verification: Harness CV monitors baseline metrics (CPU, Memory) to ensure the canary pods are stable. (2) Stage 2: Chaos Engineering Experiment: Type: Custom Stage (or a dedicated CE stage if available in newer Harness versions). Steps: Chaos Experiment Setup (Shell Script/Harness CE Step): name: Inject Network Latency to Canary type: ShellScript spec: script: | # Assuming LitmusChaos is installed in the cluster and a LitmusChaos connector is set up in Harness CANARY_PODS=$(kubectl get pods -n my-namespace -l app=my-api-service,version=canary -o jsonpath='{.items[*].metadata.name}') echo "Targeting canary pods: $CANARY_PODS" # Apply a LitmusChaos experiment manifest for network latency cat <<EOF | kubectl apply -f - apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: api-latency-canary namespace: my-namespace spec: engineState: active chaosServiceAccount: litmus-admin # Or your custom SA experiments: - name: pod-network-latency spec: components: env: - name: POD_NAMES value: "$CANARY_PODS" # Target only canary pods - name: NETWORK_LATENCY value: "2000" # 200ms (Litmus expects microseconds) - name: TARGET_CONTAINER value: "my-api-container" # Name of your app container # ... other Litmus specific configurations EOF echo "Waiting for chaos experiment to complete..." # Use LitmusChaos CLI or K8s watch to wait for experiment status kubectl wait --for=condition=Completed chaosengine/api-latency-canary -n my-namespace --timeout=5m # Capture any LitmusChaos logs/results for further analysis. Verification (Harness CV Step): Type: Verification (e.g., Prometheus). Configuration: Baseline: Compare against control (old version) pods or a historical window. Metrics: Monitor p99_latency (target < 500ms), error_rate (target < 1%) for my-api-service. Analysis Duration: Run for 5-10 minutes during chaos injection. Failure Strategy: On Failure: Rollback (or Fail Stage). (3) Stage 3: Promote Canary / Full Rollout: Type: Deploy Stage (Kubernetes, Rolling Strategy). Steps: If chaos experiment and verification pass, promote the canary to 100% traffic. Influence on Deployment Progression: If the p99_latency or error_rate metrics exceed their defined thresholds during the chaos experiment, the Harness CV step will mark the experiment as a failure. Based on the configured failure strategy (On Failure: Rollback), the pipeline will automatically trigger a rollback of the canary deployment, preventing the non-resilient service from reaching full production. This ensures that the application is tested for resilience under controlled fault conditions before it is fully deployed, catching potential issues proactively.

A Stage is a logical grouping of steps that runs as a unit on shared infrastructure. The four core CD stage types are: (1) Deploy — deploys a service to an environment using strategies like Canary, Blue-Green, or Rolling; (2) Approval — pauses execution for manual or Jira/ServiceNow ticket approval; (3) Custom Stage — runs arbitrary shell/plugin steps not tied to a service deployment; (4) Pipeline Chaining — triggers another pipeline as a child. Stages can run in parallel by placing them side-by-side in the visual editor or setting parallel: true in YAML.

Likely causes: (1) Verification step failing silently — Harness CV is evaluating metrics and the ML model is marking the canary unhealthy, but the stage is configured with On Failure: Ignore. Check the CV step's detailed log in Pipeline Studio execution view. (2) Missing approval gate — a manual approval step is waiting with no notification configured. (3) Traffic provider misconfiguration — the Istio VirtualService or AWS ALB listener rule was created but the weight update API call silently failed; check Delegate logs on the delegate pod. (4) Timeout not set — the stage has an indefinite wait for a metric window. Fix: set explicit progressDeadlineSeconds on the canary config and configure CV failure strategy to Rollback.

Harness encrypts secrets at rest using AES-256 with keys stored in its Secret Manager (Harness Built-in, AWS KMS, GCP KMS, HashiCorp Vault, or Azure Key Vault). In transit, secrets flow from Harness SaaS to the Delegate over an outbound-only HTTPS WebSocket — the Delegate initiates the connection, so no inbound ports are needed. For AWS credentials: (1) Preferred — IRSA/Workload Identity: attach an IAM role to the Delegate's Kubernetes service account; the pipeline never touches raw credentials. (2) Second choice — Harness AWS Cloud Provider: configure once in Harness with stored secret. (3) Anti-pattern: storing AWS_ACCESS_KEY_ID as a pipeline variable — even masked, it appears in the Delegate process env. Log masking: Harness automatically masks any value accessed via <+secrets.getValue("my-secret")> but raw env var injection bypasses masking.

Harness CV uses unsupervised ML (CUSUM and Holt-Winters variants) to compare the canary's metric/log behavior against a control node running the old version simultaneously. Data sources: Prometheus, Datadog, New Relic, AppDynamics, Splunk, Elastic, Google Cloud Operations, Dynatrace, and custom HTTP metrics. Tuning for noisy services: (1) Increase analysis duration — set durationInMinutes: 30 minimum for services with 5-min traffic ramps. (2) Custom thresholds — override ML with fixed thresholds: thresholdType: ABSOLUTE, value: 500 for p99 latency. (3) Sensitivity — lower from HIGH to MEDIUM or LOW; HIGH fires on 1-sigma deviations, MEDIUM on 2-sigma. (4) Ignore list — exclude metrics legitimately different in canary (e.g., JVM GC spikes on startup). (5) Baseline window — set LAST_SUCCESSFUL_PIPELINE_EXECUTION rather than LAST_30_MINUTES for services with diurnal patterns. (6) Node filtering — for Kubernetes, pin baseline to nodes with matching instance types to avoid hardware-level metric noise.

Stage 1 — Build & Test (CI Stage): build Docker image, push to ECR, run pytest integration tests against an ephemeral namespace using Harness Test Intelligence. On failure: fail fast, send Slack alert. Stage 2 — Canary us-east-1 (Deploy Stage, Canary strategy): 10% traffic via Istio VirtualService. CV step monitors error_rate, p99_latency, slo_burn_rate_1h from Prometheus for 15 minutes, sensitivity MEDIUM. On CV failure: auto-rollback, post incident to PagerDuty via Harness HTTP step, halt pipeline. On success: promote to 100%. Stage 3 — SLO Gate (Custom Stage): HTTP step calls Prometheus to assert burn_rate < 0.1 over 30-min window. On failure: trigger rollback stage. Stage 4 — Multi-Region Promote (two Deploy Stages in parallel): eu-west-1 and ap-southeast-1 use Rolling strategy with health checks. Failure strategy: On Stage Failure: Rollback for each region independently — eu-west-1 failure doesn't roll back ap-southeast-1. Rollback Stage (triggered by failure expression): runs helm rollback in each region. Notifications at every gate transition via Slack webhook stored as Harness secret.

Use Harness pipeline stages with Approval steps: (1) Dev stage — Deploy stage with on trigger: merge to main, no approval step. (2) Staging stage — needs: dev, add a Harness Approval step with type: UserGroup specifying which groups must approve. Set autoRejectPreviousStagesOnPass: false so dev changes don't auto-promote. (3) Prod stage — needs: staging, add two Approval steps: first a UserGroup approval, second a Jira Create Ticket or ServiceNow Approval step that requires a valid ticket key in the pipeline input. (4) Use pipeline variables: pipeline.variables.ticket_ref passed at trigger time. (5) For the prod stage, add a condition: <+pipeline.variables.ticket_ref> != "" on the stage execution. (6) Use enforcements via Harness Policy Engine (OPA) to ensure only authorized users can trigger the prod stage. (7) To prevent concurrent deploys to prod: set concurrency: prod-deploy on the prod stage with cancelInProgress: false so deploys queue.

Harness supports multiple secret managers: Built-in (AES-256), AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, and Custom. Migration from Harness text secrets to AWS Secrets Manager: (1) Create an AWS Secrets Manager connector in Harness with cross-account IAM role (if multi-account). (2) For each secret, migrate the value to AWS: aws secretsmanager create-secret --name /harness/prod/db-password --secret-string "...". (3) In Harness, replace secrets.getValue("my-secret") references with the AWS SM reference: secrets.getValue("aws_secrets_manager:/harness/prod/db-password"). (4) Test in a non-prod pipeline first — verify the secret resolves correctly by running a dry-run or a step that echoes (ensure masking is on). (5) Migrate in batches by environment — dev first, then staging, then prod. (6) Keep the old Harness text secret as a fallback with a deprecation label. Rollback: set Harness text secret back as the active reference and re-run the pipeline. Risk: if the AWS SM connector is misconfigured (e.g., permissions revoked), ALL pipelines fail. Mitigation: set up a health check pipeline that runs every 5 minutes to verify secret resolution and alerts on failure. For DR: replicate AWS SM secrets to a secondary region.

Harness Pipeline Templates allow defining reusable stage and pipeline blueprints. Strategy: (1) Create a Stage Template for each deployment type: cd-deploy-staging-template and cd-deploy-prod-template. The template includes: checkout, build, scan (Trivy/Snyk), deploy (canary for prod, rolling for staging), verification (CV with Prometheus), and rollback. (2) Templates use Runtime Inputs for customizable parameters: <+service.name>, <+env.name>, <+artifacts.primary.tag>, <+pipeline.variables.slack_channel>. (3) Each service's pipeline uses the template with specific values: templateRef: cd-deploy-prod-template; values: {service: service-a, env: prod, slack_channel: team-a-alerts}. (4) Governance: use Harness Policy Engine (OPA) to enforce that all pipelines must use the approved templates. OPA rule: deny[msg] { input.pipeline.stages[_].template.templateRef != "cd-deploy-prod-template" }. (5) For customization, templates expose explicit overrides (e.g., skip_scan: false) — never allow arbitrary YAML passthrough. (6) Version templates: when updating, create v2, deprecate v1 with a warning in the template YAML. (7) Audit: policies: templateVersion: ">= 2.0" to enforce that only current templates are in use. This standardizes 80% of the pipeline while leaving 20% for service-specific needs.

A Connector stores credentials and configuration for external tools. Types: Kubernetes Cluster Connector, Docker Registry, GitHub/GitLab/Bitbucket, AWS/GCP/Azure cloud providers, Jira, ServiceNow, PagerDuty, Helm Chart Repository, and many more. Connectors are referenced by pipelines and governed by Harness connectors with scope settings.

Organization is the top-level container grouping related projects. Project is where pipelines, services, environments, and resources live. Hierarchy: Account -> Organizations -> Projects. RBAC can be assigned at any level. A typical setup: an Organization per business unit, a Project per application or team.

A Delegate is a lightweight agent running in your infrastructure that executes pipeline tasks. It connects outbound to Harness SaaS over HTTPS/WebSocket (port 443). It runs deployments, connects to cloud providers, pulls artifacts, and runs scripts. Delegates are the bridge between Harness and your environments — no inbound ports needed.

Harness supports Canary (gradual traffic shift), Blue-Green (two full environments with instant switch), Rolling (one pod at a time), and Basic (all at once). These can be used with Kubernetes, Helm, Native Helm, ECS, Serverless, TAS, and SSH/WinRM deployments.

An Environment is a logical deployment target (Dev, Staging, Prod). An Infrastructure Definition specifies the actual infrastructure details (Kubernetes namespace, cluster, AWS region, etc.) within that environment. Multiple Infrastructure Definitions can exist per Environment (e.g., us-east-1 and eu-west-1 clusters in the Prod environment).

CCM uses a Delegate to connect to cloud provider billing APIs (AWS CUR, GCP BigQuery, Azure) and Kubernetes cost data via metrics. Features: (1) Cluster cost visibility — per-namespace, per-pod, per-label cost breakdown; (2) Anomaly detection — ML-based spending alerts; (3) Budgets — set thresholds and receive notifications; (4) Recommendations — rightsizing, idle resources, and savings plans; (5) Perspectives — custom cost views filtered by dimensions.

Feature Flags allow toggling functionality on/off without redeploying. Flags are evaluated server-side in Harness SaaS or via a local SDK. Integration with CD: (1) use flags to control canary rollout percentages; (2) target specific user segments (beta testers, internal teams); (3) kill-switch misbehaving features instantly; (4) run A/B tests by splitting traffic. Flags have types: boolean, string, number, and JSON.

Variables can be defined at: (1) Pipeline-level — scoped to the pipeline run, overridden per execution; (2) Stage-level — scoped to a single stage; (3) Service-level — scoped to all deployments of that service; (4) Environment-level — overrides per environment; (5) Project-level — shared across all pipelines in the project. Precedence (highest): pipeline input -> stage -> service/environment -> project. Use <+variable.scope.name> syntax to reference them.

Harness uses Artifact Sources (Docker Registry, ECR, GCR, Artifactory, Nexus, etc.) to automatically resolve the latest version based on tag regex. Example: regex: v.* resolves the latest version with a v prefix. Promote the same artifact version across environments by using the same build step in CI — the artifact tag flows through Dev -> Staging -> Prod automatically without manual re-entry.

An Input Set captures runtime values for a pipeline (variables, artifact tags, environment overrides) and saves them as a reusable configuration. Common use: create Input Sets per environment (dev-input-set, prod-input-set) so running a pipeline with the prod Input Set pre-fills all production values. Input Sets can be triggered via API, webhooks, or the UI.

In the Service definition, specify dependent services that must be deployed first. Harness handles ordering during multi-service deployments. You can also use pipeline stages with needs: for ordering. For runtime dependency checking, add an init container or a helm hook that waits for the dependent service to become ready via kubectl wait or a readiness endpoint.

RBAC has three components: (1) Roles — predefined (Account Admin, Project Admin, Pipeline Executor) or custom with specific permissions; (2) Resource Groups — scope of resources (all pipelines, specific connectors, all secrets); (3) User Groups — users assigned to roles + resource group bindings. Inheritance: Project-level roles are inherited from Organization/Account unless overridden. Use SSO/LDAP/SCIM for user group sync from Okta, Azure AD, or Google Workspace.

Use the ServiceNow Approval step in a Harness pipeline. Configure a Harness Connector for ServiceNow with credentials. The step can: (1) Create a change request in ServiceNow with template, priority, risk, and description fields; (2) Wait for approval (standard, normal, or emergency change); (3) Post deployment results back (close, cancel, update). This ensures every prod deployment has a ServiceNow ticket linked.

Overrides allow customizing Service configuration per Environment without modifying the Service definition. Types: (1) Manifest Overrides — override Kubernetes YAML values per environment; (2) Config Overrides — environment-specific ConfigMap/Secret values; (3) Variable Overrides — set Service variables per environment; (4) Infrastructure Overrides — override namespace, cluster, or region. Overrides are applied at deploy time, merging with Service defaults.

Stage 1 — Deploy Canary (10%): Kubernetes Canary strategy, 10% traffic via Istio VirtualService. Stage 2 — SLO Verification (CV step): monitor error_rate, p99_latency, slo_burn_rate_1h from Prometheus for 10 minutes. Configure failOnNoAnalysis: true. Stage 3 — Decision: Harness HTTP step queries a Metrics API to check burn_rate < 0.1. If fails, trigger Rollback Stage via a Harness Failure Strategy with onFailure: rollback. Rollback Stage: K8s Apply step that sets the stable service selector, followed by scaling the canary deployment to 0. Rollback triggers a PagerDuty notification. On success: promote to 100% via a second Deploy stage.

Use pipeline stage templates with environment abstraction. (1) Create three Infrastructure Definitions in the Production Environment: one per cloud provider (EKS us-east-1, GKE us-central1, AKS westus). (2) Create a Deploy stage with a matrix strategy: strategy: matrix: cloud: [aws, gcp, azure]. (3) Use conditional logic in manifests: {{ if eq .Values.cloud "aws" }}...{{ end }}. (4) Each matrix iteration selects the appropriate cloud connector and Infrastructure Definition using Harness expressions. (5) Use Run steps for cloud-specific setup (e.g., aws eks update-kubeconfig, gcloud container clusters get-credentials). (6) Parallelize with stage parallelism; add a manual approval gate before prod promotion across all three clouds.

Harness CV uses log-based and metric-based verification. Metric analysis: it applies CUSUM (cumulative sum) and Holt-Winters (exponential smoothing) to compare canary metrics against baseline from the same time window. If the canary's metrics deviate beyond the ML-calculated threshold (configurable sensitivity), it flags the deployment as unhealthy. Log analysis: it uses log clustering (by message pattern similarity), comparing log event rates between canary and baseline. A significant increase in error-level logs triggers a health verdict. Key healthy metrics: error rate (must not spike), p95/p99 latency (within threshold), throughput (consistent), CPU/memory (no regression). The ML model auto-adjusts thresholds based on historical data, reducing noise on services with natural variability.

Challenges with StatefulSet blue-green: StatefulSets have persistent identities (PVCs, network IDs) that are hard to duplicate. Strategy: (1) Use two StatefulSets (app-blue, app-green) with different PVCs. (2) Pre-warm the new environment by restoring a snapshot or replicating from the primary (using a pre-deploy hook). (3) Use a custom Harness step to switch the read/write endpoint: update a Service selector or a DNS record to point to the new StatefulSet. (4) Data migration: use a Helm pre-upgrade hook Job that runs pg_dump/mysql dump in the old environment and restores in the new one. (5) Rollback: keep the old StatefulSet running and the Service pointing at it until migration is verified. (6) For read-heavy workloads, use a read replica pattern — keep both running during cutover. Harness does not natively handle database migration — use custom scripts, Liquibase, Flyway, or Alembic in pipeline steps.

Harness IaCM allows managing Terraform, Terragrunt, and OpenTofu state within Harness. Integration: (1) Store Terraform state in a Harness-managed backend (S3, GCS, or Harness built-in). (2) Use IaCM Pipeline Steps: Terraform Plan and Terraform Apply steps in your CD pipeline. (3) Run IaCM before the Deploy stage to provision infrastructure. (4) Use Harness's drift detection: IaCM periodically runs terraform plan and alerts on drift. (5) Policy enforcement: run OPA policies against Terraform plans via Harness Policy Engine. (6) Cost estimation: IaCM shows cost impact of infrastructure changes before apply.

Stage 1 — Pre-validation: verify primary region health, snapshot the database, record current RPO. Stage 2 — Failover: uses custom Run steps to update DNS/Global Accelerator to point to the secondary region. Stage 3 — Verification: deploy test workloads to the secondary region, run integration tests, measure time from failover initiation to application serving traffic (RTO measurement). Stage 4 — Validation: compare data freshness between the failover database and the last backup window (RPO validation). Stage 5 — Failback: restore primary region, run tests, update DNS. Stage 6 — Reporting: Harness HTTP step posts RTO/RPO results to a dashboard (Grafana, Datadog). Use Harness SLO tracking over multiple runs to establish baseline RTO/RPO trends. Automate the entire pipeline on a schedule (weekly or monthly).

Add STO stages/steps in your pipeline: (1) Scan step: configure a scanner (Trivy, Snyk, Aqua, SonarQube, Checkmarx, etc.) via Harness STO connector. (2) Pipeline step runs the scan against the built artifact or Kubernetes manifest. (3) Policy enforcement: use Harness OPA/Policy Engine to block the pipeline if critical or high-severity findings exceed thresholds (e.g., deny[msg] { count(high_cves) > 0 }). (4) Results are stored in Harness and viewable in the STO dashboard. (5) Best practice: run SAST in CI, SCA in CI, DAST in staging CD, container scan in both CI and CD. Harness STO normalizes results from 15+ scanners into a unified severity model.

Harness CE orchestrates chaos experiments using LitmusChaos or Chaos Mesh. Architecture: (1) Harness deploys a Chaos Delegate (separate from the CD Delegate) that runs chaos probes. (2) Experiments are defined as CRDs (ChaosEngine, ChaosExperiment) executed on target clusters. (3) In a pipeline, add a Chaos step that references an experiment: type: Chaos, spec: {experimentName: pod-delete, tolerance: 500ms}. (4) The step monitors the steady-state hypothesis via probes (HTTP, command, PromQL). (5) Results: pass/fail + steady-state validation metrics. Custom experiments: write your own ChaosEngine YAML with custom faults (network latency, CPU spike, pod kill, DNS error) and upload to Harness CE via the UI or Git. Use sequential or parallel execution modes for multi-fault experiments.

Causes: (1) Delegate is overloaded — check CPU/memory with kubectl top pod -n harness-delegate. (2) Delegate heartbeat lost — Delegate hasn't pinged Harness Manager in 5+ minutes; check network connectivity (firewall, proxy, DNS). (3) Task selector mismatch — pipeline references a tag that no Delegate has (e.g., delegate-selector: prod-k8s but all Delegates only have dev tags). (4) Delegate version too old — upgrade Delegate to match Harness SaaS version. (5) Delegate replicas insufficient — increase from 1 to 2+ for HA. Diagnostics: check Delegate logs with kubectl logs -n harness-delegate deploy/harness-delegate --tail=100, verify kubectl get pods shows Running, check Delegate Overview in Harness UI showing green heartbeat. Fixes: restart Delegate, increase resources, verify network egress to app.harness.io, add matching selectors.

A Pipeline is an end-to-end workflow that can include CI, CD, Approval, Custom, and Pipeline stages. Core components: Stages (logical phases), Steps (individual actions within a stage), Variables, Input Sets, Triggers, and Failure Strategies. Pipelines are defined in YAML or via the visual editor.

Harness supports storing pipeline definitions in Git repositories. This enables version control, pull request reviews, and audit trails for pipeline changes. Configure in Project Settings > Git Experience. Choose between Harness-managed (stored in Harness) or Git-managed (stored in your repo, Harness syncs from Git).

Add an Approval stage with step type Jira Approval. Configure: (1) Jira connector with credentials. (2) Issue criteria: status, assignee, or custom field conditions. (3) Polling interval (default 30s). The pipeline pauses until the Jira ticket meets the criteria. For manual approvals, use the Harness Approval step with User Groups. Combine both: require Jira ticket approval + manual approval for production deployments.

Create a Trigger in the Project > Triggers section. Types: (1) Git Event — Webhook-based, triggers on push, PR create, PR merge. (2) Schedule — cron-based. (3) New Artifact — triggers when a new artifact version is available. (4) Pipeline completion — chain pipelines. Git triggers require a webhook configured in your Git provider pointing to Harness. Use branch conditions and file path filters to limit which changes trigger the pipeline.

SME is a self-hosted version running in your own infrastructure (K8s cluster). Differences: (1) Data never leaves your network — compliance/air-gapped. (2) You manage upgrades, scaling, and backups. (3) Includes a built-in Harness Manager (UI + API) and can use your own DB (PostgreSQL) and storage. (4) Replicas can be scaled independently. SaaS is simpler to operate, while SME is for regulated industries with strict data residency requirements.

Use a multi-target deployment strategy: (1) Define multiple Infrastructure Definitions in the Environment — one per cluster/region. (2) Enable Deploy to all targets or use a matrix strategy: strategy: matrix: region: [us-east-1, eu-west-1, ap-southeast-1]. (3) Each target uses its own Kubernetes Connector. (4) Use Harness expressions to reference region-specific values: <+infra.connector.name>, <+infra.namespace>. (5) Choose parallel or sequential execution in the stage settings.

Harness can create and manage Istio VirtualService and DestinationRule resources for traffic splitting. Configuration: (1) In the Deploy stage, select Canary strategy. (2) Set traffic routing to use Istio. (3) Harness renders an Istio VirtualService with weighted destinations (e.g., weight: 90 for stable, weight: 10 for canary). (4) Progression: Harness increases canary weight in steps (10% → 25% → 50% → 75% → 100%) based on CV health checks. (5) On success: VirtualService points 100% to the new version. (6) On failure with auto-rollback: Harness reverts VirtualService to 100% stable. No manual Istio manifest editing needed.

Goal: every pipeline step authenticates via a short-lived SPIFFE identity, not long-lived secrets. Implementation: (1) Deploy SPIRE agent on your Kubernetes cluster, one per node (DaemonSet). (2) SPIRE server issues SVIDs (SPIFFE Verifiable Identity Documents) to workloads based on k8s Pod annotations. (3) In the Harness deploy pipeline, use an init container that requests an SVID via the SPIRE agent Unix socket. (4) The application container receives the SVID and uses mTLS to authenticate to downstream services. (5) Harness Steps that call external services (Vault, DB, Cloud API) use the workload's SPIFFE identity via a sidecar that proxies mTLS connections. (6) Harness CD can verify the SVID's validity before proceeding, ensuring each deployment has verifiable identity. (7) This replaces static service account tokens and API keys with ephemeral, automatically rotated identities.

Harness GitOps uses ArgoCD under the hood but provides a simplified UI and policy layer. Setup: (1) Install Harness GitOps agent in your cluster (a wrapper around ArgoCD). (2) Connect repositories (GitHub, GitLab, Bitbucket) in the GitOps module. (3) Create GitOps Applications: specify source repo path, target cluster, and sync policy (auto-sync with prune). (4) Harness adds: RBAC (who can sync), approval gates (manual sync approval), drift detection alerts, and integration with CD pipelines. (5) Use Multi-Service and Multi-Environment dashboards. (6) For promotion between environments, use Harness PR Pipelines that auto-create PRs in the GitOps repo to update image tags.

Harness SRM tracks SLOs from monitoring tools (Prometheus, Datadog, New Relic, Splunk) and uses them as deployment gates. Integration: (1) Define SLOs in SRM (e.g., p99 latency < 500ms, error budget burn rate < 2). (2) In the CD pipeline, add a CV step with analysisType: SLO. (3) During canary, SRM compares the canary's real-time metrics against the SLO burn rate. (4) If the canary breaches the SLO, the pipeline auto-rolls back. (5) If the error budget is already exhausted, the pipeline blocks the deployment before it starts. This prevents deploys when the service is already in a degraded state.