HashiCorp Vault
Think it through.

Vault is a secrets management platform that securely stores, controls access to, and audited access of secrets (API keys, passwords, certificates, encryption keys). Core problems: (1) Centralized secrets storage — no more secrets in config files or env vars. (2) Dynamic secrets — generate credentials on-demand with TTL (no long-lived static creds). (3) Encryption as a Service — encrypt/decrypt data with centralized key management. (4) Identity-based access — authenticate via Kubernetes, AWS IAM, LDAP, JWT/OIDC.

A secrets engine handles reading, generating, or encrypting secrets. Types: (1) KV (Key-Value) — stores static secrets at versioned paths. (2) AWS/GCP/Azure — generates dynamic cloud credentials with IAM roles. (3) Database — generates dynamic DB credentials with TTL (PostgreSQL, MySQL, MongoDB). (4) PKI — generates on-demand TLS certificates. (5) Transit — encryption-as-a-service (data stays in app, keys in Vault). (6) TOTP — generates time-based one-time passwords. Engines are enabled at paths: vault secrets enable -path=aws aws.

Vault policies grant permissions to paths using HCL: path "secret/data/myapp/*" { capabilities = ["read", "list"] }; path "aws/creds/myrole" { capabilities = ["read"] }. Capabilities: read, create, update, delete, list, sudo, deny. Policies are additive — a deny overrides other capabilities. Attach policies to auth methods: vault write auth/kubernetes/role/myrole bound_service_account_names=myapp policies=myapp-policy. Latest Vault recommends policy-as-code in version-controlled HCL files.

When Vault starts, it is sealed — it cannot read or serve any secrets. Unsealing requires a threshold of unseal keys (Shamir's Secret Sharing). Default: 5 key shares, 3 key threshold. Each unseal key decrypts part of the master key. Once the threshold is met, Vault loads the master key into memory and becomes operational. Auto-unseal: use cloud KMS (AWS KMS, GCP CKMS, Azure KeyVault) or a transit engine from another Vault cluster — eliminates manual unseal. The master key is encrypted by the KMS key; Vault automatically unseals using cloud KMS. Recommended for production.

The Vault Kubernetes auth method validates JWTs from the Kubernetes API server. Workflow: (1) App pod runs with a ServiceAccount. (2) Vault Agent Injector (mutating webhook) annotates the pod: vault.hashicorp.com/agent-inject: true. (3) Vault Agent sidecar authenticates using the pod's JWT at /var/run/secrets/kubernetes.io/serviceaccount/token. (4) Vault validates the JWT against the K8s API server, checks bound ServiceAccount/namespace, and issues a Vault token. (5) Agent sidecar fetches secrets and writes them to a shared volume (or injects as env vars). No static secrets in pod specs.

Configure a database connection: vault write database/config/postgresql plugin_name=postgresql-database-plugin allowed_roles=myrole connection_url="postgresql://{{username}}:{{password}}@host:5432/db". Create a role with TTL: vault write database/roles/myrole db_name=postgresql creation_statements="CREATE USER {{name}} WITH PASSWORD {{password}} VALID UNTIL {{expiration}}; GRANT SELECT ON ALL TABLES IN SCHEMA public TO {{name}};" default_ttl=1h max_ttl=24h. When apps read database/creds/myrole, Vault creates a new DB user, returns the credentials, and schedules automatic revocation after the TTL. Benefits: no static DB passwords, each app instance gets unique creds, automatic rotation.

Enable PKI: vault secrets enable pki. Generate a root CA (self-signed) or intermediate CA. Create a role: vault write pki/roles/internal allowed_domains=myapp.internal allow_subdomains=true max_ttl=72h key_type=ec. Apps request certificates: vault write pki/issue/internal common_name=api.myapp.internal ttl=24h. Vault returns signed cert + private key + CA chain. Automatic renewal: use Vault Agent template with renewal_ttl to keep certs fresh. Benefits: (1) Short-lived certs (hours, not months). (2) Automatic revocation via TTL expiry. (3) No manual CSR process. (4) EC or RSA key types. (5) Integration with cert-manager for Kubernetes: cert-manager Issuer of type Vault.

Vault stores data in the storage backend (Consul, Raft, Integrated Storage, file). Backup strategies: (1) Raft snapshot: vault operator raft snapshot save backup.snap (works with integrated storage/Consul). (2) File backend: snapshot the storage file. (3) Consul: consul snapshot save backup.snap. Restore: vault operator raft snapshot restore backup.snap. DR: For Vault Enterprise, Performance Replicas and Disaster Recovery clusters replicate data across regions. RPO/RTO considerations: Raft snapshots provide point-in-time recovery. Always back up the unseal keys or auto-unseal KMS configuration — without them, the backup is unrecoverable.

Vault Agent is a sidecar that authenticates with Vault, fetches/secrets, and manages token lifecycle. It runs alongside the app container. Agent Injector is a Kubernetes mutating admission webhook that automatically injects Vault Agent sidecars into pods based on annotations. Annotations: vault.hashicorp.com/agent-inject: true, vault.hashicorp.com/role: myapp, vault.hashicorp.com/agent-inject-secret-config.txt: secret/data/myapp. Agent renders secrets into a shared volume. Benefits: (1) No Vault SDK in the app. (2) Automatic token renewal. (3) Template-based secret injection. (4) Zero code changes for existing apps.