Secrets Management Overview
Kubernetes Secrets are objects that contain small amounts of sensitive data. Without proper secrets management, passwords, API keys, and tokens can be exposed, leading to compromised systems. This guide covers encryption, external secret stores, and best practices.
💡 Key Insight: By default, Kubernetes secrets are stored unencrypted in etcd. Always enable
encryption at rest and use external secret management tools.
Secret Types in Kubernetes
- Opaque: Default secret type for arbitrary data
- kubernetes.io/service-account-token: Service account tokens
- kubernetes.io/dockercfg: Serialized Docker config
- kubernetes.io/dockercfg-json: Docker registry credentials
- kubernetes.io/basic-auth: Credentials for basic authentication
- kubernetes.io/ssh-auth: SSH authentication data
- kubernetes.io/tls: TLS certificate and key pair
- bootstrap.kubernetes.io/token: Bootstrap token data
# Create encryption configuration file
/etc/kubernetes/enc/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- aescbc:
keys:
- name: key1
secret:
- identity: {}
Configure API Server
# Update /etc/kubernetes/manifests/kube-apiserver.yaml
--encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
--encryption-provider-config-automatic-reload=true
⚠️ Important: Encrypting existing secrets requires re-creating them. Plan for downtime
or use gradual migration strategies.
Configure API Server
# Update /etc/kubernetes/manifests/kube-apiserver.yaml
--encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml
--encryption-provider-config-automatic-reload=true
⚠️ Important: Encrypting existing secrets requires re-creating them. Plan for downtime
or use gradual migration strategies.
External Secret Management Tools
HashiCorp Vault
Industry-leading secrets management platform:
- Centralized secrets storage and access control
- Dynamic secret generation
- Audit logging and secret rotation
- Multiple authentication methods
AWS Secrets Manager
AWS-native secrets management:
- Automatic secret rotation
- IAM-based access control
- Integration with EKS
- CloudTrail audit logging
Azure Key Vault
Azure-native key and secrets management:
- Managed HSM support
- Automatic rotation
- Integration with AKS
- RBAC-based access control
External Secrets Operator (ESO)
Kubernetes operator for external secret syncing:
kubectl apply -f https://github.com/external-secrets/external-secrets/releases/download/v0.9.0/external-secrets.yaml
# Create SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-store
spec:
provider:
vault:
server: "https://vault.example.com"
path: "secret"
auth:
kubernetes:
mountPath: "kubernetes"
Secrets Management Best Practices
| Practice | Description |
|---|---|
| Encryption at Rest | Always encrypt secrets using KMS or equivalent tools |
| External Secret Stores | Use Vault, AWS Secrets Manager, or similar for production |
| RBAC for Secrets | Limit access to secrets using Kubernetes RBAC and namespace isolation |
| Secret Rotation | Implement automatic rotation for credentials and API keys |
| Audit Logging | Log all secret access and modifications for compliance |
| Never Log Secrets | Exclude secrets from logs, error messages, and monitoring |
✓ Recommendation: Use External Secrets Operator with Vault for production clusters
combining centralized management with Kubernetes-native operations.
Common Secrets Management Issues
Secrets Exposed in Logs
Solution: Implement log filtering to redact sensitive data
Hardcoded Secrets in Code
Solution: Use git hooks to prevent secret commits; scan repositories with tools like TruffleHog
Secret Sprawl
Solution: Maintain inventory of all secrets and implement lifecycle management