Certificate Management | Kubernetes Security Guide

Certificate Management Overview

Certificate management is critical for securing communication in Kubernetes clusters. Every component in Kubernetes—from the API server to kubelets to services—uses TLS certificates for authentication and encryption. Proper certificate management ensures secure communication, prevents unauthorized access, and maintains cluster integrity.

💡 Key Insight: Kubernetes clusters require hundreds of certificates for inter-component communication. Mismanagement can lead to cluster-wide outages or security breaches.

Why Certificate Management Matters

Authentication: Verify identity of cluster components and users
Encryption: Protect data in transit between cluster components
Compliance: Meet regulatory requirements (PCI-DSS, HIPAA, etc.)
Operational Continuity: Prevent service disruptions from expired certificates
Security: Prevent man-in-the-middle (MITM) attacks
Audit Trail: Track certificate creation, usage, and rotation

Kubernetes Certificate Types

1. Client Certificates

Used for authentication between components and the API server:

API Server Certificate: Serves the Kubernetes API
Kubelet Client Certificates: Kubelet authenticates to the API server
API Server Client Certificates: API server communicates with kubelets
Controller Manager Certificates: Controller manager authentication
Scheduler Certificates: Scheduler authentication

2. Service Account Tokens

Automatically generated and mounted in pods for authentication:

Stored in secrets in the cluster
Used by applications to authenticate to the API server
Should be rotated regularly

3. etcd Certificates

Secure communication between etcd cluster members and clients:

Peer certificates for etcd cluster communication
Server certificates for client connections
Client certificates for API server access

4. User/External Certificates

For external users accessing the cluster:

Client certificates for kubectl
External service authentication

Certificate Lifecycle Management

Generation

Certificates are generated during cluster initialization using kubeadm or other tools:

# Check existing certificates
sudo kubeadm certs check-expiration

# Generate new certificates
sudo kubeadm certs renew all
            

Validation

Regular validation ensures certificates are valid and not expired:

Check certificate expiration dates
Verify certificate issuer and chain
Validate certificate purpose

# View certificate details
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

# Check expiration
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
            

Rotation

Certificates must be rotated before expiration (typically yearly for kubeadm):

Manual rotation for static cluster components
Automated rotation for workload certificates (using cert-manager)
Zero-downtime rotation strategies

Revocation

Compromised certificates should be revoked immediately:

Update API server to reject revoked certificates
Regenerate all dependent certificates
Update all cluster components

Certificate Management Tools

Cert-Manager

Industry standard for managing certificates in Kubernetes:

Automatic certificate generation and renewal
Support for multiple CA providers (Let's Encrypt, Vault, etc.)
Automatic secret creation and updates
Certificate monitoring and alerting

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Create certificate resource
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-cert
spec:
  secretName: tls-secret
  issuerRef:
    name: letsencrypt-prod
  dnsNames:
  - example.com
            

Vault

Secure certificate and secret storage:

Centralized PKI management
Dynamic certificate generation
Short-lived credentials

External Certificate Authorities

Integration with enterprise CAs
Let's Encrypt for public certificates
AWS Certificate Manager
Azure Key Vault

Certificate Management Best Practices

Practice	Description
Automated Rotation	Use cert-manager or similar tools to automate rotation before expiration
Regular Monitoring	Monitor certificate expiration dates and set up alerts (30, 7, 1 days before expiry)
Secure Storage	Store private keys in secure locations, encrypt at rest, limit access
Certificate Pinning	Pin certificates for critical connections to prevent MITM attacks
Strong Key Length	Use 2048-bit RSA or 256-bit ECDSA keys minimum
Short TTL	Use shorter TTLs for certificates to reduce blast radius of compromise
Regular Audits	Audit certificate inventory, validity, and permissions regularly
Disaster Recovery	Maintain backups of CA certificates and private keys in secure vaults

⚠️ Warning: Never commit private keys to version control. Use secret management tools and follow least privilege principles.

Implementation Steps

Step 1: Audit Current Certificates

# Check all cluster certificates
kubeadm certs check-expiration

# Check component certificates individually
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -dates
            

Step 2: Set Up Certificate Monitoring

Install monitoring/alerting tools (Prometheus, Alertmanager)
Create alerts for certificates expiring in 30, 7, and 1 day
Integrate with incident management systems

Step 3: Implement Automated Rotation

# Enable automatic certificate rotation in kubeadm
kubeadm certs renew all --schedule 30d

# Or use cert-manager for workload certificates
kubectl apply -f cert-manager-issuer.yaml
            

Step 4: Regular Testing

Test certificate rotation in staging environment
Verify zero-downtime rotation
Document rotation procedures

Common Certificate Issues & Solutions

Certificate Expired

Symptom: API server returns certificate validation errors

Solution: Renew certificates immediately using kubeadm certs renew all

Certificate Authority Not Trusted

Symptom: kubectl commands fail with certificate validation errors

Solution: Verify CA certificate configuration in kubeconfig

Service Certificate Mismatch

Symptom: Services fail to establish TLS connections

Solution: Verify certificate CN/SAN matches service DNS names

Unscheduled Pod Due to Certificate Issues

Symptom: Pods stuck in pending state

Solution: Check kubelet certificates and API server connectivity