Kubernetes Secrets Management Patterns

💡 Quick Answer: Secure secrets in Kubernetes with External Secrets Operator, Sealed Secrets, Vault, and SOPS. Encryption at rest, rotation, and zero-trust patterns.

The Problem

This is a critical skill for managing production Kubernetes clusters at scale. Without it, teams face operational complexity, security risks, and reliability issues.

The Solution

Don’t commit raw Secrets to Git. Sync them from an external manager with the External Secrets Operator. This ExternalSecret pulls a database password from Vault into a native Kubernetes Secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials        # the K8s Secret that gets created
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/prod/db
        property: password

Enable encryption at rest so etcd never stores plaintext Secrets:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Rotate keys and credentials on a schedule, and restrict Secret reads with least-privilege RBAC.

Common Issues

Troubleshooting

Check logs and events first. Most issues have clear error messages pointing to the root cause.

Best Practices

• Follow the principle of least privilege for all configurations
• Test in staging before applying to production
• Monitor and alert on key metrics
• Document your runbooks for the team

Key Takeaways

• Essential knowledge for Kubernetes operations at scale
• Start simple and evolve your approach as needed
• Automation reduces human error and operational toil
• Share learnings across your team