Kubernetes Secret Management: How to Stop Leaking Credentials Before They Burn Down Your Production Environment

TL;DR Quick Answer: Native Kubernetes Secrets are base64-encoded, not encrypted — storing them in Git or relying on etcd defaults is a critical vulnerability. Production-grade Kubernetes Secret Management requires envelope encryption at rest, dynamic secret injection via tools like HashiCorp Vault or AWS Secrets Manager, and GitOps-safe workflows using External Secrets Operator (ESO) or Sealed Secrets. This article walks you through the full architecture, toolchain, and hardened configurations you need to ship with confidence.

The Dirty Truth About Kubernetes Secrets Nobody Talks About

Every engineering team that has ever deployed a containerised application has brushed up against Kubernetes Secret Management. Most of them got it wrong on the first attempt — and some of them paid for it in production. The uncomfortable truth is that a native kubectl create secret command does not encrypt your data. It base64-encodes it. That is not encryption. That is obfuscation with a lowercase "o", and any attacker with read access to your etcd cluster or your Git history can reverse it in milliseconds.

At Apargo, we run multi-tenant SaaS platforms, AI inference pipelines, and customer-facing automation products like AI Greentick — all of which carry credentials, API keys, and database connection strings that would be catastrophic if exposed. We have invested deeply in hardening our secret management posture, and this article is the distilled engineering playbook from that work.

Why Default Kubernetes Secret Management Is a Security Liability

Before we discuss solutions, it is worth understanding exactly what the default behaviour gives you — and why it falls short.

Base64 Is Not Encryption

When you run kubectl create secret generic db-creds --from-literal=password=supersecret, Kubernetes stores the value as c3VwZXJzZWNyZXQ= in etcd. Any engineer with kubectl get secret permissions can decode it instantly:

# Decoding a "secret" takes exactly one command
echo "c3VwZXJzZWNyZXQ=" | base64 --decode
# Output: supersecret

This means that RBAC misconfiguration, a compromised CI/CD pipeline token, or a developer with overly broad cluster access can silently exfiltrate every credential in your namespace.

etcd Is Unencrypted by Default

Even if your RBAC is airtight, etcd — the key-value store backing your cluster state — stores secrets in plaintext unless you explicitly configure encryption at rest. According to the official Kubernetes documentation, encryption at rest is an opt-in feature, and the majority of self-managed clusters never enable it. A single etcd snapshot in the wrong hands is a full credential dump.

GitOps Workflows Amplify the Risk

Modern teams use GitOps tools like ArgoCD or Flux to declaratively manage cluster state. If you commit a raw Kubernetes Secret manifest to Git — even a private repository — you have permanently baked credentials into your version history. git log does not forget. Neither do attackers.

The Production-Grade Kubernetes Secret Management Architecture

A robust Kubernetes Secret Management architecture has four distinct layers:

1. Encryption at Rest — Ensure etcd encrypts secret objects using AES-GCM or KMS provider envelopes.
2. Centralised Secret Store — Use an external vault (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) as the source of truth.
3. GitOps-Safe Injection — Use External Secrets Operator or Sealed Secrets to safely reference secrets in Git without exposing values.
4. Least-Privilege Access Controls — Enforce tight RBAC, namespace isolation, and audit logging on every secret read.

Layer 1: Enabling Envelope Encryption at Rest in etcd

The first line of defence in any serious Kubernetes Secret Management setup is encrypting secrets inside etcd. Kubernetes supports this via an EncryptionConfiguration manifest applied to the API server.

# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # KMS envelope encryption (recommended for production)
      - kms:
          name: aws-kms-provider
          endpoint: unix:///var/run/kmsplugin/socket.sock
          cachesize: 1000
          timeout: 3s
      # AES-GCM fallback (acceptable for non-KMS setups)
      - aesgcm:
          keys:
            - name: key1
              secret: 
      # Identity provider MUST be last — it stores plaintext
      - identity: {}

Pass this configuration to the kube-apiserver with the flag --encryption-provider-config=/etc/kubernetes/encryption-config.yaml. After applying, run a forced re-encryption of all existing secrets:

# Re-encrypt all existing secrets in all namespaces
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

With AWS KMS as the envelope key provider, you gain hardware-backed key management, automatic key rotation, and CloudTrail audit logs for every decrypt operation — reducing your blast radius by an estimated 80% compared to a plaintext etcd setup.

Layer 2: HashiCorp Vault as Your Centralised Secret Store

Encryption at rest hardens etcd, but it does not solve the problem of secret lifecycle management — rotation, revocation, dynamic generation, and audit trails. That is where HashiCorp Vault becomes indispensable.

Vault's Kubernetes Auth Method

Vault integrates natively with Kubernetes via its kubernetes auth method. Pods authenticate to Vault using their projected service account tokens (PSAT), and Vault returns a short-lived token scoped to specific secret paths. No static credentials are ever stored in the cluster.

# Enable the Kubernetes auth method in Vault
vault auth enable kubernetes

# Configure it with your cluster's API server details
vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

# Create a policy that grants read access to a specific secret path
vault policy write app-db-policy - <

# Enable the database secrets engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/production-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.internal:5432/appdb" \
  username="vault-root" \
  password="vault-root-password"

# Create a role that generates credentials with a 1-hour TTL
vault write database/roles/app-role \
  db_name=production-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Add the ESO Helm repository
helm repo add external-secrets https://charts.external-secrets.io

# Install ESO into your cluster
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

# SecretStore — defines the connection to AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

---
# ExternalSecret — declares which secret to sync and how
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h          # Re-sync every hour automatically
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: db-credentials        # Name of the resulting Kubernetes Secret
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD    # Key in the Kubernetes Secret
      remoteRef:
        key: production/db-creds
        property: password      # Property inside the AWS secret JSON

# Install the Sealed Secrets controller
helm install sealed-secrets \
  sealed-secrets/sealed-secrets \
  -n kube-system

# Seal a secret using the cluster's public key
kubectl create secret generic db-creds \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-creds.yaml

# The output (safe to commit to Git):
# apiVersion: bitnami.com/v1alpha1
# kind: SealedSecret
# metadata:
#   name: db-creds
# spec:
#   encryptedData:
#     password: AgBy8hgF3k... (RSA-OAEP encrypted ciphertext)