Config on Kubernetes: Secrets, restarts, rotation, feature flags, drift Intermediate

This page answers: how config gets into a pod, what you must do when it changes so that it is actually applied, and how to change secrets and flags without causing an incident.

ConfigMap and Secret into the pod

spec:
  containers:
    - name: api
      image: registry.example.com/payment-api@sha256:...   # same digest in every environment
      envFrom:
        - configMapRef:
            name: payment-api-config        # DATABASE_POOL_MAX, PAYMENT_API_BASE_URL, LOG_LEVEL
      env:
        - name: PAYMENT_API_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: payment-api-secrets
              key: PAYMENT_API_SECRET_KEY

What the Kubernetes docs say that is often overlooked:

Fact	Consequence
A ConfigMap “does not provide secrecy or encryption”; max 1 MiB	Do not put secrets in a ConfigMap
Secret values are base64, which “obscures them but does NOT provide any useful level of confidentiality”	base64 is not encryption
By default Secrets are stored unencrypted in etcd	Enable encryption at rest and least-privilege RBAC
Anyone allowed to create pods in a namespace can read every Secret in that namespace	Split namespaces along permission boundaries
ConfigMaps consumed as env vars “are not updated automatically and require a pod restart”	After editing the ConfigMap, running pods still use the old value
Volume mounts are updated eventually (kubelet sync + cache); subPath mounts are not	Files can change at runtime, but the app has to reread them itself

Editing a ConfigMap does not trigger a rollout

A Deployment only rolls out when the pod template (.spec.template) changes. Editing a ConfigMap does not change the template, so it does not roll out by itself.

Loading diagram…

Common ways to make the template change with the config:

• Helm: an annotation containing a checksum of the config.
• Kustomize: configMapGenerator appends a hash to the ConfigMap name.
• Manually: kubectl rollout restart after the edit.

Kubernetes does not roll back automatically. If new pods exit because of bad config, the Deployment reports ProgressDeadlineExceeded after progressDeadlineSeconds (default 600) and stops there.

With the default maxUnavailable of 25%, some old pods may already be removed before the new pods fail. Correct readiness configuration is in Health probes.

Restart or hot reload?

Restart when config changes (default)

• Config is immutable for the life of the process; code does not have to handle “the value changed mid-request”.
• The rollout goes through readiness and graceful shutdown; with bad config new pods never become ready and old pods keep running.
• A config change has a history, like a deploy.

Hot reload inside the process

• Every change has to propagate through pools, clients and caches initialized from old values; half-old, half-new state is easy to end up with.
• Bad config gets loaded into pods that are currently serving, with no readiness check to stop it.
• Only worth it for a small set of values: log level, rotating secrets, flags.

Default: restart. Values that need to change at runtime are usually feature flags, so put them in a flag service instead of writing your own hot reload. The legitimate exception is secret rotation.

Secret rotation: build the hook in advance

Secrets will have to change: periodically, or when leaked. If you read a secret once at startup, the only way to change it is a restart. That is fine if the process has a transition phase:

1. Create the new credential at the provider, keeping the old credential valid.
2. Update the Secret and do a rollout restart. New pods use the new key, old pods still use the old, still-valid key.
3. Once every pod runs the new key, revoke the old one.

For keys that you verify (webhook signing secret, JWT signing key), the app must accept two values at once during the transition:

• WEBHOOK_SIGNING_SECRETS=new,old, tried in order.
• Or JWTs with a kid that selects the key.

Without this hook, rotation turns into a planned incident.

To change secrets without a restart: mount the Secret as a file (no subPath) and reread it when the file changes or when the provider returns 401. For handling a leaked secret, see Secure coding.

Feature flags: config that changes at runtime

Pete Hodgson (“Feature Toggles” on martinfowler.com) splits flags into four categories, which differ in longevity and dynamism:

Category	Used for	Longevity	Decision
Release	Merging unfinished code into main, turning it on when ready	Short, “transitionary by nature”	Nearly static
Experiment	A/B tests	Long enough to get data	Per request/user
Ops	Kill switch, shedding load from secondary features	Usually short; some kill switches live long	Must change very fast
Permissioning	Features for premium plans, beta users	Can be very long	Per request

Loading diagram…

The original article calls flags “inventory which comes with a carrying cost”. In practice:

• Each flag gets an owner and an expiry date when created. An expired release flag is a ticket, not decoration.
• You can set a limit on the number of live flags, as the original article suggests, to force cleanup.
• A kill switch (a long-lived ops flag) is a deliberate exception: turn off recommendations when the DB is overloaded. Check periodically that turning it off still works.
• Flags read at runtime should have a fallback value when the flag service does not respond. The fallback for a new feature is false.

An env var FEATURE_X_ENABLED is still fine for a static release flag. When you need per-user targeting or to switch off within seconds, move to a flag service.

Config drift and dev/prod parity

12-factor names three gaps between dev and prod: time, personnel, tools. For config, drift usually looks like this:

• Staging has a key that production lacks. Discovered during the production deploy.
• Someone runs kubectl edit on the production ConfigMap at 2 a.m. and nobody records it. The next manifest apply overwrites it.
• Local uses SQLite, production uses PostgreSQL. 12-factor says it directly: “resists the urge to use different backing services”.

How to prevent it:

1. Config (except secrets) lives in git and is applied through CI or GitOps. No manual edits on the cluster.
2. The schema in loadConfig is the single source of truth. CI checks that every environment has every key in the schema.
3. Local runs the same kind of DB, broker and cache as production (containers), just smaller.
4. Rename or remove keys expand–contract style: read both new and old keys in one release, update every environment, then drop the old key.

Renaming an env var without breaking the deploy

The team wants to rename PAY_TIMEOUT (seconds) to PAYMENT_API_TIMEOUT_MS (milliseconds). Changing it in one PR means the new release reads the new key while production’s Secret and ConfigMap still only have the old key.

Do it in three steps:

1. Release N: loadConfig reads PAYMENT_API_TIMEOUT_MS, falling back to PAY_TIMEOUT times 1000, and logs a “deprecated key” warning.
2. Update the ConfigMap in every environment to the new key. The warning disappears from the logs.
3. Release N+1: delete the branch that reads the old key. The old key is no longer in the schema; if it lingers in some environment, the CI schema check reports it.

Next: Common mistakes.